phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
@@ -69,6 +69,15 @@ os.environ["BOR_RECENCY_HALF_LIFE_DAYS"] = str(
|
||||
_Settings.model_fields["recency_half_life_days"].default
|
||||
)
|
||||
|
||||
# Phase 122 (task 01): the same leak class for the image-document
|
||||
# toggle — an operator's local ``.env`` may legitimately carry
|
||||
# ``BOR_IMAGES``/``BOR_IMAGE_EXTENSIONS``/``BOR_IMAGE_DIR``, and the
|
||||
# "off by default" pins (the ``/api/config`` ``images`` flag, the
|
||||
# walk's images-off behavior) must see the code defaults.
|
||||
os.environ["BOR_IMAGES"] = str(_Settings.model_fields["images"].default)
|
||||
os.environ["BOR_IMAGE_EXTENSIONS"] = str(_Settings.model_fields["image_extensions"].default)
|
||||
os.environ["BOR_IMAGE_DIR"] = str(_Settings.model_fields["image_dir"].default)
|
||||
|
||||
from app.db import SessionLocal, db_available # noqa: E402
|
||||
from app.main import app as fastapi_app # noqa: E402
|
||||
|
||||
|
||||
+73
-5
@@ -26,6 +26,14 @@ Implements just enough of the aipi surface:
|
||||
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
|
||||
marker as a substring, so the summary branch would otherwise
|
||||
shadow every folder-summary call
|
||||
- ``IMAGE_DESCRIPTION_MODE`` (phase 122, image documents) -> the
|
||||
fixed ``IMAGE_DESCRIPTION_ANSWER`` description: the CHAT model's
|
||||
(vision) reply to ``summarizer.describe_image``'s multimodal user
|
||||
message (the marker is its first text part; the call is
|
||||
non-streaming, so only this answer path serves it). The reply is
|
||||
the image document's whole content + summary, so it is byte-
|
||||
stable and token-dense (its cosine against the story suite's
|
||||
question clears the mock-calibrated gate)
|
||||
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
|
||||
- otherwise -> upbeat answer quoting the provided document context
|
||||
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
|
||||
@@ -592,19 +600,46 @@ def _messages(body: dict[str, Any]) -> list[dict[str, str]]:
|
||||
return body.get("messages", [])
|
||||
|
||||
|
||||
def _content_text(content: Any) -> str:
|
||||
"""The TEXT of one message's content (phase 122 list safety).
|
||||
|
||||
String content passes through byte-identical (as does an absent
|
||||
value or an explicit ``None`` — the ``or ""`` semantics the
|
||||
:func:`_context` docstring pins). A multimodal part LIST — the
|
||||
phase-122 image description's ``[{type: "text", …},
|
||||
{type: "image_url", …}]``, the only list-content message the app
|
||||
produces (``app.rag.summarizer.describe_image``) — contributes its
|
||||
text parts joined (the ``image_url`` part carries no text): the
|
||||
trigger checks and marker branches read the text, and no existing
|
||||
string-content request is affected.
|
||||
"""
|
||||
if isinstance(content, list):
|
||||
return " ".join(
|
||||
part.get("text", "")
|
||||
for part in content
|
||||
if isinstance(part, dict) and part.get("type") == "text"
|
||||
)
|
||||
return content or ""
|
||||
|
||||
|
||||
def _system(body: dict[str, Any]) -> str:
|
||||
return " ".join(m.get("content", "") for m in _messages(body) if m.get("role") == "system")
|
||||
return " ".join(
|
||||
_content_text(m.get("content"))
|
||||
for m in _messages(body)
|
||||
if m.get("role") == "system"
|
||||
)
|
||||
|
||||
|
||||
def _user(body: dict[str, Any]) -> str:
|
||||
parts = [m.get("content", "") for m in _messages(body) if m.get("role") == "user"]
|
||||
parts = [_content_text(m.get("content")) for m in _messages(body) if m.get("role") == "user"]
|
||||
return parts[-1] if parts else ""
|
||||
|
||||
|
||||
def _context(body: dict[str, Any]) -> str:
|
||||
"""The document context is the longest system/user message in practice.
|
||||
|
||||
``m.get("content") or ""`` (NOT ``m.get("content", "")``): a well-formed
|
||||
``_content_text(m.get("content"))`` (the ``or ""`` semantics, NOT
|
||||
``m.get("content", "")``): a well-formed
|
||||
OpenAI tool-call message carries ``content: None`` EXPLICITLY (the app's
|
||||
agent loop appends exactly that — ``app/rag/agent.py``), and a forced
|
||||
final answer after a tool round (the round-cap path) reaches this helper
|
||||
@@ -612,9 +647,12 @@ def _context(body: dict[str, Any]) -> str:
|
||||
an explicit ``None`` and crashes ``len()`` with a 500 (phase 93 task 04
|
||||
caught it via the deterministic single-read flow's ALREADY_IN_CONTEXT
|
||||
loop); ``or ""`` treats absent and explicit-None alike, so the fallback
|
||||
composes deterministically instead of traceback-ing."""
|
||||
composes deterministically instead of traceback-ing. Phase 122: a
|
||||
multimodal part list maps to its text parts (see
|
||||
:func:`_content_text`), so ``len()``/slicing never meet a list.
|
||||
"""
|
||||
msgs = _messages(body)
|
||||
return max((m.get("content") or "" for m in msgs), key=len)
|
||||
return max((_content_text(m.get("content")) for m in msgs), key=len)
|
||||
|
||||
|
||||
LONG_ANSWER_TRIGGER = "write a long answer"
|
||||
@@ -739,6 +777,25 @@ HISTORY_TRIGGER = "echo my history"
|
||||
#: phrase, so every other suite is unaffected.
|
||||
FOLDER_MAP_TRIGGER = "repeat your folder map"
|
||||
|
||||
#: Phase 122 (image documents, LOCKED A3): the fixed description the
|
||||
#: mock's vision (CHAT) model returns for an ``IMAGE_DESCRIPTION_MODE``
|
||||
#: request — the multimodal user message
|
||||
#: ``[{type: "text", …IMAGE_DESCRIPTION_MODE…}, {type: "image_url",
|
||||
#: …}]`` (``app.rag.summarizer.describe_image``, the app's only
|
||||
#: list-content message). It becomes the image document's WHOLE
|
||||
#: ``content`` AND ``summary`` (the only embedded text — the embedding
|
||||
#: model never sees pixels), so the text is byte-stable across runs and
|
||||
#: token-dense: the story suite's question ("What is shown in the
|
||||
#: homelab network diagram?", cosine ≈0.38 against the mock's
|
||||
#: token-overlap embeddings) clears the mock-calibrated gate (0.30)
|
||||
#: and the citation usefulness floor (0.15).
|
||||
IMAGE_DESCRIPTION_ANSWER = (
|
||||
"A network diagram of the homelab server room: a core router on top, "
|
||||
"a core switch in the middle, and three labeled subnets at the bottom — "
|
||||
"VLAN 10 office, VLAN 20 lab, and VLAN 30 storage — with a legend of "
|
||||
"cable runs. Title: Homelab Network Map."
|
||||
)
|
||||
|
||||
TABLE_ANSWER = (
|
||||
"Here's the shape, in a table:\n"
|
||||
"\n"
|
||||
@@ -2178,6 +2235,17 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
answer = "Knowledge base outline:\n- " + " ".join(
|
||||
TOKEN_RE.findall(user.lower())[:8]
|
||||
)
|
||||
elif "IMAGE_DESCRIPTION_MODE" in user:
|
||||
# Phase 122 (image documents, LOCKED A3): the CHAT model's
|
||||
# (vision) image description — a NON-STREAMING multimodal user
|
||||
# message, so only this path ever sees it. ``_content_text``
|
||||
# joins the text parts, so the marker (the first text part)
|
||||
# lands in ``user``. Checked before the user-trigger branches:
|
||||
# the marker is a fixed app constant, and a real question is
|
||||
# never expected to type it (the other user triggers are owner
|
||||
# phrasings a question might legitimately contain — this one
|
||||
# is an app-to-app marker).
|
||||
answer = IMAGE_DESCRIPTION_ANSWER
|
||||
elif TABLE_TRIGGER in user.lower():
|
||||
# Markdown tables (phase 44, TODO.md L6): the story E2E's
|
||||
# deterministic table answer — a 3-column table, the
|
||||
|
||||
@@ -149,27 +149,29 @@ def test_api_config_serves_both_names(testy_server: str, app_server: str) -> Non
|
||||
# "Save as doc" gating; both instances run with BOR_DOCS_REPO
|
||||
# empty, so it is the inert false here. Phase 62 (task 01): the
|
||||
# endpoint grew with the UI-customization keys; phase 91
|
||||
# (task 03) deleted the retired CSS-file theming's ``theme`` key —
|
||||
# the five keys below are the entire contract (this suite's
|
||||
# instances carry no UI-customization overrides, so the string
|
||||
# keys are their defaults).
|
||||
# (task 03) deleted the retired CSS-file theming's ``theme`` key;
|
||||
# phase 122 (task 01) added the ``images`` flag (inert false here
|
||||
# — neither instance sets BOR_IMAGES) — the six keys below are
|
||||
# the entire contract (this suite's instances carry no
|
||||
# UI-customization overrides, so the string keys are their
|
||||
# defaults).
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["app_name"] == TESTY_NAME
|
||||
assert body["docs_repo_configured"] is False
|
||||
assert body["images"] is False
|
||||
|
||||
# The shared conftest instance keeps the default (the other
|
||||
# suites' title/label contract rides on it) — and its key set
|
||||
# tracks the endpoint contract (five keys after phase 91,
|
||||
# task 03).
|
||||
# tracks the endpoint contract (six keys after phase 122, task 01).
|
||||
r2 = httpx.get(f"{app_server}/api/config", timeout=5)
|
||||
assert r2.status_code == 200
|
||||
r2_body = r2.json()
|
||||
assert set(r2_body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert r2_body["app_name"] == DEFAULT_NAME
|
||||
|
||||
|
||||
@@ -0,0 +1,499 @@
|
||||
"""Phase 122 E2E (Playwright): standalone image documents — the user path.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_image_documents.py -v --no-cov
|
||||
|
||||
The suite's app instance runs with ``BOR_IMAGES=true`` (module env
|
||||
override — the conftest per-suite-app pattern, leak guards included):
|
||||
the standalone PNG arrives through the REAL admin upload flow (a zip
|
||||
with a single image member — phase 90's no-scan contract: the upload
|
||||
unpacks + registers, the RAG page's "Sync sources" button scans), the
|
||||
mock's vision model (``IMAGE_DESCRIPTION_MODE`` branch —
|
||||
``tests/e2e/mock_llm.py``) writes the description, and the document is
|
||||
first-class at every surface:
|
||||
|
||||
* the Sources page lists it with the 48px thumbnail (the image bytes
|
||||
route, alt = the summary);
|
||||
* the document viewer renders the image with the description below it
|
||||
(the summary panel is suppressed — the summary IS the description);
|
||||
* a grounded chat question shows the compact inline figure in the
|
||||
answer's sources block (image + the summary caption, the "shown in
|
||||
the chat nicely" contract).
|
||||
|
||||
The negative case drives a SECOND module app with ``BOR_IMAGES``
|
||||
forced ``false`` (the DEFAULT contract — an operator's local ``.env``
|
||||
cannot leak the toggle in either direction: both apps pin the value
|
||||
explicitly): the same upload + sync produces NO image document (the
|
||||
walk is blind to the file, the source row stays a 0-document source).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import zipfile
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.agent import IMAGE_DOC_MARKER
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
GIT_SOURCES_URL = "/git-sources.html"
|
||||
SOURCE_NAME = "e2e-image" # the archive stem (archive_source_name)
|
||||
DOC_PATH = "pic.png"
|
||||
|
||||
#: The grounded question, carrying the house scripted-read call
|
||||
#: (``SUMMARY_SEED_READ_TRIGGER`` — the phase-119 A1 convention: a
|
||||
#: zero-read grounded turn chips NOTHING, so the image doc must be
|
||||
#: READ to earn its citation chip + figure): the mock emits the
|
||||
#: scripted ``read e2e-image/pic.png``, then echoes the tool result.
|
||||
#: Grounding: the mock's token-overlap cosine against the
|
||||
#: ``IMAGE_DESCRIPTION_ANSWER`` description is ≈0.31 (over the
|
||||
#: mock-calibrated gate (0.30), and the FTS leg corroborates —
|
||||
#: homelab/network/diagram all hit — the 0.15 floor backstop).
|
||||
IMAGE_QUESTION = (
|
||||
"Read the suggested document: read e2e-image/pic.png — "
|
||||
"what is shown in the homelab network diagram?"
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
|
||||
#: the pipeline is content-agnostic, the well-formed bytes keep the
|
||||
#: upload + serve + render path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module apps: images ON (the story) and images forced OFF (the default
|
||||
# contract). Each owns its port + its scratch upload/source/image dirs.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(mock_llm: int, app_port: int, *, images: bool, scratch: Path) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-122
|
||||
knobs: ``BOR_IMAGES`` pinned EXPLICITLY (true for the story app,
|
||||
false for the default app — process env ranks above an operator's
|
||||
local gitignored ``.env``, so the contract under test cannot leak
|
||||
in either direction) and the image/upload homes in the suite's
|
||||
scratch dir (the app under test must not write image copies into
|
||||
the owner's real ``~/bor-sources``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — the story's question
|
||||
# grounds on these values (see IMAGE_QUESTION).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 122: the toggle under test + the suite-private homes.
|
||||
env["BOR_IMAGES"] = "true" if images else "false"
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
|
||||
"""The story app — ``BOR_IMAGES=true`` (the module env override;
|
||||
the conftest session app is never started in this isolated run,
|
||||
so no port clash)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_image_on")
|
||||
port = int(os.environ.get("E2E_APP_PORT_IMAGES", "8150"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, images=True, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The default-contract app — ``BOR_IMAGES=false`` (the LOCKED A3
|
||||
default; only the negative test starts it)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_image_off")
|
||||
port = int(os.environ.get("E2E_APP_PORT_IMAGES_OFF", "8151"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, images=False, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_url(default_app_server: str) -> str:
|
||||
return default_app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The fixture archive: ONE standalone PNG at the root — source
|
||||
``e2e-image``, document path ``pic.png``."""
|
||||
root = tmp_path_factory.mktemp("bor_image_zip")
|
||||
(root / DOC_PATH).write_bytes(PNG_1X1)
|
||||
archive = root / f"{SOURCE_NAME}.zip"
|
||||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.write(root / DOC_PATH, arcname=DOC_PATH)
|
||||
return archive
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + UI helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry per test (the E2E isolation pattern): the
|
||||
suites share one Postgres, and a leftover row would corrupt the
|
||||
counts. ``sources_meta`` (the KB generation counter) resets with
|
||||
the KB — the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _upload_zip(page: Page, app_url: str, archive: Path) -> None:
|
||||
"""The real admin upload flow: form login → the git-sources page →
|
||||
pick the archive → submit → the terminal result line (phase 90:
|
||||
"Uploaded <source> — press Sync sources to import it.")."""
|
||||
login(page, app_url, next=GIT_SOURCES_URL)
|
||||
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
page.set_input_files("#archive-upload-file", str(archive))
|
||||
page.click("#archive-upload-btn")
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
|
||||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||||
"""The RAG page's "Sync sources" button (the scan — phase 90 A3):
|
||||
click → "Syncing…" → the terminal "Synced HH:MM" label + the fresh
|
||||
counts in #sync-result (the never-stale contract, test_git_source_
|
||||
dates' lifecycle wait)."""
|
||||
page.goto(app_url + "/sources.html")
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||||
)
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(page.locator("#sync-result")).to_have_text(expected_result)
|
||||
|
||||
|
||||
def _drill_to_source(page: Page, name: str) -> None:
|
||||
"""Top level → the source (the file table then holds its direct
|
||||
files — ours is at the source root)."""
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module seed: upload + sync the fixture image through the real UI
|
||||
# (the story E2E truncate/re-import-per-module pattern).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str,
|
||||
browser: Browser,
|
||||
zip_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""Upload the single-PNG zip through the admin UI, then scan it
|
||||
with the RAG page's Sync button (the mock's vision model describes
|
||||
the image in-process). Yields the seeded doc's identity (the bytes
|
||||
route + the description) for the render assertions. The module
|
||||
app's ``/api/config`` flag is asserted first — the env override
|
||||
must have reached the app under test, or the whole premise of the
|
||||
suite is void."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, app_url, zip_path)
|
||||
_sync_via_ui(page, app_url, expected_result="1 added")
|
||||
with SessionLocal() as db:
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert doc is not None, "the uploaded image must become a document"
|
||||
assert doc.is_image is True and doc.image_path is not None
|
||||
# content == summary == the mock's description (the ONLY
|
||||
# embedded text of the doc — the pipeline contract, LOCKED A3).
|
||||
assert doc.content == IMAGE_DESCRIPTION_ANSWER
|
||||
assert doc.summary == IMAGE_DESCRIPTION_ANSWER
|
||||
finally:
|
||||
page.close()
|
||||
yield {
|
||||
"image_url": f"/api/documents/{doc.id}/image",
|
||||
"description": IMAGE_DESCRIPTION_ANSWER,
|
||||
}
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Sources page: the image doc is listed with its thumbnail
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sources_page_lists_the_image_doc(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The KB total is the image doc (1 doc, 2 chunks — the one
|
||||
content chunk + the phase-30 ``is_summary`` chunk), the drill-down
|
||||
tree reaches it, and its row carries the FIXED 48px thumbnail box:
|
||||
a lazy ``<img>`` from the bytes route (not the glyph fallback —
|
||||
the fixture PNG is served), ``alt`` = the summary (the WCAG
|
||||
contract)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("2")
|
||||
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
row = page.locator("#docs-tbody tr", has_text=DOC_PATH)
|
||||
expect(row).to_have_count(1)
|
||||
|
||||
box = row.locator(".kb-doc-thumb")
|
||||
expect(box).to_be_visible()
|
||||
img = row.locator(".kb-doc-thumb-img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(img).to_have_attribute("alt", seeded["description"])
|
||||
expect(row.locator(".kb-doc-thumb-glyph")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Document viewer: the image renders, the description below it
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_document_viewer_renders_image_and_description(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The Sources row's path link opens the same-page document modal
|
||||
(phase 26): the ``is_image`` content block renders the PERSISTENT
|
||||
bytes first (``.doc-image-img`` from the bytes route, ``alt`` =
|
||||
the summary), and the description follows in the normal content
|
||||
slot (``pre.doc-raw`` — the plain-content path). The labeled
|
||||
Summary panel is SUPPRESSED for the verbatim-description case
|
||||
(summary === content — the importer invariant; the panel would
|
||||
duplicate the text right below the image)."""
|
||||
login(page, app_url)
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
|
||||
|
||||
modal = page.locator("#doc-modal")
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text("pic")
|
||||
|
||||
img = modal.locator(".doc-image-img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000) # the bytes route serves the PNG
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("alt", seeded["description"])
|
||||
|
||||
# The description below the image (the doc's readable content IS
|
||||
# the vision description).
|
||||
expect(modal.locator("pre.doc-raw")).to_have_text(seeded["description"])
|
||||
# The verbatim case: no duplicate Summary panel.
|
||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
||||
# No "Image unavailable" note — the copy is intact.
|
||||
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Chat: a grounded question shows the inline image in the sources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_chat_sources_block_shows_the_inline_image(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""A question the mock grounds on the image doc (cosine ≈0.31,
|
||||
FTS-corroborated), scripted to READ it (phase 119 A1 — a zero-read
|
||||
turn chips nothing): the mock's echo carries the agent ``read``
|
||||
result VERBATIM — including the task-05 ``IMAGE_DOC_MARKER`` line
|
||||
(the model saw the description, not raw bytes) — and the answer's
|
||||
sources block carries the citation chip (the READ doc clears the
|
||||
usefulness floor) AND the compact inline figure — the ``<img>``
|
||||
from the bytes route, the visible caption + ``alt`` settling to
|
||||
the document's summary (the figure fetches it from the content
|
||||
endpoint — the frame carries no summary), the "shown in the chat
|
||||
nicely" contract."""
|
||||
login(page, app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
page.fill("#message-input", IMAGE_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
# The scripted read's echoed result — the marker line (task 05)
|
||||
# and the description itself (the model's view of the image).
|
||||
expect(bubble).to_contain_text(IMAGE_DOC_MARKER, timeout=30_000)
|
||||
expect(bubble).to_contain_text(seeded["description"], timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
# The citation chip (the text affordance stays — the figure is
|
||||
# additive, not a replacement).
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
|
||||
|
||||
# The inline figure: image + caption, both settling to the summary
|
||||
# (the content fetch resolves the alt/caption after the title).
|
||||
fig = page.locator(".msg.brain .source-image")
|
||||
expect(fig).to_have_count(1)
|
||||
img = fig.locator(".source-image-img")
|
||||
expect(img).to_be_visible(timeout=15_000)
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("alt", seeded["description"], timeout=15_000)
|
||||
caption = fig.locator(".source-image-caption")
|
||||
expect(caption).to_have_text(seeded["description"], timeout=15_000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. The default contract: BOR_IMAGES off (the default) → the same
|
||||
# upload + sync produces NO image document
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_default_env_upload_produces_no_image_doc(
|
||||
page: Page, default_app_url: str, zip_path: Path
|
||||
) -> None:
|
||||
"""The off-by-default contract end-to-end: with ``BOR_IMAGES``
|
||||
false, the SAME upload + sync is byte-identical to the pre-phase
|
||||
walk — the PNG is invisible to the scan (0 files, 0 added), the
|
||||
source row stays a registered 0-document source, and NO documents
|
||||
row (let alone an ``is_image`` one) exists. The file still lands
|
||||
on disk (the unpack is unchanged — only the walk filter changes)."""
|
||||
if not _health_db(default_app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{default_app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is False, "the default app must run with images off"
|
||||
|
||||
_truncate_all()
|
||||
try:
|
||||
_upload_zip(page, default_app_url, zip_path)
|
||||
_sync_via_ui(
|
||||
page, default_app_url, expected_result="0 added · 0 unchanged"
|
||||
)
|
||||
|
||||
# No document at all — the image file is not even a "file" to
|
||||
# the images-off walk (not unknown, not indexed).
|
||||
with SessionLocal() as db:
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.is_image.is_(True)))
|
||||
is None
|
||||
)
|
||||
assert db.scalar(select(Document).where(Document.source == SOURCE_NAME)) is None
|
||||
|
||||
# The Sources page: 0 docs, the source row (registered) drills
|
||||
# to an EMPTY file table.
|
||||
page.goto(default_app_url + "/sources.html")
|
||||
expect(page.locator("#stat-docs")).to_have_text("0")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("0")
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{SOURCE_NAME}")')
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(0)
|
||||
finally:
|
||||
_truncate_all()
|
||||
@@ -164,14 +164,15 @@ def test_config_serves_the_overrides(custom_server: str) -> None:
|
||||
r = httpx.get(f"{CUSTOM_URL}/api/config", timeout=5)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
# The five-key set (the phase-39/59/62 endpoint contract, phase 91
|
||||
# task 03: the retired CSS-file theming's ``theme`` key is gone)
|
||||
# with the two customization overrides — the app NAME stays the
|
||||
# default (this suite does not re-test BOR_APP_NAME; that is the
|
||||
# phase-39 suite's job).
|
||||
# The six-key set (the phase-39/59/62 endpoint contract, phase 91
|
||||
# task 03: the retired CSS-file theming's ``theme`` key is gone;
|
||||
# phase 122 task 01: the ``images`` flag) with the two
|
||||
# customization overrides — the app NAME stays the default (this
|
||||
# suite does not re-test BOR_APP_NAME; that is the phase-39
|
||||
# suite's job).
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["app_name"] == DEFAULT_NAME
|
||||
assert body["input_placeholder"] == CUSTOM_PLACEHOLDER
|
||||
|
||||
+8
-1
@@ -1,6 +1,8 @@
|
||||
"""Shared test fakes (no network, deterministic)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.config import Settings
|
||||
from app.rag.llm import LLMError
|
||||
|
||||
@@ -15,6 +17,11 @@ class FakeEmbedder:
|
||||
``"Summary of <first token of the user content>"`` and raises
|
||||
:class:`LLMError` when the content contains the sentinel word
|
||||
``SUMMARY-BLOWUP`` (drives the importer's fail-soft summary path).
|
||||
``content`` may be a string or a phase-122 multimodal part list
|
||||
(``dict[str, Any]`` messages, the ``LLMClient.chat`` shape) — the
|
||||
text-summary body above is string-only; a subclass handling the
|
||||
multimodal image description (task 03's vision mock) overrides
|
||||
``chat``.
|
||||
"""
|
||||
|
||||
def __init__(self, dim: int = 768) -> None:
|
||||
@@ -36,7 +43,7 @@ class FakeEmbedder:
|
||||
return vec
|
||||
|
||||
async def chat(
|
||||
self, messages: list[dict[str, str]], model: str | None = None
|
||||
self, messages: list[dict[str, Any]], model: str | None = None
|
||||
) -> str:
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
|
||||
@@ -48,25 +48,30 @@ def test_health_reports_ok(client) -> None:
|
||||
|
||||
|
||||
def test_config_returns_default_app_metadata(client, db: Session) -> None:
|
||||
"""GET /api/config is public (anonymous) and returns exactly five
|
||||
"""GET /api/config is public (anonymous) and returns exactly six
|
||||
keys — the phase-39 app metadata, the phase-59 docs flag (inert
|
||||
false while BOR_DOCS_REPO is empty — the "Save as doc" gating),
|
||||
and the phase-62 UI customization strings (composer placeholder,
|
||||
footer line). Phase 91: with an empty ui_settings table the
|
||||
effective strings are the env defaults (B1 — DB-over-env, the row
|
||||
absent here); the retired CSS-file theming's ``theme`` key is gone
|
||||
(task 03 — the five keys are the entire contract)."""
|
||||
the phase-122 images flag (default false in the test env —
|
||||
LOCKED A3: off by default), and the phase-62 UI customization
|
||||
strings (composer placeholder, footer line). Phase 91: with an
|
||||
empty ui_settings table the effective strings are the env defaults
|
||||
(B1 — DB-over-env, the row absent here); the retired CSS-file
|
||||
theming's ``theme`` key is gone (task 03 — the six keys are the
|
||||
entire contract)."""
|
||||
_clear_ui_settings(db)
|
||||
r = client.get("/api/config")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["app_name"] == "Brain of Reese"
|
||||
assert body["version"] == get_settings().app_version
|
||||
assert body["docs_repo_configured"] is False
|
||||
# Phase 122 (task 01): the images flag is the default off (the
|
||||
# test env sets no BOR_IMAGES) — a real bool, not a truthy string.
|
||||
assert body["images"] is False
|
||||
# Phase 62: UNSET => the phase-61 neutral copy stands (the
|
||||
# byte-identical contract).
|
||||
assert body["input_placeholder"] == "Ask me anything…"
|
||||
@@ -91,7 +96,7 @@ def test_config_follows_overridden_app_name(client, db: Session) -> None:
|
||||
body = r.json()
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["app_name"] == "Brain of Testy"
|
||||
assert body["version"] == "0.1.0"
|
||||
@@ -122,7 +127,7 @@ def test_config_serves_ui_customization_overrides(client, db: Session) -> None:
|
||||
body = r.json()
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["input_placeholder"] == "Ask the vault…"
|
||||
assert body["footer_text"] == "Powered by my own models"
|
||||
@@ -153,6 +158,29 @@ def test_config_docs_flag_tracks_settings(client, db: Session) -> None:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def test_config_images_flag_tracks_settings(client, db: Session) -> None:
|
||||
"""Phase 122 (task 01): ``images`` mirrors ``settings.images``
|
||||
(``BOR_IMAGES``) — a real bool (never a truthy string) that flips
|
||||
true the moment the toggle is on: that flag is the entire frontend
|
||||
gating of the image affordances (the phase-123 attach control,
|
||||
optionally the Sources page hint)."""
|
||||
from app.config import Settings
|
||||
from app.main import app as fastapi_app
|
||||
|
||||
_clear_ui_settings(db)
|
||||
fastapi_app.dependency_overrides[get_settings] = lambda: Settings(
|
||||
images=True,
|
||||
)
|
||||
try:
|
||||
r = client.get("/api/config")
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert isinstance(body["images"], bool)
|
||||
assert body["images"] is True
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
|
||||
def test_suggestions_returns_list(client) -> None:
|
||||
# Phase 79: the chips are user-gated — sign in as the admin first
|
||||
# (the test's purpose is the list shape, not the auth contract).
|
||||
|
||||
@@ -431,8 +431,11 @@ def test_document_content_admin_contract(client: TestClient, db) -> None:
|
||||
"content",
|
||||
"indexed_at",
|
||||
"chunks",
|
||||
"is_image", # added in phase 122 (task 04) — always present
|
||||
}
|
||||
assert body["summary"] is None
|
||||
assert body["is_image"] is False # text doc — and no image_url key (never null)
|
||||
assert "image_url" not in body
|
||||
|
||||
# Unknown docs still 404 (same shape as the phase-16 pin).
|
||||
r = client.get("/api/documents/content", params={"source": "docs", "path": "nope.md"})
|
||||
|
||||
@@ -2076,3 +2076,124 @@ def test_done_event_related_defaults_empty_and_old_payload_parses() -> None:
|
||||
dumped = ChatDoneEvent(**new_payload).model_dump()
|
||||
assert [r["path"] for r in dumped["related"]] == ["b.md"]
|
||||
assert [r["path"] for r in dumped["sources"]] == ["a.md"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 122, task 05 — the SSE source frame's OPTIONAL image_url (the
|
||||
# shared ``source_ref_with_image`` builder: present on an image doc's
|
||||
# ref only, omitted — never null — on every text ref). Task 06
|
||||
# finalizes the phase-122 suite here with the story-level pins.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_image_doc(db) -> Document:
|
||||
"""An ``is_image`` documents row the agent can ``read`` (no chunks
|
||||
needed — the read tool resolves the row by ``(source, path)`` and
|
||||
serves its ``content``; the frame needs ``is_image`` + ``id``
|
||||
only). The fixture's teardown truncates the tables."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="docs",
|
||||
path="pic.png",
|
||||
full_path="/tmp/pic.png",
|
||||
title="pic",
|
||||
content="A red square on a white background.",
|
||||
summary="A red square on a white background.",
|
||||
content_hash="0" * 64,
|
||||
created_at=_FIXTURE_CREATED_AT,
|
||||
is_image=True,
|
||||
image_path="/tmp/pic.png",
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
return doc
|
||||
|
||||
|
||||
def test_grounded_turn_reading_image_doc_frame_carries_image_url_only_for_it(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""Task 05 wire contract: a mocked grounded answer whose agent
|
||||
READS an image doc → the done frame's ref for THAT doc alone
|
||||
carries ``image_url`` (the bytes route the frontend's sources
|
||||
block renders from); the text-doc related refs carry NO
|
||||
``image_url`` key at all (the omission rule — the key is absent,
|
||||
never null)."""
|
||||
doc = _seed_image_doc(db)
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "docs/pic.png"},
|
||||
)
|
||||
]
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# The read image doc is the citation surface (phase 119, A1) — and
|
||||
# its ref is the frame's ONLY image_url carrier.
|
||||
sources = done["sources"]
|
||||
assert [(s["source"], s["path"]) for s in sources] == [("docs", "pic.png")]
|
||||
assert sources[0]["image_url"] == f"/api/documents/{doc.id}/image"
|
||||
# The related tier (ranks 6–7 for the Kubernetes question) is text
|
||||
# docs — the key is ABSENT on every one of them, not null.
|
||||
related = done["related"]
|
||||
assert related
|
||||
for ref in related:
|
||||
assert "image_url" not in ref
|
||||
|
||||
|
||||
def test_text_only_grounded_turn_frame_has_no_image_url_key_anywhere(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""The omission rule at the BYTE level (the phase's byte-identity
|
||||
criterion): a grounded turn whose cited + related docs are ALL
|
||||
text docs serializes a done frame with no ``image_url`` key
|
||||
anywhere — checked on the raw wire text (not a re-serialized
|
||||
dict), and every ref keeps exactly the pre-phase-122 key set."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "docs/homelab/kubernetes.md"},
|
||||
)
|
||||
]
|
||||
]
|
||||
)
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||
try:
|
||||
with client.stream("POST", "/api/chat", json={"message": QUESTION}) as r:
|
||||
assert r.status_code == 200
|
||||
buf = ""
|
||||
raw_frames: list[str] = []
|
||||
for part in r.iter_text():
|
||||
buf += part
|
||||
while "\n\n" in buf:
|
||||
frame, buf = buf.split("\n\n", 1)
|
||||
frame = frame.strip()
|
||||
if frame.startswith("data:"):
|
||||
raw_frames.append(frame)
|
||||
finally:
|
||||
fastapi_app.dependency_overrides.clear()
|
||||
|
||||
done_raw = [f for f in raw_frames if '"type": "done"' in f]
|
||||
assert len(done_raw) == 1
|
||||
# The BYTE check: the key is absent from the wire text itself.
|
||||
assert "image_url" not in done_raw[0]
|
||||
done = json.loads(done_raw[0].removeprefix("data:").strip())
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
("docs", "homelab/kubernetes.md")
|
||||
]
|
||||
for ref in done["sources"] + done["related"]:
|
||||
assert set(ref) == {"source", "path", "title"} # the pre-122 key set
|
||||
|
||||
@@ -7,22 +7,33 @@ Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import inspect
|
||||
import itertools
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import delete, func, select, text
|
||||
|
||||
import app.api.docs as docs_api
|
||||
import app.rag.importer as rag_importer
|
||||
from app.config import Settings
|
||||
from app.core import tokens as token_service
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Chunk, Document, FolderSummary, GitSource
|
||||
from app.rag import git_sources as rag_git_sources
|
||||
from app.rag.folder_summaries import missing_folder_summaries
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import LLMError
|
||||
from app.rag.summarizer import DESCRIBE_PROMPT
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
_TREE_TABLES = "chunks, documents, folder_summaries, git_sources"
|
||||
|
||||
@@ -823,3 +834,603 @@ def test_docs_tree_stat_walk_equivalence_with_flat_list(admin_client, db) -> Non
|
||||
)
|
||||
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 122, task 02 — image ingest end-to-end (the full ``import_sources``
|
||||
# pipeline against the real DB; task 06 finalizes the phase-122 suite here
|
||||
# with the image route + content-endpoint shapes).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A real 1×1 transparent PNG — the importer is content-agnostic (it
|
||||
#: never parses the image), but a well-formed fixture keeps the tests
|
||||
#: honest about what a real upload looks like.
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
def _image_llm[
|
||||
ImageLLM: FakeEmbedder
|
||||
](tmp_path, llm_cls: type[ImageLLM] = FakeEmbedder, **kwargs) -> ImageLLM:
|
||||
"""A fake LLM with the phase-122 image knobs (toggle ON by default;
|
||||
the image dir defaults under *tmp_path* unless overridden).
|
||||
*llm_cls* (task 03) may be the mock vision client (``_MockVisionLLM``
|
||||
below) for the no-seam-patch end-to-end path — the PEP 695 type
|
||||
parameter keeps the helper's return type honest (``chat_models``).
|
||||
"""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
kwargs.setdefault("images", True)
|
||||
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||
llm = llm_cls()
|
||||
llm.settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return llm
|
||||
|
||||
|
||||
def _patch_description(monkeypatch, description) -> None:
|
||||
"""Pin the task-02 seam (``rag_importer._describe_or_skip``) to
|
||||
*description*. Task 03 fills the seam with the CHAT model's vision
|
||||
call — these end-to-end mechanics do not change with it."""
|
||||
|
||||
async def _fake(llm, *, data, source, rel, full_path):
|
||||
return description
|
||||
|
||||
monkeypatch.setattr(rag_importer, "_describe_or_skip", _fake)
|
||||
|
||||
|
||||
def _cleanup_source(db, source: str) -> None:
|
||||
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_import_sources_images_on_indexes_image_docs(
|
||||
db, tmp_path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""``images=True`` end-to-end: a standalone image in a source becomes
|
||||
a Document — bytes digested, the persistent copy in ``image_dir``
|
||||
(``<doc-id>.png``), ``content`` = the (mock) description, and ONLY
|
||||
that text is embedded (the content chunks + the phase-30 summary
|
||||
chunk, 768 dims) — while a text doc in the same source stays a
|
||||
plain text doc."""
|
||||
source_root = tmp_path / "imgsource"
|
||||
source_root.mkdir()
|
||||
(source_root / "diagram.png").write_bytes(PNG_1X1)
|
||||
(source_root / "notes.md").write_text("# Notes\n\nBody text.\n", encoding="utf-8")
|
||||
llm = _image_llm(tmp_path)
|
||||
_patch_description(monkeypatch, "A network diagram of the homelab VLANs.")
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources([source_root], llm, session=db, prune=True)
|
||||
)
|
||||
assert (summary.files, summary.added, summary.images_failed) == (2, 2, 0)
|
||||
assert summary.formats == {"md": 1, "png": 1}
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "diagram.png"
|
||||
)
|
||||
)
|
||||
assert doc is not None, "the image file must become a document"
|
||||
assert doc.is_image is True and doc.image_path is not None
|
||||
assert doc.content == "A network diagram of the homelab VLANs."
|
||||
assert doc.title == "diagram" # the non-markdown stem rule
|
||||
copy = Path(doc.image_path)
|
||||
assert copy.parent == Path(llm.settings.image_dir)
|
||||
assert copy.name == f"{doc.id}.png"
|
||||
assert copy.read_bytes() == PNG_1X1
|
||||
|
||||
# The ONLY embedded text is the description (the embedding model
|
||||
# never sees pixels): every content chunk carries it, the
|
||||
# phase-30 summary chunk exists + is embedded, 768 dims. Task
|
||||
# 03: the description IS the summary (stored verbatim — no
|
||||
# ``lite`` call, no pointer line), so the summary chunk mirrors
|
||||
# ``doc.content`` exactly.
|
||||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [doc.content]
|
||||
assert doc.summary == doc.content
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert summary_chunks[0].content == doc.content
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
|
||||
# The text doc is untouched by the image machinery.
|
||||
md = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "notes.md"
|
||||
)
|
||||
)
|
||||
assert md is not None and md.is_image is False and md.image_path is None
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
class _MockVisionLLM(FakeEmbedder):
|
||||
"""The phase-122 mock VISION client (task 03): the CHAT model
|
||||
answers the multimodal describe call (the image bytes' data URL)
|
||||
with a fixed, retrieval-oriented description; text (``lite``) calls
|
||||
keep the ``FakeEmbedder`` behaviour. The REAL
|
||||
``rag_importer._describe_or_skip`` → ``summarizer.describe_image``
|
||||
chain runs end-to-end against it (no seam patch); every chat
|
||||
call's model is recorded (``chat_models``)."""
|
||||
|
||||
DESCRIPTION = (
|
||||
"A network diagram of the homelab VLANs: the core switch, the "
|
||||
"router, and three labeled subnets."
|
||||
)
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.chat_models: list[str | None] = []
|
||||
|
||||
async def chat(self, messages, model=None):
|
||||
self.chat_calls.append(list(messages))
|
||||
self.chat_models.append(model)
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
# The phase-122 describe call — the multimodal message.
|
||||
return self.DESCRIPTION
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def test_import_sources_mock_vision_client_end_to_end(db, tmp_path) -> None:
|
||||
"""Task 03 end-to-end (NO seam patch): a fixture PNG through the
|
||||
mock vision client — the real ``_describe_or_skip`` →
|
||||
``describe_image`` → CHAT-model call — yields a doc whose
|
||||
``content`` == ``summary`` == the description, with its
|
||||
``is_summary`` chunk embedded, and whose ONLY embedded text is that
|
||||
description (the embedding model never sees pixels)."""
|
||||
source_root = tmp_path / "visione2e"
|
||||
source_root.mkdir()
|
||||
(source_root / "diagram.png").write_bytes(PNG_1X1)
|
||||
llm = _image_llm(tmp_path, _MockVisionLLM)
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources([source_root], llm, session=db)
|
||||
)
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
# The describe call went to the CHAT model (LOCKED A3), once.
|
||||
assert llm.chat_models == [llm.settings.llm_chat_model]
|
||||
|
||||
# The wire shape: the multimodal user message — the fixed
|
||||
# prompt's text part + the image's data-URL part.
|
||||
(message,) = llm.chat_calls[0]
|
||||
assert message["role"] == "user"
|
||||
content: Any = message["content"] # the multimodal part list
|
||||
assert content[0] == {"type": "text", "text": DESCRIBE_PROMPT}
|
||||
assert content[1]["type"] == "image_url"
|
||||
assert content[1]["image_url"]["url"].startswith("data:image/png;base64,")
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "diagram.png"
|
||||
)
|
||||
)
|
||||
assert doc is not None, "the image file must become a document"
|
||||
assert doc.is_image is True and doc.image_path is not None
|
||||
assert doc.content == _MockVisionLLM.DESCRIPTION
|
||||
assert doc.summary == _MockVisionLLM.DESCRIPTION # task 03: verbatim
|
||||
|
||||
# The ONLY embedded text of the doc is the description — twice
|
||||
# (the one content chunk + the is_summary chunk), 768 dims.
|
||||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert all(c.content == _MockVisionLLM.DESCRIPTION for c in chunks)
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
embedded = [t for batch in llm.calls for t in batch]
|
||||
assert embedded == [_MockVisionLLM.DESCRIPTION] * 2
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
class _MockVisionFailsLLM(FakeEmbedder):
|
||||
"""A NON-VISION chat model (LOCKED A3's honest failure): the
|
||||
multimodal describe call raises (the SDK errors — a chat model
|
||||
without vision rejects the ``image_url`` part), text (``lite``)
|
||||
calls keep the ``FakeEmbedder`` behaviour."""
|
||||
|
||||
async def chat(self, messages, model=None):
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
raise LLMError("simulated non-vision chat model (test sentinel)")
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def test_import_sources_failing_vision_skips_image_keeps_sync_green(
|
||||
db, tmp_path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""LOCKED A3 fail-soft end-to-end (the task-06 integration pin):
|
||||
a fixture PNG through a NON-VISION chat model — the real seam, no
|
||||
patch — skips the image doc (``images_failed == 1``, NO row, NO
|
||||
orphan copy — not even the image dir) while the TEXT doc in the
|
||||
same source is indexed as usual (the sync completes, no row
|
||||
mutation anywhere for the failed image)."""
|
||||
source_root = tmp_path / "visionfail"
|
||||
source_root.mkdir()
|
||||
(source_root / "diagram.png").write_bytes(PNG_1X1)
|
||||
(source_root / "notes.md").write_text("# Notes\n\nBody.\n", encoding="utf-8")
|
||||
llm = _image_llm(tmp_path, _MockVisionFailsLLM)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(
|
||||
import_sources([source_root], llm, session=db)
|
||||
)
|
||||
assert (summary.files, summary.added, summary.images_failed) == (2, 1, 1)
|
||||
# The image: no row, no copy (the dir itself was never created).
|
||||
assert (
|
||||
db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "diagram.png"
|
||||
)
|
||||
)
|
||||
is None
|
||||
)
|
||||
assert not Path(llm.settings.image_dir).expanduser().exists()
|
||||
# The text doc indexed as usual (the sync stayed green).
|
||||
md = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "notes.md"
|
||||
)
|
||||
)
|
||||
assert md is not None and md.is_image is False
|
||||
# The importer's warning names the document (the PLAN §9 signal).
|
||||
warnings = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if r.name == "app.importer" and "image description failed" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
assert f"source={source_root.name} path=diagram.png" in warnings[0].getMessage()
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_import_sources_images_off_ignores_and_prune_guard_protects(
|
||||
db, tmp_path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""``images=False`` (the default) end-to-end: the walk ignores the
|
||||
image file entirely (not counted, no row, no copy), and a
|
||||
``prune=True`` run MUST NOT delete a pre-existing image doc — the
|
||||
LOCKED prune guard (invisible to the walk ≠ deleted)."""
|
||||
source_root = tmp_path / "imgsrc_off"
|
||||
source_root.mkdir()
|
||||
(source_root / "diagram.png").write_bytes(PNG_1X1)
|
||||
_patch_description(monkeypatch, "A network diagram.")
|
||||
try:
|
||||
llm_on = _image_llm(tmp_path)
|
||||
s_on = asyncio.run(import_sources([source_root], llm_on, session=db))
|
||||
assert s_on.added == 1
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "diagram.png"
|
||||
)
|
||||
)
|
||||
assert doc is not None
|
||||
copy = Path(doc.image_path)
|
||||
assert copy.exists()
|
||||
|
||||
# Toggle OFF (a fresh fake on the code defaults): the walk is
|
||||
# blind to the file, and prune protects the pre-existing image
|
||||
# doc + its copy.
|
||||
llm_off = FakeEmbedder() # Settings(_env_file=None) → images False
|
||||
s_off = asyncio.run(
|
||||
import_sources([source_root], llm_off, session=db, prune=True)
|
||||
)
|
||||
assert (s_off.files, s_off.added, s_off.pruned) == (0, 0, 0)
|
||||
assert db.scalar(select(Document).where(Document.id == doc.id)) is not None
|
||||
assert copy.exists(), "the copy survives with the doc"
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_import_sources_toggle_on_prunes_deleted_image_with_copy(
|
||||
db, tmp_path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Toggle ON, the image file deleted: the normal prune runs — the
|
||||
doc row AND its ``image_dir`` copy are removed (the copy's
|
||||
lifecycle is tied to the row)."""
|
||||
source_root = tmp_path / "imgsrc_prune"
|
||||
source_root.mkdir()
|
||||
(source_root / "diagram.png").write_bytes(PNG_1X1)
|
||||
llm_on = _image_llm(tmp_path)
|
||||
_patch_description(monkeypatch, "A network diagram.")
|
||||
try:
|
||||
asyncio.run(import_sources([source_root], llm_on, session=db))
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "diagram.png"
|
||||
)
|
||||
)
|
||||
assert doc is not None
|
||||
copy = Path(doc.image_path)
|
||||
assert copy.exists()
|
||||
|
||||
(source_root / "diagram.png").unlink()
|
||||
s = asyncio.run(
|
||||
import_sources([source_root], llm_on, session=db, prune=True)
|
||||
)
|
||||
assert (s.pruned, s.added, s.unchanged) == (1, 0, 0)
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.source == source_root.name))
|
||||
is None
|
||||
)
|
||||
assert not copy.exists(), "the image_dir copy is deleted with the doc"
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 122 (task 04) — the image BYTES route + the content/tree wire
|
||||
# affordance (the serve side of the image-document contract).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_image_doc(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
*,
|
||||
source: str = "ImgSrc",
|
||||
path: str = "pic.png",
|
||||
data: bytes = PNG_1X1,
|
||||
content: str = "A red square on a white background.",
|
||||
image_path: str | None = "auto",
|
||||
doc_id: str | None = None,
|
||||
) -> Document:
|
||||
"""One ``is_image`` document row with its persistent copy (the
|
||||
importer's ``image_dir`` layout) under *tmp_path*; ``image_path``
|
||||
``"auto"`` writes the copy, ``None`` leaves the row without a copy
|
||||
(the lost-copy corner)."""
|
||||
doc = Document(
|
||||
id=uuid.UUID(doc_id) if doc_id else uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{path}",
|
||||
title=path.rsplit(".", 1)[0],
|
||||
content=content,
|
||||
content_hash=hashlib.sha256(data).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||
summary=content,
|
||||
is_image=True,
|
||||
)
|
||||
if image_path == "auto":
|
||||
copy = tmp_path / f"{doc.id}{Path(path).suffix}"
|
||||
copy.write_bytes(data)
|
||||
doc.image_path = str(copy)
|
||||
elif image_path is not None:
|
||||
doc.image_path = image_path
|
||||
db.add(doc)
|
||||
db.flush()
|
||||
db.add_all(
|
||||
[
|
||||
Chunk(
|
||||
document_id=doc.id, position=0, content=content, embedding=[0.01] * 768
|
||||
),
|
||||
Chunk(
|
||||
document_id=doc.id,
|
||||
position=-1,
|
||||
content=content,
|
||||
embedding=[0.01] * 768,
|
||||
is_summary=True,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return doc
|
||||
|
||||
|
||||
def _cleanup_kb(db) -> None:
|
||||
"""Truncate the KB tables. The settle commit matters: SQLAlchemy
|
||||
does NOT autoflush pending ORM objects before a raw ``text()``
|
||||
statement — a pending chunks INSERT flushed *after* the TRUNCATE
|
||||
would FK-violate (the document row is already gone), so any
|
||||
pending state is committed (then truncated) first."""
|
||||
db.commit()
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("ext", "mime"),
|
||||
[
|
||||
(".png", "image/png"),
|
||||
(".jpg", "image/jpeg"),
|
||||
(".jpeg", "image/jpeg"),
|
||||
(".webp", "image/webp"),
|
||||
(".gif", "image/gif"),
|
||||
(".bmp", "image/bmp"),
|
||||
],
|
||||
)
|
||||
def test_image_route_serves_exact_bytes_with_content_type(
|
||||
admin_client: TestClient, db, tmp_path: Path, ext: str, mime: str
|
||||
) -> None:
|
||||
"""The serve contract: the route streams the EXACT stored bytes
|
||||
(a per-extension sentinel — a mix-up between the six formats is
|
||||
caught) with the extension's ``Content-Type`` (the
|
||||
``IMAGE_MIMES`` map — one map, one truth) and
|
||||
``Cache-Control: private, max-age=3600`` (content-hashed bytes —
|
||||
long enough, bustable by re-upload)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
data = PNG_1X1 + ext.encode("ascii") # per-extension sentinel bytes
|
||||
doc = _seed_image_doc(db, tmp_path, path=f"pic{ext}", data=data)
|
||||
r = admin_client.get(f"/api/documents/{doc.id}/image")
|
||||
assert r.status_code == 200
|
||||
assert r.headers["content-type"] == mime # exact type per extension
|
||||
assert r.headers["cache-control"] == "private, max-age=3600"
|
||||
assert r.content == data # the exact uploaded bytes, nothing else
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_image_route_404_matrix(admin_client: TestClient, db, tmp_path: Path) -> None:
|
||||
"""Every non-servable case is 404 ``document not found`` (the
|
||||
router's unknown-document shape — the same detail string the
|
||||
content endpoint uses): a missing id, a MALFORMED id (an unparseable
|
||||
string maps here, not to a 422 — a guessed id is an unknown
|
||||
document), a text doc, an image doc whose ``image_path`` is NULL,
|
||||
and a row whose copy was lost on disk (defensive — the row exists,
|
||||
the bytes don't)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
|
||||
db.commit() # the module's _seed_doc leaves the row uncommitted
|
||||
text_doc_id = db.scalar(
|
||||
select(Document.id).where(
|
||||
Document.source == "TextSrc", Document.path == "note.md"
|
||||
)
|
||||
)
|
||||
no_copy = _seed_image_doc(db, tmp_path, path="nopy.png", image_path=None)
|
||||
lost = _seed_image_doc(db, tmp_path, path="lost.png")
|
||||
assert lost.image_path is not None # the "auto" copy was written
|
||||
Path(lost.image_path).unlink() # the copy is lost (the row remains)
|
||||
|
||||
for doc_id in (
|
||||
str(uuid.uuid4()), # missing id
|
||||
"not-a-uuid", # malformed id → 404, not 422
|
||||
str(text_doc_id), # text doc
|
||||
str(no_copy.id), # image doc, image_path NULL
|
||||
str(lost.id), # image doc, copy lost
|
||||
):
|
||||
r = admin_client.get(f"/api/documents/{doc_id}/image")
|
||||
assert r.status_code == 404, doc_id
|
||||
assert r.json() == {"detail": "document not found"}, doc_id
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_image_route_requires_user_like_the_content_endpoint(
|
||||
admin_client: TestClient, db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 79 posture (the task's "PUBLIC, like the document content
|
||||
endpoint" — the content endpoint has been user-gated since phase
|
||||
79; the ONLY anonymous surface is the shared chats, PLAN A10): an
|
||||
anonymous caller gets 401 ``authentication required`` before any
|
||||
row is read (a FRESH client — the module's fixture ``client``
|
||||
stays unsigned here), a signed-in caller gets the bytes."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
doc = _seed_image_doc(db, tmp_path)
|
||||
anonymous = TestClient(fastapi_app)
|
||||
r = anonymous.get(f"/api/documents/{doc.id}/image")
|
||||
assert r.status_code == 401
|
||||
assert r.json() == {"detail": "authentication required"}
|
||||
# The signed-in client (admin) passes.
|
||||
assert admin_client.get(f"/api/documents/{doc.id}/image").status_code == 200
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_content_endpoint_exposes_the_image_affordance(
|
||||
admin_client: TestClient, db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The content endpoint (the viewer's data source): ``is_image`` is
|
||||
ALWAYS present (text doc: false — the one new key; the wire shape
|
||||
gains nothing else), and ``image_url`` — the bytes route's path —
|
||||
is present for an image doc and ABSENT for a text doc (never null,
|
||||
the ``DocContent`` omission rule)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
|
||||
db.commit() # the app's endpoint session reads committed data only
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "TextSrc", "path": "note.md"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["is_image"] is False
|
||||
assert "image_url" not in body # absent — never null (text doc)
|
||||
|
||||
doc = _seed_image_doc(db, tmp_path, source="ImgSrc", path="pic.png")
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "ImgSrc", "path": "pic.png"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["is_image"] is True
|
||||
assert body["image_url"] == f"/api/documents/{doc.id}/image"
|
||||
assert body["content"] == body["summary"] # the description (task 03)
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def _tree_file_nodes_all(sources) -> list[dict]:
|
||||
"""Every file node of a tree response, walked recursively (all
|
||||
sources — env-registered 0-document sources may join the response
|
||||
when the ``git_sources`` table is truncated, and they carry no
|
||||
file nodes; the assertions below hold over whatever files exist).
|
||||
"""
|
||||
files: list[dict] = []
|
||||
|
||||
def _walk(node: dict) -> None:
|
||||
for child in node.get("children", ()):
|
||||
if child["kind"] == "file":
|
||||
files.append(child)
|
||||
else:
|
||||
_walk(child)
|
||||
|
||||
for source in sources:
|
||||
_walk(source)
|
||||
return files
|
||||
|
||||
|
||||
def test_tree_image_file_node_affordance_and_text_node_byte_identical(
|
||||
admin_client: TestClient, db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The tree (the RAG view's single fetch): an image doc's file node
|
||||
carries the thumbnail affordance (``is_image`` true,
|
||||
``image_url`` = the bytes route's path, ``summary`` verbatim — the
|
||||
RAG view's thumbnail ``alt``); EVERY text file node keeps the
|
||||
pre-phase wire shape byte-identically (the six keys — no
|
||||
``is_image``/``image_url``/``summary`` — the phase's
|
||||
byte-identical criterion: the fields are row-driven, so a KB with
|
||||
no image rows serializes exactly as pre-phase)."""
|
||||
_truncate_tree_tables(db)
|
||||
try:
|
||||
base = datetime.now(UTC)
|
||||
_seed_doc(db, "MixedSrc", "a.md", "A", 1, base)
|
||||
img = _seed_image_doc(db, tmp_path, source="MixedSrc", path="pic.png")
|
||||
|
||||
r = admin_client.get("/api/docs/tree") # both seeds committed (_seed_image_doc)
|
||||
assert r.status_code == 200
|
||||
files = {f["path"]: f for f in _tree_file_nodes_all(r.json()["sources"])}
|
||||
|
||||
# Text node: byte-identical pre-phase wire shape (no image keys).
|
||||
assert set(files["a.md"]) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
|
||||
# Image node: the affordance rides the node.
|
||||
pic = files["pic.png"]
|
||||
assert pic["is_image"] is True
|
||||
assert pic["image_url"] == f"/api/documents/{img.id}/image"
|
||||
assert pic["summary"] == "A red square on a white background."
|
||||
finally:
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
|
||||
def test_tree_with_no_image_rows_is_byte_identical(admin_client: TestClient, db) -> None:
|
||||
"""The pre-phase KB (no ``is_image`` rows): the image-docs map is
|
||||
empty and EVERY file node serializes in the pre-phase shape — the
|
||||
row-driven fields introduce no wire change at all (the phase's
|
||||
byte-identical criterion, the toggle irrelevant)."""
|
||||
_truncate_tree_tables(db)
|
||||
try:
|
||||
base = datetime.now(UTC)
|
||||
_seed_doc(db, "PlainSrc", "x.md", "X", 2, base)
|
||||
db.commit() # the app's endpoint session reads committed data only
|
||||
r = admin_client.get("/api/docs/tree")
|
||||
assert r.status_code == 200
|
||||
files = _tree_file_nodes_all(r.json()["sources"])
|
||||
assert len(files) == 1 # the seeded doc (env sources carry no files)
|
||||
assert set(files[0]) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
finally:
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
@@ -422,11 +422,16 @@ def test_content_200_all_fields(client, db) -> None:
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
# Wire-additive (phase 106, task 05): ``created_at`` joins the
|
||||
# content shape (after ``summary``, before ``content``).
|
||||
# content shape (after ``summary``, before ``content``) — and
|
||||
# (phase 122, task 04) ``is_image`` joins it ALWAYS present
|
||||
# (text docs: false); ``image_url`` is ABSENT for a text doc
|
||||
# (never null — the ``DocContent`` omission rule).
|
||||
assert set(body) == {
|
||||
"source", "path", "title", "format", "summary", "created_at",
|
||||
"content", "indexed_at", "chunks",
|
||||
"content", "indexed_at", "chunks", "is_image",
|
||||
}
|
||||
assert body["is_image"] is False
|
||||
assert "image_url" not in body # absent — never null (text doc)
|
||||
datetime.fromisoformat(body["created_at"]) # raises if not ISO-8601
|
||||
assert body["source"] == "Homelab"
|
||||
assert body["path"] == "kubernetes.md"
|
||||
|
||||
@@ -0,0 +1,344 @@
|
||||
"""Integration: migration 0022 (documents.is_image + image_path) schema
|
||||
contract (phase 122, task 02).
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0021.py`` (information_schema assertions on the state
|
||||
the migration must leave). The tests target the 0021 → 0022 step
|
||||
explicitly so later migrations cannot break the pins:
|
||||
|
||||
* upgrade 0021 → 0022 → ``is_image`` exists with the full contract —
|
||||
BOOLEAN, NOT NULL, server default ``false`` — and ``image_path`` —
|
||||
TEXT, NULLABLE, no server default — while the 0021 ``documents``
|
||||
schema (``content``/``content_hash`` NOT NULL, ``summary`` NULLABLE,
|
||||
``created_at`` + ``created_at_manual``, the (source, path) unique
|
||||
constraint — asserted column-based, since the suite's table
|
||||
self-heal renames copied constraints) survives;
|
||||
* a ``documents`` row inserted while the DB is at 0021 backfills
|
||||
``is_image`` to ``false`` and ``image_path`` to NULL (every
|
||||
pre-phase-122 row is a text doc — the LOCKED A3 default);
|
||||
* the ORM contract agrees: a freshly inserted ``Document`` without the
|
||||
image fields reads ``is_image is False`` / ``image_path is None``,
|
||||
and one with them round-trips through a fresh session;
|
||||
* downgrade to 0021 → both columns are GONE (A13) while the row
|
||||
survives; upgrade back to 0022 → the columns are back (round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import Document
|
||||
|
||||
SOURCE = "mig0022"
|
||||
PATH_TEXT = "notes/readme.md"
|
||||
PATH_IMAGE = "notes/diagram.png"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown
|
||||
upgrades to head no matter what happened, so the dev DB is never
|
||||
left below head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
|
||||
# on ``documents``, which would deadlock the repair's
|
||||
# ``ALTER TABLE`` (0022) forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default) for one documents
|
||||
column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'documents' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _insert_sql(db: Session, path: str) -> uuid.UUID:
|
||||
"""Insert one documents row with the PRE-0022 column set (the 0021
|
||||
shape — the image columns, when present, are omitted so their
|
||||
backfill is what the row reads)."""
|
||||
row_id = uuid.uuid4()
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO documents (id, source, path, full_path, title,"
|
||||
" content, content_hash)"
|
||||
" VALUES (:id, :s, :p, :f, :t, :c, :h)"
|
||||
),
|
||||
{
|
||||
"id": row_id,
|
||||
"s": SOURCE,
|
||||
"p": path,
|
||||
"f": f"/tmp/{SOURCE}/{path}",
|
||||
"t": path.rsplit("/", 1)[-1],
|
||||
"c": "content",
|
||||
"h": "0" * 64,
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return row_id
|
||||
|
||||
|
||||
def _delete(db: Session, row_id: uuid.UUID) -> None:
|
||||
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": row_id})
|
||||
db.commit()
|
||||
|
||||
|
||||
def _unique_column_sets(db: Session) -> set[tuple[str, ...]]:
|
||||
"""The column tuples of every UNIQUE constraint on ``documents``.
|
||||
|
||||
Column-based (not name-based): the integration suite's table
|
||||
self-heal (``tests/integration/conftest.py``) rewrites bloated
|
||||
tables via ``CREATE TABLE (LIKE …)``, which renames copied
|
||||
constraints (PG auto-names them) — the (source, path) uniqueness
|
||||
contract is what must hold, not the original name.
|
||||
"""
|
||||
rows = db.execute(
|
||||
text(
|
||||
"SELECT (SELECT string_agg(a.attname, ',' ORDER BY k.ord)"
|
||||
" FROM unnest(c.conkey) WITH ORDINALITY k(attnum, ord)"
|
||||
" JOIN pg_attribute a"
|
||||
" ON a.attrelid = c.conrelid AND a.attnum = k.attnum)"
|
||||
" FROM pg_constraint c"
|
||||
" WHERE c.contype = 'u' AND c.conrelid = 'documents'::regclass"
|
||||
)
|
||||
).fetchall()
|
||||
return {tuple(r[0].split(",")) for r in rows}
|
||||
|
||||
|
||||
def test_upgrade_to_0022_adds_image_columns(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0021 → 0022: ``is_image`` exists with the full contract
|
||||
(BOOLEAN, NOT NULL, server default ``false`` — every pre-phase-122
|
||||
row is a text doc) and ``image_path`` (TEXT, NULLABLE, no server
|
||||
default — NULL for text docs), both ABSENT at 0021; a pre-0022 row
|
||||
backfills ``is_image`` to ``false`` + ``image_path`` to NULL; and
|
||||
the 0021 table contract survives the additive upgrade."""
|
||||
command.downgrade(alembic, "0021") # start from the pre-0022 state
|
||||
assert _version(db) == "0021"
|
||||
assert _column(db, "is_image") is None, "is_image must be absent at 0021"
|
||||
assert _column(db, "image_path") is None, "image_path must be absent at 0021"
|
||||
|
||||
pre_id = _insert_sql(db, PATH_TEXT) # the 0021 column set
|
||||
try:
|
||||
command.upgrade(alembic, "0022")
|
||||
assert _version(db) == "0022", "alembic_version must be at 0022"
|
||||
|
||||
is_image = _column(db, "is_image")
|
||||
assert is_image is not None, "documents.is_image is missing"
|
||||
assert is_image[0] == "boolean", "is_image must be BOOLEAN"
|
||||
assert is_image[1] == "NO", "is_image must be NOT NULL"
|
||||
assert is_image[2] == "false", (
|
||||
"is_image must carry the `false` server default — every"
|
||||
" pre-phase-122 row is a text doc"
|
||||
)
|
||||
|
||||
image_path = _column(db, "image_path")
|
||||
assert image_path is not None, "documents.image_path is missing"
|
||||
assert image_path[0] == "text", "image_path must be TEXT"
|
||||
assert image_path[1] == "YES", "image_path must be NULLABLE"
|
||||
assert image_path[2] is None, (
|
||||
"image_path must carry NO server default — NULL is the"
|
||||
" text-doc value"
|
||||
)
|
||||
|
||||
# The pre-0022 row backfilled to (false, NULL) — a text doc.
|
||||
row = db.execute(
|
||||
text("SELECT is_image, image_path FROM documents WHERE id = :id"),
|
||||
{"id": pre_id},
|
||||
).fetchone()
|
||||
assert row is not None, "the pre-0022 row must survive the upgrade"
|
||||
assert row[0] is False, "the backfilled is_image must be false"
|
||||
assert row[1] is None, "the backfilled image_path must be NULL"
|
||||
|
||||
# A row written without the image columns reads the same
|
||||
# (the Python-side defaults are False/None — same values).
|
||||
new_id = _insert_sql(db, PATH_IMAGE)
|
||||
try:
|
||||
backfilled = db.execute(
|
||||
text("SELECT is_image, image_path FROM documents WHERE id = :id"),
|
||||
{"id": new_id},
|
||||
).fetchone()
|
||||
assert backfilled == (False, None), (
|
||||
"an omitted image state must read (false, NULL)"
|
||||
)
|
||||
finally:
|
||||
_delete(db, new_id)
|
||||
|
||||
# The 0021 schema survives the additive upgrade.
|
||||
content = _column(db, "content")
|
||||
assert content is not None and content[0] == "text" and content[1] == "NO", (
|
||||
"documents.content (0001) must keep its 0021 contract"
|
||||
)
|
||||
hash_col = _column(db, "content_hash")
|
||||
assert (
|
||||
hash_col is not None
|
||||
and hash_col[0] == "character varying"
|
||||
and hash_col[1] == "NO"
|
||||
), "documents.content_hash (0001) must survive the upgrade"
|
||||
summary = _column(db, "summary")
|
||||
assert summary is not None and summary[0] == "text" and summary[1] == "YES", (
|
||||
"documents.summary (phase 30) must survive the upgrade"
|
||||
)
|
||||
created = _column(db, "created_at")
|
||||
assert created is not None and created[0] == "timestamp with time zone"
|
||||
assert created[1] == "NO" and "now()" in str(created[2]), (
|
||||
"documents.created_at (0020) must keep its `now()` server default"
|
||||
)
|
||||
manual = _column(db, "created_at_manual")
|
||||
assert manual is not None and manual[0] == "boolean" and manual[1] == "NO"
|
||||
assert manual[2] == "false", (
|
||||
"documents.created_at_manual (0020) must keep its `false` default"
|
||||
)
|
||||
assert ("source", "path") in _unique_column_sets(db), (
|
||||
"the (source, path) unique constraint must survive the upgrade"
|
||||
)
|
||||
finally:
|
||||
_delete(db, pre_id)
|
||||
|
||||
|
||||
def test_orm_image_fields_round_trip(db: Session, alembic: Config) -> None:
|
||||
"""The ORM contract agrees with the column contract: a freshly
|
||||
inserted ``Document`` WITHOUT the image fields reads ``is_image is
|
||||
False`` / ``image_path is None`` (the text-doc default state), and
|
||||
one WITH them round-trips the pair through a FRESH session."""
|
||||
command.upgrade(alembic, "head")
|
||||
text_doc = Document(
|
||||
source=SOURCE,
|
||||
path=PATH_TEXT,
|
||||
full_path=f"/tmp/{SOURCE}/{PATH_TEXT}",
|
||||
title="readme",
|
||||
content="# readme\n",
|
||||
content_hash="1" * 64,
|
||||
)
|
||||
image_doc = Document(
|
||||
source=SOURCE,
|
||||
path=PATH_IMAGE,
|
||||
full_path=f"/tmp/{SOURCE}/{PATH_IMAGE}",
|
||||
title="diagram",
|
||||
content="A description of the diagram.",
|
||||
content_hash="2" * 64,
|
||||
is_image=True,
|
||||
image_path="/srv/bor-images/diagram.png",
|
||||
)
|
||||
db.add(text_doc)
|
||||
db.add(image_doc)
|
||||
db.commit()
|
||||
try:
|
||||
with SessionLocal() as fresh:
|
||||
reloaded_text = fresh.get(Document, text_doc.id)
|
||||
assert reloaded_text is not None, "the text row must be readable"
|
||||
assert reloaded_text.is_image is False, (
|
||||
"an omitted is_image must read the False default"
|
||||
)
|
||||
assert reloaded_text.image_path is None, (
|
||||
"an omitted image_path must read NULL"
|
||||
)
|
||||
reloaded_image = fresh.get(Document, image_doc.id)
|
||||
assert reloaded_image is not None, "the image row must be readable"
|
||||
assert reloaded_image.is_image is True
|
||||
assert reloaded_image.image_path == "/srv/bor-images/diagram.png"
|
||||
finally:
|
||||
_delete(db, text_doc.id)
|
||||
_delete(db, image_doc.id)
|
||||
|
||||
|
||||
def test_downgrade_to_0021_drops_the_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0022 → 0021: both image columns are gone (A13 — fully
|
||||
reversible) while the row + its 0021 columns survive, and the rest
|
||||
of the 0021 table contract (``content``, ``content_hash``,
|
||||
``created_at``) is intact."""
|
||||
command.upgrade(alembic, "head")
|
||||
row = Document(
|
||||
source=SOURCE,
|
||||
path=PATH_IMAGE,
|
||||
full_path=f"/tmp/{SOURCE}/{PATH_IMAGE}",
|
||||
title="diagram",
|
||||
content="A description of the diagram.",
|
||||
content_hash="3" * 64,
|
||||
is_image=True,
|
||||
image_path="/srv/bor-images/diagram.png",
|
||||
)
|
||||
db.add(row)
|
||||
db.commit()
|
||||
try:
|
||||
command.downgrade(alembic, "0021")
|
||||
assert _version(db) == "0021"
|
||||
assert _column(db, "is_image") is None, "is_image must be dropped"
|
||||
assert _column(db, "image_path") is None, "image_path must be dropped"
|
||||
|
||||
surviving = db.execute(
|
||||
text(
|
||||
"SELECT source, path, title, content, content_hash, created_at"
|
||||
" FROM documents WHERE id = :id"
|
||||
),
|
||||
{"id": row.id},
|
||||
).fetchone()
|
||||
assert surviving is not None, "the row must survive the column drops"
|
||||
assert surviving[0] == SOURCE and surviving[1] == PATH_IMAGE
|
||||
assert surviving[3] == "A description of the diagram."
|
||||
assert surviving[4] == "3" * 64
|
||||
assert surviving[5] is not None, "created_at must survive the drops"
|
||||
|
||||
assert ("source", "path") in _unique_column_sets(db), (
|
||||
"the (source, path) unique constraint must survive the downgrade"
|
||||
)
|
||||
finally:
|
||||
_delete(db, row.id)
|
||||
# Repair: the fixture teardown re-upgrades to head.
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_the_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0021, then upgrade back to 0022: both columns are
|
||||
back with the full contract (``is_image`` BOOLEAN NOT NULL default
|
||||
``false``; ``image_path`` TEXT NULLABLE no default)."""
|
||||
command.downgrade(alembic, "0021")
|
||||
command.upgrade(alembic, "0022")
|
||||
assert _version(db) == "0022", "round-trip upgrade must land at 0022"
|
||||
|
||||
is_image = _column(db, "is_image")
|
||||
assert is_image is not None, "documents.is_image must be back"
|
||||
assert is_image[0] == "boolean", "is_image must be BOOLEAN after the round-trip"
|
||||
assert is_image[1] == "NO", "is_image must be NOT NULL after the round-trip"
|
||||
assert is_image[2] == "false", (
|
||||
"is_image must still carry the `false` server default"
|
||||
)
|
||||
|
||||
image_path = _column(db, "image_path")
|
||||
assert image_path is not None, "documents.image_path must be back"
|
||||
assert image_path[0] == "text"
|
||||
assert image_path[1] == "YES"
|
||||
assert image_path[2] is None, "image_path must still carry NO server default"
|
||||
@@ -216,10 +216,11 @@ def test_admin_semantic_fields_round_trip(client: TestClient, db: Session) -> No
|
||||
|
||||
|
||||
def _config_keys() -> set[str]:
|
||||
"""The /api/config key set after task 03: the five phase-39/59/62
|
||||
keys — the retired CSS-file theming's ``theme`` key is gone."""
|
||||
"""The /api/config key set after task 03 (phase 91): the retired
|
||||
CSS-file theming's ``theme`` key is gone; phase 122 (task 01) added
|
||||
the ``images`` flag — the six keys below are the contract."""
|
||||
return {"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text"}
|
||||
"images", "input_placeholder", "footer_text"}
|
||||
|
||||
|
||||
def test_api_config_env_only_deployment_returns_env_strings(client: TestClient) -> None:
|
||||
|
||||
@@ -179,6 +179,7 @@ def test_content_known_pair_maps_to_doc_content() -> None:
|
||||
title="Deep Mark",
|
||||
content="# Deep Mark\n\nbody",
|
||||
content_hash="f" * 64,
|
||||
is_image=False, # phase 122: the stub session never applies defaults
|
||||
)
|
||||
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
|
||||
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC) # phase 106
|
||||
@@ -190,11 +191,15 @@ def test_content_known_pair_maps_to_doc_content() -> None:
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
# Wire-additive (phase 106, task 05): the pre-date keys are all
|
||||
# still there, joined by ``created_at``.
|
||||
# still there, joined by ``created_at`` — and (phase 122, task 04)
|
||||
# by ``is_image`` (ALWAYS present; text docs: false) while
|
||||
# ``image_url`` is ABSENT (never null — the omission rule).
|
||||
assert set(body) == {
|
||||
"source", "path", "title", "format", "summary", "created_at",
|
||||
"content", "indexed_at", "chunks",
|
||||
"content", "indexed_at", "chunks", "is_image",
|
||||
}
|
||||
assert body["is_image"] is False
|
||||
assert "image_url" not in body # absent — never null (text doc)
|
||||
assert body["source"] == "Homelab"
|
||||
assert body["path"] == "notes/deep mark.md"
|
||||
assert body["title"] == "Deep Mark"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -970,12 +970,14 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
"""PLAN §9 summary line: the phase-30 counters sit between
|
||||
``embed_batches`` and ``formats``; the phase-118 backfill counter
|
||||
sits between ``summary_errors`` and ``dates_updated``; the
|
||||
phase-106 date-refresh counter sits before ``formats``."""
|
||||
phase-106 date-refresh counter and the phase-122 ``images_failed``
|
||||
counter sit after ``dates_updated``, before ``formats``."""
|
||||
s = ImportSummary()
|
||||
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
|
||||
s.summaries, s.summary_errors = 2, 1
|
||||
s.summary_backfilled = 1
|
||||
s.dates_updated = 0
|
||||
s.images_failed = 0
|
||||
s.formats = {"md": 1, "yaml": 2}
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
s.log()
|
||||
@@ -983,7 +985,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
assert line == (
|
||||
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 formats=yaml:2,md:1"
|
||||
"dates_updated=0 images_failed=0 formats=yaml:2,md:1"
|
||||
)
|
||||
|
||||
|
||||
@@ -1103,13 +1105,17 @@ def test_no_progress_means_no_prewalk(
|
||||
excluded: frozenset[str] = EXCLUDED_DIRS,
|
||||
ignore: tuple[str, ...] = (),
|
||||
include_hidden: bool = False,
|
||||
image_extensions: frozenset[str] = frozenset(),
|
||||
) -> list[Path]:
|
||||
# Phase 89: the walker gained the ``ignore`` keyword; phase 105:
|
||||
# the ``include_hidden`` flag — the sentinel accepts (and forwards)
|
||||
# both to stay a drop-in.
|
||||
# the ``include_hidden`` flag; phase 122: the ``image_extensions``
|
||||
# set — the sentinel accepts (and forwards) all three to stay a
|
||||
# drop-in.
|
||||
nonlocal walk_calls
|
||||
walk_calls += 1
|
||||
return real_walker(r, extensions, excluded, ignore, include_hidden)
|
||||
return real_walker(
|
||||
r, extensions, excluded, ignore, include_hidden, image_extensions
|
||||
)
|
||||
|
||||
monkeypatch.setattr(importer, "iter_importable_files", counting)
|
||||
try:
|
||||
|
||||
@@ -169,14 +169,21 @@ def test_summaries_present_and_absent() -> None:
|
||||
one, two = _folder_nodes(source)
|
||||
assert one.summary == "One desc."
|
||||
assert two.summary is None
|
||||
# File nodes carry no summary key at all (the 00_phase.md shape);
|
||||
# since phase 106 they DO carry the creation date (``created_at``
|
||||
# — the RAG view's ``Created`` column).
|
||||
# File nodes carry no summary key at all on the WIRE (the
|
||||
# 00_phase.md shape — the model_dump set check below is the wire
|
||||
# pin); since phase 106 they DO carry the creation date
|
||||
# (``created_at`` — the RAG view's ``Created`` column).
|
||||
file = _file_nodes(one)[0]
|
||||
assert set(file.model_dump()) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
assert "summary" not in file.__class__.model_fields
|
||||
# Phase 122 (task 04): the image-affordance fields DO exist on the
|
||||
# class now (image file nodes set them — the wire omission for text
|
||||
# nodes is the ``KbTreeFile`` serializer, pinned by the model_dump
|
||||
# set check above: a text node never leaks the three keys).
|
||||
from app.schemas import KbTreeFile
|
||||
|
||||
assert {"is_image", "image_url", "summary"} <= set(KbTreeFile.model_fields)
|
||||
|
||||
|
||||
def test_file_metadata_unchanged_in_tree() -> None:
|
||||
@@ -546,3 +553,83 @@ def test_updated_at_does_not_leak_across_sources() -> None:
|
||||
a, b = build_kb_tree(["A", "B"], rows, {})
|
||||
assert a.updated_at == C0
|
||||
assert b.updated_at == C3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 122 (task 04) — the image-docs affordance on tree file nodes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
DOC_ID = "11111111-2222-3333-4444-555555555555"
|
||||
|
||||
|
||||
def test_image_file_node_carries_the_affordance_and_text_node_unchanged() -> None:
|
||||
"""The ``images`` map (``{(source, path): (doc_id, summary)}``) turns
|
||||
a file node into the image-docs node: ``is_image`` true,
|
||||
``image_url`` = the bytes route's path built from the mapped id,
|
||||
``summary`` verbatim (the RAG view's thumbnail ``alt``). A file
|
||||
node NOT in the map keeps the pre-phase wire shape byte-identically
|
||||
(the three image keys are OMITTED, not false/null)."""
|
||||
rows = [
|
||||
("S", "one/a.md", "A", 1, T0, C0),
|
||||
("S", "one/pic.png", "pic", 2, T0, C0),
|
||||
]
|
||||
images = {("S", "one/pic.png"): (DOC_ID, "A red square on a white background.")}
|
||||
(source,) = build_kb_tree(["S"], rows, {}, images)
|
||||
one = _folder_nodes(source)[0]
|
||||
# File nodes keep the FULL source-relative path (the phase-97
|
||||
# shape — the folder prefix rides the node).
|
||||
files = {f.path: f for f in _file_nodes(one)}
|
||||
|
||||
# Text node: the pre-phase wire shape, byte-identical (no image keys).
|
||||
assert set(files["one/a.md"].model_dump()) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
|
||||
# Image node: the affordance rides the node.
|
||||
dumped = files["one/pic.png"].model_dump()
|
||||
assert dumped["kind"] == "file"
|
||||
assert dumped["is_image"] is True
|
||||
assert dumped["image_url"] == f"/api/documents/{DOC_ID}/image"
|
||||
assert dumped["summary"] == "A red square on a white background."
|
||||
# The catalogue fields still ride verbatim.
|
||||
assert (dumped["title"], dumped["chunks"]) == ("pic", 2)
|
||||
assert (dumped["created_at"], dumped["indexed_at"]) == (C0, T0)
|
||||
|
||||
|
||||
def test_image_file_node_null_summary_and_missing_map_entry() -> None:
|
||||
"""A mapped image node with a NULL summary (the fail-soft backfill
|
||||
corner) keeps ``summary: null`` on the wire (meaningful — the alt
|
||||
falls back client-side). A map entry that points at a NON-existent
|
||||
(source, path) affects nothing (the builder only reads mapped keys
|
||||
it meets in the catalogue rows)."""
|
||||
rows = [
|
||||
("S", "one/ghost.png", "ghost", 1, T0, C0),
|
||||
("S", "one/other.md", "Other", 1, T0, C0),
|
||||
]
|
||||
images = {
|
||||
("S", "one/ghost.png"): (DOC_ID, None),
|
||||
("S", "one/absent.png"): (DOC_ID, "never matched"),
|
||||
}
|
||||
(source,) = build_kb_tree(["S"], rows, {}, images)
|
||||
one = _folder_nodes(source)[0]
|
||||
files = {f.path: f for f in _file_nodes(one)}
|
||||
ghost = files["one/ghost.png"].model_dump()
|
||||
assert ghost["is_image"] is True
|
||||
assert ghost["summary"] is None # null stays (the alt fallback corner)
|
||||
assert ghost["image_url"] == f"/api/documents/{DOC_ID}/image"
|
||||
assert set(files["one/other.md"].model_dump()) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
|
||||
|
||||
def test_image_affordance_absent_without_the_map() -> None:
|
||||
"""No map (the default) → every file node is the pre-phase shape,
|
||||
even for ``.png`` paths: the fields are map-driven (the endpoint
|
||||
composes the map from the ``is_image`` rows), never path-guessed —
|
||||
a pre-phase KB serializes byte-identically."""
|
||||
rows = [("S", "pic.png", "pic", 1, T0, C0)]
|
||||
(source,) = build_kb_tree(["S"], rows, {})
|
||||
(file,) = _file_nodes(source)
|
||||
assert set(file.model_dump()) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
|
||||
@@ -53,18 +53,23 @@ def test_app_config_dict_carries_the_docs_flag() -> None:
|
||||
body = app_config(s)
|
||||
# Phase 62 (task 01): the response grew to the phase-62 UI
|
||||
# customization keys; phase 91 (task 03) deleted the retired
|
||||
# CSS-file theming's ``theme`` key — the five keys below are the
|
||||
# entire endpoint contract.
|
||||
# CSS-file theming's ``theme`` key; phase 122 (task 01) added the
|
||||
# ``images`` flag — the six keys below are the entire endpoint
|
||||
# contract.
|
||||
assert set(body) == {
|
||||
"app_name", "version", "docs_repo_configured",
|
||||
"input_placeholder", "footer_text",
|
||||
"images", "input_placeholder", "footer_text",
|
||||
}
|
||||
assert body["docs_repo_configured"] is s.docs_configured
|
||||
assert body["docs_repo_configured"] is False
|
||||
assert body["images"] is False # LOCKED A3: off by default
|
||||
|
||||
s2 = _settings(docs_repo="/srv/docs-repo")
|
||||
assert app_config(s2)["docs_repo_configured"] is True
|
||||
|
||||
s3 = _settings(images=True)
|
||||
assert app_config(s3)["images"] is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# brand.js — the flag + promise are surfaced the way app_name is
|
||||
|
||||
@@ -114,23 +114,25 @@ def test_no_literal_46rem_width_remains() -> None:
|
||||
# ---------- the four reading-column selectors ----------
|
||||
|
||||
|
||||
def test_the_four_reading_columns_use_the_token() -> None:
|
||||
""".chat-shell, .shared-shell, .doc-md and
|
||||
.doc-summary:has(+ .doc-md) each cap with
|
||||
max-width: var(--chat-column) — and exactly those four rules use
|
||||
the token for a max-width (no other selector)."""
|
||||
def test_the_reading_columns_use_the_token() -> None:
|
||||
""".chat-shell, .shared-shell, .doc-md, .doc-image (phase 122,
|
||||
task 04 — the viewer's image block rides the SAME reading column
|
||||
as its .doc-md sibling) and .doc-summary:has(+ .doc-md) each cap
|
||||
with max-width: var(--chat-column) — and exactly those five rules
|
||||
use the token for a max-width (no other selector)."""
|
||||
css = _css()
|
||||
for selector in (
|
||||
".chat-shell",
|
||||
".shared-shell",
|
||||
".doc-md",
|
||||
".doc-image",
|
||||
".doc-summary:has(+ .doc-md)",
|
||||
):
|
||||
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
|
||||
f"{selector} must cap with max-width: var(--chat-column)"
|
||||
)
|
||||
assert css.count("max-width: var(--chat-column)") == 4, (
|
||||
"exactly the four reading-column selectors use the token"
|
||||
assert css.count("max-width: var(--chat-column)") == 5, (
|
||||
"exactly the reading-column selectors use the token"
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user