phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s

**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
This commit is contained in:
2026-09-25 01:54:23 -04:00
parent 0f77e9a876
commit a19d78d284
63 changed files with 5484 additions and 111 deletions
+9
View File
@@ -69,6 +69,15 @@ os.environ["BOR_RECENCY_HALF_LIFE_DAYS"] = str(
_Settings.model_fields["recency_half_life_days"].default
)
# Phase 122 (task 01): the same leak class for the image-document
# toggle — an operator's local ``.env`` may legitimately carry
# ``BOR_IMAGES``/``BOR_IMAGE_EXTENSIONS``/``BOR_IMAGE_DIR``, and the
# "off by default" pins (the ``/api/config`` ``images`` flag, the
# walk's images-off behavior) must see the code defaults.
os.environ["BOR_IMAGES"] = str(_Settings.model_fields["images"].default)
os.environ["BOR_IMAGE_EXTENSIONS"] = str(_Settings.model_fields["image_extensions"].default)
os.environ["BOR_IMAGE_DIR"] = str(_Settings.model_fields["image_dir"].default)
from app.db import SessionLocal, db_available # noqa: E402
from app.main import app as fastapi_app # noqa: E402
+73 -5
View File
@@ -26,6 +26,14 @@ Implements just enough of the aipi surface:
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
marker as a substring, so the summary branch would otherwise
shadow every folder-summary call
- ``IMAGE_DESCRIPTION_MODE`` (phase 122, image documents) -> the
fixed ``IMAGE_DESCRIPTION_ANSWER`` description: the CHAT model's
(vision) reply to ``summarizer.describe_image``'s multimodal user
message (the marker is its first text part; the call is
non-streaming, so only this answer path serves it). The reply is
the image document's whole content + summary, so it is byte-
stable and token-dense (its cosine against the story suite's
question clears the mock-calibrated gate)
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
- otherwise -> upbeat answer quoting the provided document context
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
@@ -592,19 +600,46 @@ def _messages(body: dict[str, Any]) -> list[dict[str, str]]:
return body.get("messages", [])
def _content_text(content: Any) -> str:
"""The TEXT of one message's content (phase 122 list safety).
String content passes through byte-identical (as does an absent
value or an explicit ``None`` — the ``or ""`` semantics the
:func:`_context` docstring pins). A multimodal part LIST — the
phase-122 image description's ``[{type: "text", …},
{type: "image_url", …}]``, the only list-content message the app
produces (``app.rag.summarizer.describe_image``) — contributes its
text parts joined (the ``image_url`` part carries no text): the
trigger checks and marker branches read the text, and no existing
string-content request is affected.
"""
if isinstance(content, list):
return " ".join(
part.get("text", "")
for part in content
if isinstance(part, dict) and part.get("type") == "text"
)
return content or ""
def _system(body: dict[str, Any]) -> str:
return " ".join(m.get("content", "") for m in _messages(body) if m.get("role") == "system")
return " ".join(
_content_text(m.get("content"))
for m in _messages(body)
if m.get("role") == "system"
)
def _user(body: dict[str, Any]) -> str:
parts = [m.get("content", "") for m in _messages(body) if m.get("role") == "user"]
parts = [_content_text(m.get("content")) for m in _messages(body) if m.get("role") == "user"]
return parts[-1] if parts else ""
def _context(body: dict[str, Any]) -> str:
"""The document context is the longest system/user message in practice.
``m.get("content") or ""`` (NOT ``m.get("content", "")``): a well-formed
``_content_text(m.get("content"))`` (the ``or ""`` semantics, NOT
``m.get("content", "")``): a well-formed
OpenAI tool-call message carries ``content: None`` EXPLICITLY (the app's
agent loop appends exactly that — ``app/rag/agent.py``), and a forced
final answer after a tool round (the round-cap path) reaches this helper
@@ -612,9 +647,12 @@ def _context(body: dict[str, Any]) -> str:
an explicit ``None`` and crashes ``len()`` with a 500 (phase 93 task 04
caught it via the deterministic single-read flow's ALREADY_IN_CONTEXT
loop); ``or ""`` treats absent and explicit-None alike, so the fallback
composes deterministically instead of traceback-ing."""
composes deterministically instead of traceback-ing. Phase 122: a
multimodal part list maps to its text parts (see
:func:`_content_text`), so ``len()``/slicing never meet a list.
"""
msgs = _messages(body)
return max((m.get("content") or "" for m in msgs), key=len)
return max((_content_text(m.get("content")) for m in msgs), key=len)
LONG_ANSWER_TRIGGER = "write a long answer"
@@ -739,6 +777,25 @@ HISTORY_TRIGGER = "echo my history"
#: phrase, so every other suite is unaffected.
FOLDER_MAP_TRIGGER = "repeat your folder map"
#: Phase 122 (image documents, LOCKED A3): the fixed description the
#: mock's vision (CHAT) model returns for an ``IMAGE_DESCRIPTION_MODE``
#: request — the multimodal user message
#: ``[{type: "text", …IMAGE_DESCRIPTION_MODE…}, {type: "image_url",
#: …}]`` (``app.rag.summarizer.describe_image``, the app's only
#: list-content message). It becomes the image document's WHOLE
#: ``content`` AND ``summary`` (the only embedded text — the embedding
#: model never sees pixels), so the text is byte-stable across runs and
#: token-dense: the story suite's question ("What is shown in the
#: homelab network diagram?", cosine ≈0.38 against the mock's
#: token-overlap embeddings) clears the mock-calibrated gate (0.30)
#: and the citation usefulness floor (0.15).
IMAGE_DESCRIPTION_ANSWER = (
"A network diagram of the homelab server room: a core router on top, "
"a core switch in the middle, and three labeled subnets at the bottom — "
"VLAN 10 office, VLAN 20 lab, and VLAN 30 storage — with a legend of "
"cable runs. Title: Homelab Network Map."
)
TABLE_ANSWER = (
"Here's the shape, in a table:\n"
"\n"
@@ -2178,6 +2235,17 @@ def compose_answer(body: dict[str, Any]) -> str:
answer = "Knowledge base outline:\n- " + " ".join(
TOKEN_RE.findall(user.lower())[:8]
)
elif "IMAGE_DESCRIPTION_MODE" in user:
# Phase 122 (image documents, LOCKED A3): the CHAT model's
# (vision) image description — a NON-STREAMING multimodal user
# message, so only this path ever sees it. ``_content_text``
# joins the text parts, so the marker (the first text part)
# lands in ``user``. Checked before the user-trigger branches:
# the marker is a fixed app constant, and a real question is
# never expected to type it (the other user triggers are owner
# phrasings a question might legitimately contain — this one
# is an app-to-app marker).
answer = IMAGE_DESCRIPTION_ANSWER
elif TABLE_TRIGGER in user.lower():
# Markdown tables (phase 44, TODO.md L6): the story E2E's
# deterministic table answer — a 3-column table, the
+10 -8
View File
@@ -149,27 +149,29 @@ def test_api_config_serves_both_names(testy_server: str, app_server: str) -> Non
# "Save as doc" gating; both instances run with BOR_DOCS_REPO
# empty, so it is the inert false here. Phase 62 (task 01): the
# endpoint grew with the UI-customization keys; phase 91
# (task 03) deleted the retired CSS-file theming's ``theme`` key —
# the five keys below are the entire contract (this suite's
# instances carry no UI-customization overrides, so the string
# keys are their defaults).
# (task 03) deleted the retired CSS-file theming's ``theme`` key;
# phase 122 (task 01) added the ``images`` flag (inert false here
# — neither instance sets BOR_IMAGES) — the six keys below are
# the entire contract (this suite's instances carry no
# UI-customization overrides, so the string keys are their
# defaults).
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["app_name"] == TESTY_NAME
assert body["docs_repo_configured"] is False
assert body["images"] is False
# The shared conftest instance keeps the default (the other
# suites' title/label contract rides on it) — and its key set
# tracks the endpoint contract (five keys after phase 91,
# task 03).
# tracks the endpoint contract (six keys after phase 122, task 01).
r2 = httpx.get(f"{app_server}/api/config", timeout=5)
assert r2.status_code == 200
r2_body = r2.json()
assert set(r2_body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert r2_body["app_name"] == DEFAULT_NAME
+499
View File
@@ -0,0 +1,499 @@
"""Phase 122 E2E (Playwright): standalone image documents — the user path.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_image_documents.py -v --no-cov
The suite's app instance runs with ``BOR_IMAGES=true`` (module env
override — the conftest per-suite-app pattern, leak guards included):
the standalone PNG arrives through the REAL admin upload flow (a zip
with a single image member — phase 90's no-scan contract: the upload
unpacks + registers, the RAG page's "Sync sources" button scans), the
mock's vision model (``IMAGE_DESCRIPTION_MODE`` branch —
``tests/e2e/mock_llm.py``) writes the description, and the document is
first-class at every surface:
* the Sources page lists it with the 48px thumbnail (the image bytes
route, alt = the summary);
* the document viewer renders the image with the description below it
(the summary panel is suppressed — the summary IS the description);
* a grounded chat question shows the compact inline figure in the
answer's sources block (image + the summary caption, the "shown in
the chat nicely" contract).
The negative case drives a SECOND module app with ``BOR_IMAGES``
forced ``false`` (the DEFAULT contract — an operator's local ``.env``
cannot leak the toggle in either direction: both apps pin the value
explicitly): the same upload + sync produces NO image document (the
walk is blind to the file, the source row stays a 0-document source).
"""
from __future__ import annotations
import base64
import json
import os
import re
import subprocess
import sys
import zipfile
from collections.abc import Iterator
from pathlib import Path
import httpx
import pytest
from playwright.sync_api import Browser, Page, expect
from sqlalchemy import select, text
from app.config import Settings
from app.db import SessionLocal
from app.models import Document
from app.rag.agent import IMAGE_DOC_MARKER
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
REPO = Path(__file__).resolve().parents[2]
GIT_SOURCES_URL = "/git-sources.html"
SOURCE_NAME = "e2e-image" # the archive stem (archive_source_name)
DOC_PATH = "pic.png"
#: The grounded question, carrying the house scripted-read call
#: (``SUMMARY_SEED_READ_TRIGGER`` — the phase-119 A1 convention: a
#: zero-read grounded turn chips NOTHING, so the image doc must be
#: READ to earn its citation chip + figure): the mock emits the
#: scripted ``read e2e-image/pic.png``, then echoes the tool result.
#: Grounding: the mock's token-overlap cosine against the
#: ``IMAGE_DESCRIPTION_ANSWER`` description is ≈0.31 (over the
#: mock-calibrated gate (0.30), and the FTS leg corroborates —
#: homelab/network/diagram all hit — the 0.15 floor backstop).
IMAGE_QUESTION = (
"Read the suggested document: read e2e-image/pic.png — "
"what is shown in the homelab network diagram?"
)
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
#: the pipeline is content-agnostic, the well-formed bytes keep the
#: upload + serve + render path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
# ---------------------------------------------------------------------------
# Module apps: images ON (the story) and images forced OFF (the default
# contract). Each owns its port + its scratch upload/source/image dirs.
# ---------------------------------------------------------------------------
def _app_env(mock_llm: int, app_port: int, *, images: bool, scratch: Path) -> dict[str, str]:
"""The conftest app env (leak guards included) with the phase-122
knobs: ``BOR_IMAGES`` pinned EXPLICITLY (true for the story app,
false for the default app — process env ranks above an operator's
local gitignored ``.env``, so the contract under test cannot leak
in either direction) and the image/upload homes in the suite's
scratch dir (the app under test must not write image copies into
the owner's real ``~/bor-sources``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern) — the story's question
# grounds on these values (see IMAGE_QUESTION).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget.
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Leak guards (conftest pattern).
env["BOR_GIT_SOURCES"] = ""
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
# Phase 122: the toggle under test + the suite-private homes.
env["BOR_IMAGES"] = "true" if images else "false"
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_IMAGE_DIR"] = str(scratch / "images")
return env
@pytest.fixture(scope="module")
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
"""The story app — ``BOR_IMAGES=true`` (the module env override;
the conftest session app is never started in this isolated run,
so no port clash)."""
scratch = tmp_path_factory.mktemp("bor_image_on")
port = int(os.environ.get("E2E_APP_PORT_IMAGES", "8150"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, images=True, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
@pytest.fixture(scope="module")
def default_app_server(
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The default-contract app — ``BOR_IMAGES=false`` (the LOCKED A3
default; only the negative test starts it)."""
scratch = tmp_path_factory.mktemp("bor_image_off")
port = int(os.environ.get("E2E_APP_PORT_IMAGES_OFF", "8151"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, images=False, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def default_app_url(default_app_server: str) -> str:
return default_app_server
@pytest.fixture(scope="module")
def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The fixture archive: ONE standalone PNG at the root — source
``e2e-image``, document path ``pic.png``."""
root = tmp_path_factory.mktemp("bor_image_zip")
(root / DOC_PATH).write_bytes(PNG_1X1)
archive = root / f"{SOURCE_NAME}.zip"
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
zf.write(root / DOC_PATH, arcname=DOC_PATH)
return archive
# ---------------------------------------------------------------------------
# DB + UI helpers
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh KB + registry per test (the E2E isolation pattern): the
suites share one Postgres, and a leftover row would corrupt the
counts. ``sources_meta`` (the KB generation counter) resets with
the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
))
db.commit()
def _health_db(app_url: str) -> bool:
try:
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
except Exception: # noqa: BLE001 — unreachable is the skip case
return False
def _upload_zip(page: Page, app_url: str, archive: Path) -> None:
"""The real admin upload flow: form login → the git-sources page →
pick the archive → submit → the terminal result line (phase 90:
"Uploaded <source> — press Sync sources to import it.")."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text(
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
timeout=UPLOAD_TIMEOUT_MS,
)
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
"""The RAG page's "Sync sources" button (the scan — phase 90 A3):
click → "Syncing…" → the terminal "Synced HH:MM" label + the fresh
counts in #sync-result (the never-stale contract, test_git_source_
dates' lifecycle wait)."""
page.goto(app_url + "/sources.html")
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
)
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-result")).to_have_text(expected_result)
def _drill_to_source(page: Page, name: str) -> None:
"""Top level → the source (the file table then holds its direct
files — ours is at the source root)."""
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
# ---------------------------------------------------------------------------
# The module seed: upload + sync the fixture image through the real UI
# (the story E2E truncate/re-import-per-module pattern).
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def seeded(
app_url: str,
browser: Browser,
zip_path: Path,
) -> Iterator[dict[str, str]]:
"""Upload the single-PNG zip through the admin UI, then scan it
with the RAG page's Sync button (the mock's vision model describes
the image in-process). Yields the seeded doc's identity (the bytes
route + the description) for the render assertions. The module
app's ``/api/config`` flag is asserted first — the env override
must have reached the app under test, or the whole premise of the
suite is void."""
if not _health_db(app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
_truncate_all()
page = browser.new_page(viewport={"width": 1280, "height": 800})
try:
_upload_zip(page, app_url, zip_path)
_sync_via_ui(page, app_url, expected_result="1 added")
with SessionLocal() as db:
doc = db.scalar(
select(Document).where(
Document.source == SOURCE_NAME, Document.path == DOC_PATH
)
)
assert doc is not None, "the uploaded image must become a document"
assert doc.is_image is True and doc.image_path is not None
# content == summary == the mock's description (the ONLY
# embedded text of the doc — the pipeline contract, LOCKED A3).
assert doc.content == IMAGE_DESCRIPTION_ANSWER
assert doc.summary == IMAGE_DESCRIPTION_ANSWER
finally:
page.close()
yield {
"image_url": f"/api/documents/{doc.id}/image",
"description": IMAGE_DESCRIPTION_ANSWER,
}
_truncate_all()
# ---------------------------------------------------------------------------
# 1. Sources page: the image doc is listed with its thumbnail
# ---------------------------------------------------------------------------
def test_sources_page_lists_the_image_doc(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""The KB total is the image doc (1 doc, 2 chunks — the one
content chunk + the phase-30 ``is_summary`` chunk), the drill-down
tree reaches it, and its row carries the FIXED 48px thumbnail box:
a lazy ``<img>`` from the bytes route (not the glyph fallback —
the fixture PNG is served), ``alt`` = the summary (the WCAG
contract)."""
login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("1")
expect(page.locator("#stat-chunks")).to_have_text("2")
expect(page.locator("#sources-empty")).to_be_hidden()
_drill_to_source(page, SOURCE_NAME)
row = page.locator("#docs-tbody tr", has_text=DOC_PATH)
expect(row).to_have_count(1)
box = row.locator(".kb-doc-thumb")
expect(box).to_be_visible()
img = row.locator(".kb-doc-thumb-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("loading", "lazy")
expect(img).to_have_attribute("alt", seeded["description"])
expect(row.locator(".kb-doc-thumb-glyph")).to_have_count(0)
# ---------------------------------------------------------------------------
# 2. Document viewer: the image renders, the description below it
# ---------------------------------------------------------------------------
def test_document_viewer_renders_image_and_description(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""The Sources row's path link opens the same-page document modal
(phase 26): the ``is_image`` content block renders the PERSISTENT
bytes first (``.doc-image-img`` from the bytes route, ``alt`` =
the summary), and the description follows in the normal content
slot (``pre.doc-raw`` — the plain-content path). The labeled
Summary panel is SUPPRESSED for the verbatim-description case
(summary === content — the importer invariant; the panel would
duplicate the text right below the image)."""
login(page, app_url)
_drill_to_source(page, SOURCE_NAME)
page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
expect(page.locator("#doc-modal-title")).to_have_text("pic")
img = modal.locator(".doc-image-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000) # the bytes route serves the PNG
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("alt", seeded["description"])
# The description below the image (the doc's readable content IS
# the vision description).
expect(modal.locator("pre.doc-raw")).to_have_text(seeded["description"])
# The verbatim case: no duplicate Summary panel.
expect(modal.locator(".doc-summary")).to_have_count(0)
# No "Image unavailable" note — the copy is intact.
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
# ---------------------------------------------------------------------------
# 3. Chat: a grounded question shows the inline image in the sources
# ---------------------------------------------------------------------------
def test_chat_sources_block_shows_the_inline_image(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""A question the mock grounds on the image doc (cosine ≈0.31,
FTS-corroborated), scripted to READ it (phase 119 A1 — a zero-read
turn chips nothing): the mock's echo carries the agent ``read``
result VERBATIM — including the task-05 ``IMAGE_DOC_MARKER`` line
(the model saw the description, not raw bytes) — and the answer's
sources block carries the citation chip (the READ doc clears the
usefulness floor) AND the compact inline figure — the ``<img>``
from the bytes route, the visible caption + ``alt`` settling to
the document's summary (the figure fetches it from the content
endpoint — the frame carries no summary), the "shown in the chat
nicely" contract."""
login(page, app_url, next="/")
expect(page.locator("#kb-banner")).to_be_hidden()
page.fill("#message-input", IMAGE_QUESTION)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble").last
# The scripted read's echoed result — the marker line (task 05)
# and the description itself (the model's view of the image).
expect(bubble).to_contain_text(IMAGE_DOC_MARKER, timeout=30_000)
expect(bubble).to_contain_text(seeded["description"], timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send")
# The citation chip (the text affordance stays — the figure is
# additive, not a replacement).
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(1)
expect(chip).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
# The inline figure: image + caption, both settling to the summary
# (the content fetch resolves the alt/caption after the title).
fig = page.locator(".msg.brain .source-image")
expect(fig).to_have_count(1)
img = fig.locator(".source-image-img")
expect(img).to_be_visible(timeout=15_000)
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("alt", seeded["description"], timeout=15_000)
caption = fig.locator(".source-image-caption")
expect(caption).to_have_text(seeded["description"], timeout=15_000)
# ---------------------------------------------------------------------------
# 4. The default contract: BOR_IMAGES off (the default) → the same
# upload + sync produces NO image document
# ---------------------------------------------------------------------------
def test_default_env_upload_produces_no_image_doc(
page: Page, default_app_url: str, zip_path: Path
) -> None:
"""The off-by-default contract end-to-end: with ``BOR_IMAGES``
false, the SAME upload + sync is byte-identical to the pre-phase
walk — the PNG is invisible to the scan (0 files, 0 added), the
source row stays a registered 0-document source, and NO documents
row (let alone an ``is_image`` one) exists. The file still lands
on disk (the unpack is unchanged — only the walk filter changes)."""
if not _health_db(default_app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{default_app_url}/api/config", timeout=5).json()
assert cfg["images"] is False, "the default app must run with images off"
_truncate_all()
try:
_upload_zip(page, default_app_url, zip_path)
_sync_via_ui(
page, default_app_url, expected_result="0 added · 0 unchanged"
)
# No document at all — the image file is not even a "file" to
# the images-off walk (not unknown, not indexed).
with SessionLocal() as db:
assert (
db.scalar(select(Document).where(Document.is_image.is_(True)))
is None
)
assert db.scalar(select(Document).where(Document.source == SOURCE_NAME)) is None
# The Sources page: 0 docs, the source row (registered) drills
# to an EMPTY file table.
page.goto(default_app_url + "/sources.html")
expect(page.locator("#stat-docs")).to_have_text("0")
expect(page.locator("#stat-chunks")).to_have_text("0")
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
page.click(f'#folders-tbody a.folder-link:text-is("{SOURCE_NAME}")')
expect(page.locator("#docs-tbody tr")).to_have_count(0)
finally:
_truncate_all()
+7 -6
View File
@@ -164,14 +164,15 @@ def test_config_serves_the_overrides(custom_server: str) -> None:
r = httpx.get(f"{CUSTOM_URL}/api/config", timeout=5)
assert r.status_code == 200
body = r.json()
# The five-key set (the phase-39/59/62 endpoint contract, phase 91
# task 03: the retired CSS-file theming's ``theme`` key is gone)
# with the two customization overrides — the app NAME stays the
# default (this suite does not re-test BOR_APP_NAME; that is the
# phase-39 suite's job).
# The six-key set (the phase-39/59/62 endpoint contract, phase 91
# task 03: the retired CSS-file theming's ``theme`` key is gone;
# phase 122 task 01: the ``images`` flag) with the two
# customization overrides — the app NAME stays the default (this
# suite does not re-test BOR_APP_NAME; that is the phase-39
# suite's job).
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["app_name"] == DEFAULT_NAME
assert body["input_placeholder"] == CUSTOM_PLACEHOLDER
+8 -1
View File
@@ -1,6 +1,8 @@
"""Shared test fakes (no network, deterministic)."""
from __future__ import annotations
from typing import Any
from app.config import Settings
from app.rag.llm import LLMError
@@ -15,6 +17,11 @@ class FakeEmbedder:
``"Summary of <first token of the user content>"`` and raises
:class:`LLMError` when the content contains the sentinel word
``SUMMARY-BLOWUP`` (drives the importer's fail-soft summary path).
``content`` may be a string or a phase-122 multimodal part list
(``dict[str, Any]`` messages, the ``LLMClient.chat`` shape) — the
text-summary body above is string-only; a subclass handling the
multimodal image description (task 03's vision mock) overrides
``chat``.
"""
def __init__(self, dim: int = 768) -> None:
@@ -36,7 +43,7 @@ class FakeEmbedder:
return vec
async def chat(
self, messages: list[dict[str, str]], model: str | None = None
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
+37 -9
View File
@@ -48,25 +48,30 @@ def test_health_reports_ok(client) -> None:
def test_config_returns_default_app_metadata(client, db: Session) -> None:
"""GET /api/config is public (anonymous) and returns exactly five
"""GET /api/config is public (anonymous) and returns exactly six
keys — the phase-39 app metadata, the phase-59 docs flag (inert
false while BOR_DOCS_REPO is empty — the "Save as doc" gating),
and the phase-62 UI customization strings (composer placeholder,
footer line). Phase 91: with an empty ui_settings table the
effective strings are the env defaults (B1 — DB-over-env, the row
absent here); the retired CSS-file theming's ``theme`` key is gone
(task 03 — the five keys are the entire contract)."""
the phase-122 images flag (default false in the test env —
LOCKED A3: off by default), and the phase-62 UI customization
strings (composer placeholder, footer line). Phase 91: with an
empty ui_settings table the effective strings are the env defaults
(B1 — DB-over-env, the row absent here); the retired CSS-file
theming's ``theme`` key is gone (task 03 — the six keys are the
entire contract)."""
_clear_ui_settings(db)
r = client.get("/api/config")
assert r.status_code == 200
body = r.json()
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["app_name"] == "Brain of Reese"
assert body["version"] == get_settings().app_version
assert body["docs_repo_configured"] is False
# Phase 122 (task 01): the images flag is the default off (the
# test env sets no BOR_IMAGES) — a real bool, not a truthy string.
assert body["images"] is False
# Phase 62: UNSET => the phase-61 neutral copy stands (the
# byte-identical contract).
assert body["input_placeholder"] == "Ask me anything…"
@@ -91,7 +96,7 @@ def test_config_follows_overridden_app_name(client, db: Session) -> None:
body = r.json()
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["app_name"] == "Brain of Testy"
assert body["version"] == "0.1.0"
@@ -122,7 +127,7 @@ def test_config_serves_ui_customization_overrides(client, db: Session) -> None:
body = r.json()
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["input_placeholder"] == "Ask the vault…"
assert body["footer_text"] == "Powered by my own models"
@@ -153,6 +158,29 @@ def test_config_docs_flag_tracks_settings(client, db: Session) -> None:
fastapi_app.dependency_overrides.clear()
def test_config_images_flag_tracks_settings(client, db: Session) -> None:
"""Phase 122 (task 01): ``images`` mirrors ``settings.images``
(``BOR_IMAGES``) — a real bool (never a truthy string) that flips
true the moment the toggle is on: that flag is the entire frontend
gating of the image affordances (the phase-123 attach control,
optionally the Sources page hint)."""
from app.config import Settings
from app.main import app as fastapi_app
_clear_ui_settings(db)
fastapi_app.dependency_overrides[get_settings] = lambda: Settings(
images=True,
)
try:
r = client.get("/api/config")
assert r.status_code == 200
body = r.json()
assert isinstance(body["images"], bool)
assert body["images"] is True
finally:
fastapi_app.dependency_overrides.clear()
def test_suggestions_returns_list(client) -> None:
# Phase 79: the chips are user-gated — sign in as the admin first
# (the test's purpose is the list shape, not the auth contract).
+3
View File
@@ -431,8 +431,11 @@ def test_document_content_admin_contract(client: TestClient, db) -> None:
"content",
"indexed_at",
"chunks",
"is_image", # added in phase 122 (task 04) — always present
}
assert body["summary"] is None
assert body["is_image"] is False # text doc — and no image_url key (never null)
assert "image_url" not in body
# Unknown docs still 404 (same shape as the phase-16 pin).
r = client.get("/api/documents/content", params={"source": "docs", "path": "nope.md"})
+121
View File
@@ -2076,3 +2076,124 @@ def test_done_event_related_defaults_empty_and_old_payload_parses() -> None:
dumped = ChatDoneEvent(**new_payload).model_dump()
assert [r["path"] for r in dumped["related"]] == ["b.md"]
assert [r["path"] for r in dumped["sources"]] == ["a.md"]
# ---------------------------------------------------------------------------
# Phase 122, task 05 — the SSE source frame's OPTIONAL image_url (the
# shared ``source_ref_with_image`` builder: present on an image doc's
# ref only, omitted — never null — on every text ref). Task 06
# finalizes the phase-122 suite here with the story-level pins.
# ---------------------------------------------------------------------------
def _seed_image_doc(db) -> Document:
"""An ``is_image`` documents row the agent can ``read`` (no chunks
needed — the read tool resolves the row by ``(source, path)`` and
serves its ``content``; the frame needs ``is_image`` + ``id``
only). The fixture's teardown truncates the tables."""
doc = Document(
id=uuid.uuid4(),
source="docs",
path="pic.png",
full_path="/tmp/pic.png",
title="pic",
content="A red square on a white background.",
summary="A red square on a white background.",
content_hash="0" * 64,
created_at=_FIXTURE_CREATED_AT,
is_image=True,
image_path="/tmp/pic.png",
)
db.add(doc)
db.commit()
return doc
def test_grounded_turn_reading_image_doc_frame_carries_image_url_only_for_it(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Task 05 wire contract: a mocked grounded answer whose agent
READS an image doc → the done frame's ref for THAT doc alone
carries ``image_url`` (the bytes route the frontend's sources
block renders from); the text-doc related refs carry NO
``image_url`` key at all (the omission rule — the key is absent,
never null)."""
doc = _seed_image_doc(db)
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "docs/pic.png"},
)
]
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert done["deflected"] is False
# The read image doc is the citation surface (phase 119, A1) — and
# its ref is the frame's ONLY image_url carrier.
sources = done["sources"]
assert [(s["source"], s["path"]) for s in sources] == [("docs", "pic.png")]
assert sources[0]["image_url"] == f"/api/documents/{doc.id}/image"
# The related tier (ranks 6–7 for the Kubernetes question) is text
# docs — the key is ABSENT on every one of them, not null.
related = done["related"]
assert related
for ref in related:
assert "image_url" not in ref
def test_text_only_grounded_turn_frame_has_no_image_url_key_anywhere(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""The omission rule at the BYTE level (the phase's byte-identity
criterion): a grounded turn whose cited + related docs are ALL
text docs serializes a done frame with no ``image_url`` key
anywhere — checked on the raw wire text (not a re-serialized
dict), and every ref keeps exactly the pre-phase-122 key set."""
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "docs/homelab/kubernetes.md"},
)
]
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with client.stream("POST", "/api/chat", json={"message": QUESTION}) as r:
assert r.status_code == 200
buf = ""
raw_frames: list[str] = []
for part in r.iter_text():
buf += part
while "\n\n" in buf:
frame, buf = buf.split("\n\n", 1)
frame = frame.strip()
if frame.startswith("data:"):
raw_frames.append(frame)
finally:
fastapi_app.dependency_overrides.clear()
done_raw = [f for f in raw_frames if '"type": "done"' in f]
assert len(done_raw) == 1
# The BYTE check: the key is absent from the wire text itself.
assert "image_url" not in done_raw[0]
done = json.loads(done_raw[0].removeprefix("data:").strip())
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("docs", "homelab/kubernetes.md")
]
for ref in done["sources"] + done["related"]:
assert set(ref) == {"source", "path", "title"} # the pre-122 key set
+611
View File
@@ -7,22 +7,33 @@ Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
"""
from __future__ import annotations
import asyncio
import base64
import hashlib
import inspect
import itertools
import logging
import uuid
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import delete, func, select, text
import app.api.docs as docs_api
import app.rag.importer as rag_importer
from app.config import Settings
from app.core import tokens as token_service
from app.main import app as fastapi_app
from app.models import Chunk, Document, FolderSummary, GitSource
from app.rag import git_sources as rag_git_sources
from app.rag.folder_summaries import missing_folder_summaries
from app.rag.importer import import_sources
from app.rag.llm import LLMError
from app.rag.summarizer import DESCRIBE_PROMPT
from tests.fakes import FakeEmbedder
_TREE_TABLES = "chunks, documents, folder_summaries, git_sources"
@@ -823,3 +834,603 @@ def test_docs_tree_stat_walk_equivalence_with_flat_list(admin_client, db) -> Non
)
_truncate_tree_tables(db)
# ---------------------------------------------------------------------------
# Phase 122, task 02 — image ingest end-to-end (the full ``import_sources``
# pipeline against the real DB; task 06 finalizes the phase-122 suite here
# with the image route + content-endpoint shapes).
# ---------------------------------------------------------------------------
#: A real 1×1 transparent PNG — the importer is content-agnostic (it
#: never parses the image), but a well-formed fixture keeps the tests
#: honest about what a real upload looks like.
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
def _image_llm[
ImageLLM: FakeEmbedder
](tmp_path, llm_cls: type[ImageLLM] = FakeEmbedder, **kwargs) -> ImageLLM:
"""A fake LLM with the phase-122 image knobs (toggle ON by default;
the image dir defaults under *tmp_path* unless overridden).
*llm_cls* (task 03) may be the mock vision client (``_MockVisionLLM``
below) for the no-seam-patch end-to-end path — the PEP 695 type
parameter keeps the helper's return type honest (``chat_models``).
"""
kwargs.setdefault("_env_file", None)
kwargs.setdefault("images", True)
kwargs.setdefault("image_dir", str(tmp_path / "images"))
llm = llm_cls()
llm.settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return llm
def _patch_description(monkeypatch, description) -> None:
"""Pin the task-02 seam (``rag_importer._describe_or_skip``) to
*description*. Task 03 fills the seam with the CHAT model's vision
call — these end-to-end mechanics do not change with it."""
async def _fake(llm, *, data, source, rel, full_path):
return description
monkeypatch.setattr(rag_importer, "_describe_or_skip", _fake)
def _cleanup_source(db, source: str) -> None:
for doc in db.scalars(select(Document).where(Document.source == source)).all():
db.delete(doc)
db.commit()
def test_import_sources_images_on_indexes_image_docs(
db, tmp_path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``images=True`` end-to-end: a standalone image in a source becomes
a Document — bytes digested, the persistent copy in ``image_dir``
(``<doc-id>.png``), ``content`` = the (mock) description, and ONLY
that text is embedded (the content chunks + the phase-30 summary
chunk, 768 dims) — while a text doc in the same source stays a
plain text doc."""
source_root = tmp_path / "imgsource"
source_root.mkdir()
(source_root / "diagram.png").write_bytes(PNG_1X1)
(source_root / "notes.md").write_text("# Notes\n\nBody text.\n", encoding="utf-8")
llm = _image_llm(tmp_path)
_patch_description(monkeypatch, "A network diagram of the homelab VLANs.")
try:
summary = asyncio.run(
import_sources([source_root], llm, session=db, prune=True)
)
assert (summary.files, summary.added, summary.images_failed) == (2, 2, 0)
assert summary.formats == {"md": 1, "png": 1}
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "diagram.png"
)
)
assert doc is not None, "the image file must become a document"
assert doc.is_image is True and doc.image_path is not None
assert doc.content == "A network diagram of the homelab VLANs."
assert doc.title == "diagram" # the non-markdown stem rule
copy = Path(doc.image_path)
assert copy.parent == Path(llm.settings.image_dir)
assert copy.name == f"{doc.id}.png"
assert copy.read_bytes() == PNG_1X1
# The ONLY embedded text is the description (the embedding model
# never sees pixels): every content chunk carries it, the
# phase-30 summary chunk exists + is embedded, 768 dims. Task
# 03: the description IS the summary (stored verbatim — no
# ``lite`` call, no pointer line), so the summary chunk mirrors
# ``doc.content`` exactly.
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [doc.content]
assert doc.summary == doc.content
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert summary_chunks[0].content == doc.content
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
# The text doc is untouched by the image machinery.
md = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "notes.md"
)
)
assert md is not None and md.is_image is False and md.image_path is None
finally:
_cleanup_source(db, source_root.name)
class _MockVisionLLM(FakeEmbedder):
"""The phase-122 mock VISION client (task 03): the CHAT model
answers the multimodal describe call (the image bytes' data URL)
with a fixed, retrieval-oriented description; text (``lite``) calls
keep the ``FakeEmbedder`` behaviour. The REAL
``rag_importer._describe_or_skip`` → ``summarizer.describe_image``
chain runs end-to-end against it (no seam patch); every chat
call's model is recorded (``chat_models``)."""
DESCRIPTION = (
"A network diagram of the homelab VLANs: the core switch, the "
"router, and three labeled subnets."
)
def __init__(self) -> None:
super().__init__()
self.chat_models: list[str | None] = []
async def chat(self, messages, model=None):
self.chat_calls.append(list(messages))
self.chat_models.append(model)
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
# The phase-122 describe call — the multimodal message.
return self.DESCRIPTION
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def test_import_sources_mock_vision_client_end_to_end(db, tmp_path) -> None:
"""Task 03 end-to-end (NO seam patch): a fixture PNG through the
mock vision client — the real ``_describe_or_skip`` →
``describe_image`` → CHAT-model call — yields a doc whose
``content`` == ``summary`` == the description, with its
``is_summary`` chunk embedded, and whose ONLY embedded text is that
description (the embedding model never sees pixels)."""
source_root = tmp_path / "visione2e"
source_root.mkdir()
(source_root / "diagram.png").write_bytes(PNG_1X1)
llm = _image_llm(tmp_path, _MockVisionLLM)
try:
summary = asyncio.run(
import_sources([source_root], llm, session=db)
)
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
# The describe call went to the CHAT model (LOCKED A3), once.
assert llm.chat_models == [llm.settings.llm_chat_model]
# The wire shape: the multimodal user message — the fixed
# prompt's text part + the image's data-URL part.
(message,) = llm.chat_calls[0]
assert message["role"] == "user"
content: Any = message["content"] # the multimodal part list
assert content[0] == {"type": "text", "text": DESCRIBE_PROMPT}
assert content[1]["type"] == "image_url"
assert content[1]["image_url"]["url"].startswith("data:image/png;base64,")
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "diagram.png"
)
)
assert doc is not None, "the image file must become a document"
assert doc.is_image is True and doc.image_path is not None
assert doc.content == _MockVisionLLM.DESCRIPTION
assert doc.summary == _MockVisionLLM.DESCRIPTION # task 03: verbatim
# The ONLY embedded text of the doc is the description — twice
# (the one content chunk + the is_summary chunk), 768 dims.
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert all(c.content == _MockVisionLLM.DESCRIPTION for c in chunks)
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
embedded = [t for batch in llm.calls for t in batch]
assert embedded == [_MockVisionLLM.DESCRIPTION] * 2
finally:
_cleanup_source(db, source_root.name)
class _MockVisionFailsLLM(FakeEmbedder):
"""A NON-VISION chat model (LOCKED A3's honest failure): the
multimodal describe call raises (the SDK errors — a chat model
without vision rejects the ``image_url`` part), text (``lite``)
calls keep the ``FakeEmbedder`` behaviour."""
async def chat(self, messages, model=None):
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
raise LLMError("simulated non-vision chat model (test sentinel)")
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def test_import_sources_failing_vision_skips_image_keeps_sync_green(
db, tmp_path, caplog: pytest.LogCaptureFixture
) -> None:
"""LOCKED A3 fail-soft end-to-end (the task-06 integration pin):
a fixture PNG through a NON-VISION chat model — the real seam, no
patch — skips the image doc (``images_failed == 1``, NO row, NO
orphan copy — not even the image dir) while the TEXT doc in the
same source is indexed as usual (the sync completes, no row
mutation anywhere for the failed image)."""
source_root = tmp_path / "visionfail"
source_root.mkdir()
(source_root / "diagram.png").write_bytes(PNG_1X1)
(source_root / "notes.md").write_text("# Notes\n\nBody.\n", encoding="utf-8")
llm = _image_llm(tmp_path, _MockVisionFailsLLM)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(
import_sources([source_root], llm, session=db)
)
assert (summary.files, summary.added, summary.images_failed) == (2, 1, 1)
# The image: no row, no copy (the dir itself was never created).
assert (
db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "diagram.png"
)
)
is None
)
assert not Path(llm.settings.image_dir).expanduser().exists()
# The text doc indexed as usual (the sync stayed green).
md = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "notes.md"
)
)
assert md is not None and md.is_image is False
# The importer's warning names the document (the PLAN §9 signal).
warnings = [
r
for r in caplog.records
if r.name == "app.importer" and "image description failed" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
assert f"source={source_root.name} path=diagram.png" in warnings[0].getMessage()
finally:
_cleanup_source(db, source_root.name)
def test_import_sources_images_off_ignores_and_prune_guard_protects(
db, tmp_path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``images=False`` (the default) end-to-end: the walk ignores the
image file entirely (not counted, no row, no copy), and a
``prune=True`` run MUST NOT delete a pre-existing image doc — the
LOCKED prune guard (invisible to the walk ≠ deleted)."""
source_root = tmp_path / "imgsrc_off"
source_root.mkdir()
(source_root / "diagram.png").write_bytes(PNG_1X1)
_patch_description(monkeypatch, "A network diagram.")
try:
llm_on = _image_llm(tmp_path)
s_on = asyncio.run(import_sources([source_root], llm_on, session=db))
assert s_on.added == 1
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "diagram.png"
)
)
assert doc is not None
copy = Path(doc.image_path)
assert copy.exists()
# Toggle OFF (a fresh fake on the code defaults): the walk is
# blind to the file, and prune protects the pre-existing image
# doc + its copy.
llm_off = FakeEmbedder() # Settings(_env_file=None) → images False
s_off = asyncio.run(
import_sources([source_root], llm_off, session=db, prune=True)
)
assert (s_off.files, s_off.added, s_off.pruned) == (0, 0, 0)
assert db.scalar(select(Document).where(Document.id == doc.id)) is not None
assert copy.exists(), "the copy survives with the doc"
finally:
_cleanup_source(db, source_root.name)
def test_import_sources_toggle_on_prunes_deleted_image_with_copy(
db, tmp_path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Toggle ON, the image file deleted: the normal prune runs — the
doc row AND its ``image_dir`` copy are removed (the copy's
lifecycle is tied to the row)."""
source_root = tmp_path / "imgsrc_prune"
source_root.mkdir()
(source_root / "diagram.png").write_bytes(PNG_1X1)
llm_on = _image_llm(tmp_path)
_patch_description(monkeypatch, "A network diagram.")
try:
asyncio.run(import_sources([source_root], llm_on, session=db))
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "diagram.png"
)
)
assert doc is not None
copy = Path(doc.image_path)
assert copy.exists()
(source_root / "diagram.png").unlink()
s = asyncio.run(
import_sources([source_root], llm_on, session=db, prune=True)
)
assert (s.pruned, s.added, s.unchanged) == (1, 0, 0)
assert (
db.scalar(select(Document).where(Document.source == source_root.name))
is None
)
assert not copy.exists(), "the image_dir copy is deleted with the doc"
finally:
_cleanup_source(db, source_root.name)
# ---------------------------------------------------------------------------
# Phase 122 (task 04) — the image BYTES route + the content/tree wire
# affordance (the serve side of the image-document contract).
# ---------------------------------------------------------------------------
def _seed_image_doc(
db,
tmp_path: Path,
*,
source: str = "ImgSrc",
path: str = "pic.png",
data: bytes = PNG_1X1,
content: str = "A red square on a white background.",
image_path: str | None = "auto",
doc_id: str | None = None,
) -> Document:
"""One ``is_image`` document row with its persistent copy (the
importer's ``image_dir`` layout) under *tmp_path*; ``image_path``
``"auto"`` writes the copy, ``None`` leaves the row without a copy
(the lost-copy corner)."""
doc = Document(
id=uuid.UUID(doc_id) if doc_id else uuid.uuid4(),
source=source,
path=path,
full_path=f"/tmp/{path}",
title=path.rsplit(".", 1)[0],
content=content,
content_hash=hashlib.sha256(data).hexdigest(),
indexed_at=datetime.now(UTC),
created_at=datetime(2026, 1, 1, tzinfo=UTC),
summary=content,
is_image=True,
)
if image_path == "auto":
copy = tmp_path / f"{doc.id}{Path(path).suffix}"
copy.write_bytes(data)
doc.image_path = str(copy)
elif image_path is not None:
doc.image_path = image_path
db.add(doc)
db.flush()
db.add_all(
[
Chunk(
document_id=doc.id, position=0, content=content, embedding=[0.01] * 768
),
Chunk(
document_id=doc.id,
position=-1,
content=content,
embedding=[0.01] * 768,
is_summary=True,
),
]
)
db.commit()
return doc
def _cleanup_kb(db) -> None:
"""Truncate the KB tables. The settle commit matters: SQLAlchemy
does NOT autoflush pending ORM objects before a raw ``text()``
statement — a pending chunks INSERT flushed *after* the TRUNCATE
would FK-violate (the document row is already gone), so any
pending state is committed (then truncated) first."""
db.commit()
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
@pytest.mark.parametrize(
("ext", "mime"),
[
(".png", "image/png"),
(".jpg", "image/jpeg"),
(".jpeg", "image/jpeg"),
(".webp", "image/webp"),
(".gif", "image/gif"),
(".bmp", "image/bmp"),
],
)
def test_image_route_serves_exact_bytes_with_content_type(
admin_client: TestClient, db, tmp_path: Path, ext: str, mime: str
) -> None:
"""The serve contract: the route streams the EXACT stored bytes
(a per-extension sentinel — a mix-up between the six formats is
caught) with the extension's ``Content-Type`` (the
``IMAGE_MIMES`` map — one map, one truth) and
``Cache-Control: private, max-age=3600`` (content-hashed bytes —
long enough, bustable by re-upload)."""
_cleanup_kb(db)
try:
data = PNG_1X1 + ext.encode("ascii") # per-extension sentinel bytes
doc = _seed_image_doc(db, tmp_path, path=f"pic{ext}", data=data)
r = admin_client.get(f"/api/documents/{doc.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == mime # exact type per extension
assert r.headers["cache-control"] == "private, max-age=3600"
assert r.content == data # the exact uploaded bytes, nothing else
finally:
_cleanup_kb(db)
def test_image_route_404_matrix(admin_client: TestClient, db, tmp_path: Path) -> None:
"""Every non-servable case is 404 ``document not found`` (the
router's unknown-document shape — the same detail string the
content endpoint uses): a missing id, a MALFORMED id (an unparseable
string maps here, not to a 422 — a guessed id is an unknown
document), a text doc, an image doc whose ``image_path`` is NULL,
and a row whose copy was lost on disk (defensive — the row exists,
the bytes don't)."""
_cleanup_kb(db)
try:
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
db.commit() # the module's _seed_doc leaves the row uncommitted
text_doc_id = db.scalar(
select(Document.id).where(
Document.source == "TextSrc", Document.path == "note.md"
)
)
no_copy = _seed_image_doc(db, tmp_path, path="nopy.png", image_path=None)
lost = _seed_image_doc(db, tmp_path, path="lost.png")
assert lost.image_path is not None # the "auto" copy was written
Path(lost.image_path).unlink() # the copy is lost (the row remains)
for doc_id in (
str(uuid.uuid4()), # missing id
"not-a-uuid", # malformed id → 404, not 422
str(text_doc_id), # text doc
str(no_copy.id), # image doc, image_path NULL
str(lost.id), # image doc, copy lost
):
r = admin_client.get(f"/api/documents/{doc_id}/image")
assert r.status_code == 404, doc_id
assert r.json() == {"detail": "document not found"}, doc_id
finally:
_cleanup_kb(db)
def test_image_route_requires_user_like_the_content_endpoint(
admin_client: TestClient, db, tmp_path: Path
) -> None:
"""Phase 79 posture (the task's "PUBLIC, like the document content
endpoint" — the content endpoint has been user-gated since phase
79; the ONLY anonymous surface is the shared chats, PLAN A10): an
anonymous caller gets 401 ``authentication required`` before any
row is read (a FRESH client — the module's fixture ``client``
stays unsigned here), a signed-in caller gets the bytes."""
_cleanup_kb(db)
try:
doc = _seed_image_doc(db, tmp_path)
anonymous = TestClient(fastapi_app)
r = anonymous.get(f"/api/documents/{doc.id}/image")
assert r.status_code == 401
assert r.json() == {"detail": "authentication required"}
# The signed-in client (admin) passes.
assert admin_client.get(f"/api/documents/{doc.id}/image").status_code == 200
finally:
_cleanup_kb(db)
def test_content_endpoint_exposes_the_image_affordance(
admin_client: TestClient, db, tmp_path: Path
) -> None:
"""The content endpoint (the viewer's data source): ``is_image`` is
ALWAYS present (text doc: false — the one new key; the wire shape
gains nothing else), and ``image_url`` — the bytes route's path —
is present for an image doc and ABSENT for a text doc (never null,
the ``DocContent`` omission rule)."""
_cleanup_kb(db)
try:
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
db.commit() # the app's endpoint session reads committed data only
r = admin_client.get(
"/api/documents/content", params={"source": "TextSrc", "path": "note.md"}
)
assert r.status_code == 200
body = r.json()
assert body["is_image"] is False
assert "image_url" not in body # absent — never null (text doc)
doc = _seed_image_doc(db, tmp_path, source="ImgSrc", path="pic.png")
r = admin_client.get(
"/api/documents/content", params={"source": "ImgSrc", "path": "pic.png"}
)
assert r.status_code == 200
body = r.json()
assert body["is_image"] is True
assert body["image_url"] == f"/api/documents/{doc.id}/image"
assert body["content"] == body["summary"] # the description (task 03)
finally:
_cleanup_kb(db)
def _tree_file_nodes_all(sources) -> list[dict]:
"""Every file node of a tree response, walked recursively (all
sources — env-registered 0-document sources may join the response
when the ``git_sources`` table is truncated, and they carry no
file nodes; the assertions below hold over whatever files exist).
"""
files: list[dict] = []
def _walk(node: dict) -> None:
for child in node.get("children", ()):
if child["kind"] == "file":
files.append(child)
else:
_walk(child)
for source in sources:
_walk(source)
return files
def test_tree_image_file_node_affordance_and_text_node_byte_identical(
admin_client: TestClient, db, tmp_path: Path
) -> None:
"""The tree (the RAG view's single fetch): an image doc's file node
carries the thumbnail affordance (``is_image`` true,
``image_url`` = the bytes route's path, ``summary`` verbatim — the
RAG view's thumbnail ``alt``); EVERY text file node keeps the
pre-phase wire shape byte-identically (the six keys — no
``is_image``/``image_url``/``summary`` — the phase's
byte-identical criterion: the fields are row-driven, so a KB with
no image rows serializes exactly as pre-phase)."""
_truncate_tree_tables(db)
try:
base = datetime.now(UTC)
_seed_doc(db, "MixedSrc", "a.md", "A", 1, base)
img = _seed_image_doc(db, tmp_path, source="MixedSrc", path="pic.png")
r = admin_client.get("/api/docs/tree") # both seeds committed (_seed_image_doc)
assert r.status_code == 200
files = {f["path"]: f for f in _tree_file_nodes_all(r.json()["sources"])}
# Text node: byte-identical pre-phase wire shape (no image keys).
assert set(files["a.md"]) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
# Image node: the affordance rides the node.
pic = files["pic.png"]
assert pic["is_image"] is True
assert pic["image_url"] == f"/api/documents/{img.id}/image"
assert pic["summary"] == "A red square on a white background."
finally:
_truncate_tree_tables(db)
def test_tree_with_no_image_rows_is_byte_identical(admin_client: TestClient, db) -> None:
"""The pre-phase KB (no ``is_image`` rows): the image-docs map is
empty and EVERY file node serializes in the pre-phase shape — the
row-driven fields introduce no wire change at all (the phase's
byte-identical criterion, the toggle irrelevant)."""
_truncate_tree_tables(db)
try:
base = datetime.now(UTC)
_seed_doc(db, "PlainSrc", "x.md", "X", 2, base)
db.commit() # the app's endpoint session reads committed data only
r = admin_client.get("/api/docs/tree")
assert r.status_code == 200
files = _tree_file_nodes_all(r.json()["sources"])
assert len(files) == 1 # the seeded doc (env sources carry no files)
assert set(files[0]) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
finally:
_truncate_tree_tables(db)
+7 -2
View File
@@ -422,11 +422,16 @@ def test_content_200_all_fields(client, db) -> None:
assert r.status_code == 200
body = r.json()
# Wire-additive (phase 106, task 05): ``created_at`` joins the
# content shape (after ``summary``, before ``content``).
# content shape (after ``summary``, before ``content``) — and
# (phase 122, task 04) ``is_image`` joins it ALWAYS present
# (text docs: false); ``image_url`` is ABSENT for a text doc
# (never null — the ``DocContent`` omission rule).
assert set(body) == {
"source", "path", "title", "format", "summary", "created_at",
"content", "indexed_at", "chunks",
"content", "indexed_at", "chunks", "is_image",
}
assert body["is_image"] is False
assert "image_url" not in body # absent — never null (text doc)
datetime.fromisoformat(body["created_at"]) # raises if not ISO-8601
assert body["source"] == "Homelab"
assert body["path"] == "kubernetes.md"
+344
View File
@@ -0,0 +1,344 @@
"""Integration: migration 0022 (documents.is_image + image_path) schema
contract (phase 122, task 02).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0021.py`` (information_schema assertions on the state
the migration must leave). The tests target the 0021 → 0022 step
explicitly so later migrations cannot break the pins:
* upgrade 0021 → 0022 → ``is_image`` exists with the full contract —
BOOLEAN, NOT NULL, server default ``false`` — and ``image_path`` —
TEXT, NULLABLE, no server default — while the 0021 ``documents``
schema (``content``/``content_hash`` NOT NULL, ``summary`` NULLABLE,
``created_at`` + ``created_at_manual``, the (source, path) unique
constraint — asserted column-based, since the suite's table
self-heal renames copied constraints) survives;
* a ``documents`` row inserted while the DB is at 0021 backfills
``is_image`` to ``false`` and ``image_path`` to NULL (every
pre-phase-122 row is a text doc — the LOCKED A3 default);
* the ORM contract agrees: a freshly inserted ``Document`` without the
image fields reads ``is_image is False`` / ``image_path is None``,
and one with them round-trips through a fresh session;
* downgrade to 0021 → both columns are GONE (A13) while the row
survives; upgrade back to 0022 → the columns are back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import Document
SOURCE = "mig0022"
PATH_TEXT = "notes/readme.md"
PATH_IMAGE = "notes/diagram.png"
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown
upgrades to head no matter what happened, so the dev DB is never
left below head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``documents``, which would deadlock the repair's
# ``ALTER TABLE`` (0022) forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one documents
column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'documents' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _insert_sql(db: Session, path: str) -> uuid.UUID:
"""Insert one documents row with the PRE-0022 column set (the 0021
shape — the image columns, when present, are omitted so their
backfill is what the row reads)."""
row_id = uuid.uuid4()
db.execute(
text(
"INSERT INTO documents (id, source, path, full_path, title,"
" content, content_hash)"
" VALUES (:id, :s, :p, :f, :t, :c, :h)"
),
{
"id": row_id,
"s": SOURCE,
"p": path,
"f": f"/tmp/{SOURCE}/{path}",
"t": path.rsplit("/", 1)[-1],
"c": "content",
"h": "0" * 64,
},
)
db.commit()
return row_id
def _delete(db: Session, row_id: uuid.UUID) -> None:
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": row_id})
db.commit()
def _unique_column_sets(db: Session) -> set[tuple[str, ...]]:
"""The column tuples of every UNIQUE constraint on ``documents``.
Column-based (not name-based): the integration suite's table
self-heal (``tests/integration/conftest.py``) rewrites bloated
tables via ``CREATE TABLE (LIKE …)``, which renames copied
constraints (PG auto-names them) — the (source, path) uniqueness
contract is what must hold, not the original name.
"""
rows = db.execute(
text(
"SELECT (SELECT string_agg(a.attname, ',' ORDER BY k.ord)"
" FROM unnest(c.conkey) WITH ORDINALITY k(attnum, ord)"
" JOIN pg_attribute a"
" ON a.attrelid = c.conrelid AND a.attnum = k.attnum)"
" FROM pg_constraint c"
" WHERE c.contype = 'u' AND c.conrelid = 'documents'::regclass"
)
).fetchall()
return {tuple(r[0].split(",")) for r in rows}
def test_upgrade_to_0022_adds_image_columns(db: Session, alembic: Config) -> None:
"""Upgrade 0021 → 0022: ``is_image`` exists with the full contract
(BOOLEAN, NOT NULL, server default ``false`` — every pre-phase-122
row is a text doc) and ``image_path`` (TEXT, NULLABLE, no server
default — NULL for text docs), both ABSENT at 0021; a pre-0022 row
backfills ``is_image`` to ``false`` + ``image_path`` to NULL; and
the 0021 table contract survives the additive upgrade."""
command.downgrade(alembic, "0021") # start from the pre-0022 state
assert _version(db) == "0021"
assert _column(db, "is_image") is None, "is_image must be absent at 0021"
assert _column(db, "image_path") is None, "image_path must be absent at 0021"
pre_id = _insert_sql(db, PATH_TEXT) # the 0021 column set
try:
command.upgrade(alembic, "0022")
assert _version(db) == "0022", "alembic_version must be at 0022"
is_image = _column(db, "is_image")
assert is_image is not None, "documents.is_image is missing"
assert is_image[0] == "boolean", "is_image must be BOOLEAN"
assert is_image[1] == "NO", "is_image must be NOT NULL"
assert is_image[2] == "false", (
"is_image must carry the `false` server default — every"
" pre-phase-122 row is a text doc"
)
image_path = _column(db, "image_path")
assert image_path is not None, "documents.image_path is missing"
assert image_path[0] == "text", "image_path must be TEXT"
assert image_path[1] == "YES", "image_path must be NULLABLE"
assert image_path[2] is None, (
"image_path must carry NO server default — NULL is the"
" text-doc value"
)
# The pre-0022 row backfilled to (false, NULL) — a text doc.
row = db.execute(
text("SELECT is_image, image_path FROM documents WHERE id = :id"),
{"id": pre_id},
).fetchone()
assert row is not None, "the pre-0022 row must survive the upgrade"
assert row[0] is False, "the backfilled is_image must be false"
assert row[1] is None, "the backfilled image_path must be NULL"
# A row written without the image columns reads the same
# (the Python-side defaults are False/None — same values).
new_id = _insert_sql(db, PATH_IMAGE)
try:
backfilled = db.execute(
text("SELECT is_image, image_path FROM documents WHERE id = :id"),
{"id": new_id},
).fetchone()
assert backfilled == (False, None), (
"an omitted image state must read (false, NULL)"
)
finally:
_delete(db, new_id)
# The 0021 schema survives the additive upgrade.
content = _column(db, "content")
assert content is not None and content[0] == "text" and content[1] == "NO", (
"documents.content (0001) must keep its 0021 contract"
)
hash_col = _column(db, "content_hash")
assert (
hash_col is not None
and hash_col[0] == "character varying"
and hash_col[1] == "NO"
), "documents.content_hash (0001) must survive the upgrade"
summary = _column(db, "summary")
assert summary is not None and summary[0] == "text" and summary[1] == "YES", (
"documents.summary (phase 30) must survive the upgrade"
)
created = _column(db, "created_at")
assert created is not None and created[0] == "timestamp with time zone"
assert created[1] == "NO" and "now()" in str(created[2]), (
"documents.created_at (0020) must keep its `now()` server default"
)
manual = _column(db, "created_at_manual")
assert manual is not None and manual[0] == "boolean" and manual[1] == "NO"
assert manual[2] == "false", (
"documents.created_at_manual (0020) must keep its `false` default"
)
assert ("source", "path") in _unique_column_sets(db), (
"the (source, path) unique constraint must survive the upgrade"
)
finally:
_delete(db, pre_id)
def test_orm_image_fields_round_trip(db: Session, alembic: Config) -> None:
"""The ORM contract agrees with the column contract: a freshly
inserted ``Document`` WITHOUT the image fields reads ``is_image is
False`` / ``image_path is None`` (the text-doc default state), and
one WITH them round-trips the pair through a FRESH session."""
command.upgrade(alembic, "head")
text_doc = Document(
source=SOURCE,
path=PATH_TEXT,
full_path=f"/tmp/{SOURCE}/{PATH_TEXT}",
title="readme",
content="# readme\n",
content_hash="1" * 64,
)
image_doc = Document(
source=SOURCE,
path=PATH_IMAGE,
full_path=f"/tmp/{SOURCE}/{PATH_IMAGE}",
title="diagram",
content="A description of the diagram.",
content_hash="2" * 64,
is_image=True,
image_path="/srv/bor-images/diagram.png",
)
db.add(text_doc)
db.add(image_doc)
db.commit()
try:
with SessionLocal() as fresh:
reloaded_text = fresh.get(Document, text_doc.id)
assert reloaded_text is not None, "the text row must be readable"
assert reloaded_text.is_image is False, (
"an omitted is_image must read the False default"
)
assert reloaded_text.image_path is None, (
"an omitted image_path must read NULL"
)
reloaded_image = fresh.get(Document, image_doc.id)
assert reloaded_image is not None, "the image row must be readable"
assert reloaded_image.is_image is True
assert reloaded_image.image_path == "/srv/bor-images/diagram.png"
finally:
_delete(db, text_doc.id)
_delete(db, image_doc.id)
def test_downgrade_to_0021_drops_the_columns(db: Session, alembic: Config) -> None:
"""Downgrade 0022 → 0021: both image columns are gone (A13 — fully
reversible) while the row + its 0021 columns survive, and the rest
of the 0021 table contract (``content``, ``content_hash``,
``created_at``) is intact."""
command.upgrade(alembic, "head")
row = Document(
source=SOURCE,
path=PATH_IMAGE,
full_path=f"/tmp/{SOURCE}/{PATH_IMAGE}",
title="diagram",
content="A description of the diagram.",
content_hash="3" * 64,
is_image=True,
image_path="/srv/bor-images/diagram.png",
)
db.add(row)
db.commit()
try:
command.downgrade(alembic, "0021")
assert _version(db) == "0021"
assert _column(db, "is_image") is None, "is_image must be dropped"
assert _column(db, "image_path") is None, "image_path must be dropped"
surviving = db.execute(
text(
"SELECT source, path, title, content, content_hash, created_at"
" FROM documents WHERE id = :id"
),
{"id": row.id},
).fetchone()
assert surviving is not None, "the row must survive the column drops"
assert surviving[0] == SOURCE and surviving[1] == PATH_IMAGE
assert surviving[3] == "A description of the diagram."
assert surviving[4] == "3" * 64
assert surviving[5] is not None, "created_at must survive the drops"
assert ("source", "path") in _unique_column_sets(db), (
"the (source, path) unique constraint must survive the downgrade"
)
finally:
_delete(db, row.id)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_columns(db: Session, alembic: Config) -> None:
"""Downgrade to 0021, then upgrade back to 0022: both columns are
back with the full contract (``is_image`` BOOLEAN NOT NULL default
``false``; ``image_path`` TEXT NULLABLE no default)."""
command.downgrade(alembic, "0021")
command.upgrade(alembic, "0022")
assert _version(db) == "0022", "round-trip upgrade must land at 0022"
is_image = _column(db, "is_image")
assert is_image is not None, "documents.is_image must be back"
assert is_image[0] == "boolean", "is_image must be BOOLEAN after the round-trip"
assert is_image[1] == "NO", "is_image must be NOT NULL after the round-trip"
assert is_image[2] == "false", (
"is_image must still carry the `false` server default"
)
image_path = _column(db, "image_path")
assert image_path is not None, "documents.image_path must be back"
assert image_path[0] == "text"
assert image_path[1] == "YES"
assert image_path[2] is None, "image_path must still carry NO server default"
+4 -3
View File
@@ -216,10 +216,11 @@ def test_admin_semantic_fields_round_trip(client: TestClient, db: Session) -> No
def _config_keys() -> set[str]:
"""The /api/config key set after task 03: the five phase-39/59/62
keys — the retired CSS-file theming's ``theme`` key is gone."""
"""The /api/config key set after task 03 (phase 91): the retired
CSS-file theming's ``theme`` key is gone; phase 122 (task 01) added
the ``images`` flag — the six keys below are the contract."""
return {"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text"}
"images", "input_placeholder", "footer_text"}
def test_api_config_env_only_deployment_returns_env_strings(client: TestClient) -> None:
+7 -2
View File
@@ -179,6 +179,7 @@ def test_content_known_pair_maps_to_doc_content() -> None:
title="Deep Mark",
content="# Deep Mark\n\nbody",
content_hash="f" * 64,
is_image=False, # phase 122: the stub session never applies defaults
)
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC) # phase 106
@@ -190,11 +191,15 @@ def test_content_known_pair_maps_to_doc_content() -> None:
assert r.status_code == 200
body = r.json()
# Wire-additive (phase 106, task 05): the pre-date keys are all
# still there, joined by ``created_at``.
# still there, joined by ``created_at`` — and (phase 122, task 04)
# by ``is_image`` (ALWAYS present; text docs: false) while
# ``image_url`` is ABSENT (never null — the omission rule).
assert set(body) == {
"source", "path", "title", "format", "summary", "created_at",
"content", "indexed_at", "chunks",
"content", "indexed_at", "chunks", "is_image",
}
assert body["is_image"] is False
assert "image_url" not in body # absent — never null (text doc)
assert body["source"] == "Homelab"
assert body["path"] == "notes/deep mark.md"
assert body["title"] == "Deep Mark"
File diff suppressed because it is too large Load Diff
+11 -5
View File
@@ -970,12 +970,14 @@ def test_import_summary_log_line_includes_summary_counters(
"""PLAN §9 summary line: the phase-30 counters sit between
``embed_batches`` and ``formats``; the phase-118 backfill counter
sits between ``summary_errors`` and ``dates_updated``; the
phase-106 date-refresh counter sits before ``formats``."""
phase-106 date-refresh counter and the phase-122 ``images_failed``
counter sit after ``dates_updated``, before ``formats``."""
s = ImportSummary()
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
s.summaries, s.summary_errors = 2, 1
s.summary_backfilled = 1
s.dates_updated = 0
s.images_failed = 0
s.formats = {"md": 1, "yaml": 2}
with caplog.at_level(logging.INFO, logger="app.importer"):
s.log()
@@ -983,7 +985,7 @@ def test_import_summary_log_line_includes_summary_counters(
assert line == (
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 formats=yaml:2,md:1"
"dates_updated=0 images_failed=0 formats=yaml:2,md:1"
)
@@ -1103,13 +1105,17 @@ def test_no_progress_means_no_prewalk(
excluded: frozenset[str] = EXCLUDED_DIRS,
ignore: tuple[str, ...] = (),
include_hidden: bool = False,
image_extensions: frozenset[str] = frozenset(),
) -> list[Path]:
# Phase 89: the walker gained the ``ignore`` keyword; phase 105:
# the ``include_hidden`` flag — the sentinel accepts (and forwards)
# both to stay a drop-in.
# the ``include_hidden`` flag; phase 122: the ``image_extensions``
# set — the sentinel accepts (and forwards) all three to stay a
# drop-in.
nonlocal walk_calls
walk_calls += 1
return real_walker(r, extensions, excluded, ignore, include_hidden)
return real_walker(
r, extensions, excluded, ignore, include_hidden, image_extensions
)
monkeypatch.setattr(importer, "iter_importable_files", counting)
try:
+91 -4
View File
@@ -169,14 +169,21 @@ def test_summaries_present_and_absent() -> None:
one, two = _folder_nodes(source)
assert one.summary == "One desc."
assert two.summary is None
# File nodes carry no summary key at all (the 00_phase.md shape);
# since phase 106 they DO carry the creation date (``created_at``
# — the RAG view's ``Created`` column).
# File nodes carry no summary key at all on the WIRE (the
# 00_phase.md shape — the model_dump set check below is the wire
# pin); since phase 106 they DO carry the creation date
# (``created_at`` — the RAG view's ``Created`` column).
file = _file_nodes(one)[0]
assert set(file.model_dump()) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
assert "summary" not in file.__class__.model_fields
# Phase 122 (task 04): the image-affordance fields DO exist on the
# class now (image file nodes set them — the wire omission for text
# nodes is the ``KbTreeFile`` serializer, pinned by the model_dump
# set check above: a text node never leaks the three keys).
from app.schemas import KbTreeFile
assert {"is_image", "image_url", "summary"} <= set(KbTreeFile.model_fields)
def test_file_metadata_unchanged_in_tree() -> None:
@@ -546,3 +553,83 @@ def test_updated_at_does_not_leak_across_sources() -> None:
a, b = build_kb_tree(["A", "B"], rows, {})
assert a.updated_at == C0
assert b.updated_at == C3
# ---------------------------------------------------------------------------
# Phase 122 (task 04) — the image-docs affordance on tree file nodes
# ---------------------------------------------------------------------------
DOC_ID = "11111111-2222-3333-4444-555555555555"
def test_image_file_node_carries_the_affordance_and_text_node_unchanged() -> None:
"""The ``images`` map (``{(source, path): (doc_id, summary)}``) turns
a file node into the image-docs node: ``is_image`` true,
``image_url`` = the bytes route's path built from the mapped id,
``summary`` verbatim (the RAG view's thumbnail ``alt``). A file
node NOT in the map keeps the pre-phase wire shape byte-identically
(the three image keys are OMITTED, not false/null)."""
rows = [
("S", "one/a.md", "A", 1, T0, C0),
("S", "one/pic.png", "pic", 2, T0, C0),
]
images = {("S", "one/pic.png"): (DOC_ID, "A red square on a white background.")}
(source,) = build_kb_tree(["S"], rows, {}, images)
one = _folder_nodes(source)[0]
# File nodes keep the FULL source-relative path (the phase-97
# shape — the folder prefix rides the node).
files = {f.path: f for f in _file_nodes(one)}
# Text node: the pre-phase wire shape, byte-identical (no image keys).
assert set(files["one/a.md"].model_dump()) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
# Image node: the affordance rides the node.
dumped = files["one/pic.png"].model_dump()
assert dumped["kind"] == "file"
assert dumped["is_image"] is True
assert dumped["image_url"] == f"/api/documents/{DOC_ID}/image"
assert dumped["summary"] == "A red square on a white background."
# The catalogue fields still ride verbatim.
assert (dumped["title"], dumped["chunks"]) == ("pic", 2)
assert (dumped["created_at"], dumped["indexed_at"]) == (C0, T0)
def test_image_file_node_null_summary_and_missing_map_entry() -> None:
"""A mapped image node with a NULL summary (the fail-soft backfill
corner) keeps ``summary: null`` on the wire (meaningful — the alt
falls back client-side). A map entry that points at a NON-existent
(source, path) affects nothing (the builder only reads mapped keys
it meets in the catalogue rows)."""
rows = [
("S", "one/ghost.png", "ghost", 1, T0, C0),
("S", "one/other.md", "Other", 1, T0, C0),
]
images = {
("S", "one/ghost.png"): (DOC_ID, None),
("S", "one/absent.png"): (DOC_ID, "never matched"),
}
(source,) = build_kb_tree(["S"], rows, {}, images)
one = _folder_nodes(source)[0]
files = {f.path: f for f in _file_nodes(one)}
ghost = files["one/ghost.png"].model_dump()
assert ghost["is_image"] is True
assert ghost["summary"] is None # null stays (the alt fallback corner)
assert ghost["image_url"] == f"/api/documents/{DOC_ID}/image"
assert set(files["one/other.md"].model_dump()) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
def test_image_affordance_absent_without_the_map() -> None:
"""No map (the default) → every file node is the pre-phase shape,
even for ``.png`` paths: the fields are map-driven (the endpoint
composes the map from the ``is_image`` rows), never path-guessed —
a pre-phase KB serializes byte-identically."""
rows = [("S", "pic.png", "pic", 1, T0, C0)]
(source,) = build_kb_tree(["S"], rows, {})
(file,) = _file_nodes(source)
assert set(file.model_dump()) == {
"kind", "path", "title", "chunks", "created_at", "indexed_at"
}
+8 -3
View File
@@ -53,18 +53,23 @@ def test_app_config_dict_carries_the_docs_flag() -> None:
body = app_config(s)
# Phase 62 (task 01): the response grew to the phase-62 UI
# customization keys; phase 91 (task 03) deleted the retired
# CSS-file theming's ``theme`` key — the five keys below are the
# entire endpoint contract.
# CSS-file theming's ``theme`` key; phase 122 (task 01) added the
# ``images`` flag — the six keys below are the entire endpoint
# contract.
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"input_placeholder", "footer_text",
"images", "input_placeholder", "footer_text",
}
assert body["docs_repo_configured"] is s.docs_configured
assert body["docs_repo_configured"] is False
assert body["images"] is False # LOCKED A3: off by default
s2 = _settings(docs_repo="/srv/docs-repo")
assert app_config(s2)["docs_repo_configured"] is True
s3 = _settings(images=True)
assert app_config(s3)["images"] is True
# ---------------------------------------------------------------------------
# brand.js — the flag + promise are surfaced the way app_name is
+9 -7
View File
@@ -114,23 +114,25 @@ def test_no_literal_46rem_width_remains() -> None:
# ---------- the four reading-column selectors ----------
def test_the_four_reading_columns_use_the_token() -> None:
""".chat-shell, .shared-shell, .doc-md and
.doc-summary:has(+ .doc-md) each cap with
max-width: var(--chat-column) — and exactly those four rules use
the token for a max-width (no other selector)."""
def test_the_reading_columns_use_the_token() -> None:
""".chat-shell, .shared-shell, .doc-md, .doc-image (phase 122,
task 04 — the viewer's image block rides the SAME reading column
as its .doc-md sibling) and .doc-summary:has(+ .doc-md) each cap
with max-width: var(--chat-column) — and exactly those five rules
use the token for a max-width (no other selector)."""
css = _css()
for selector in (
".chat-shell",
".shared-shell",
".doc-md",
".doc-image",
".doc-summary:has(+ .doc-md)",
):
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
f"{selector} must cap with max-width: var(--chat-column)"
)
assert css.count("max-width: var(--chat-column)") == 4, (
"exactly the four reading-column selectors use the token"
assert css.count("max-width: var(--chat-column)") == 5, (
"exactly the reading-column selectors use the token"
)