phase: 123_chat_image_questions
All gates green. Verification complete. **Phase 123 — final verification pass (all 4 tasks already in `complete/`)** - Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer. - `uv run pytest` → **2796 passed**, exit 0 (unit + integration). - `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%). - `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation. - `uv run ruff check . && uv run pyright` → clean (0 errors). **Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run). No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`.
This commit is contained in:
+128
-2
@@ -179,6 +179,27 @@ as ``reasoning_content`` on the assistant message (the preserve-
|
||||
thinking wire convention, A4). The per-turn log line records
|
||||
``history_msgs=N`` after ``kb_chars=N`` (0 when the request carries no
|
||||
history — the two-message request stays byte-identical).
|
||||
|
||||
Question images (phase 123, TODO L6; LOCKED A5/A7): the request may
|
||||
attach ONE image to the CURRENT question — ``ChatRequest.image`` is the
|
||||
STORED path from ``POST /api/chat-images`` (``app.api.chat_images``),
|
||||
never a data URL. The turn validates it BEFORE any model call (the
|
||||
toggle gate — ``settings.images`` false settles the phase-114 hinted
|
||||
error frame; the stale-file gate — the stored bytes deleted out-of-
|
||||
band settles the same frame shape) and, when valid, the user message's
|
||||
content becomes the multimodal list ``[{type: "text", …}, {type:
|
||||
"image_url", image_url: {url: <data URL>}}]`` — built by
|
||||
:func:`build_user_content` at BOTH construction sites (this module's
|
||||
deflected-branch ``messages`` list and the grounded branch's
|
||||
``run_agent`` call, which builds its own ``[system, *history, user]``
|
||||
— the flow is pinned in ``app.rag.agent.run_agent``). The data URL is
|
||||
the shared :func:`app.rag.summarizer.image_data_url` (one
|
||||
construction, both call sites — the phase-122 describe path).
|
||||
``image=None`` keeps the plain-string content byte-identical to
|
||||
pre-phase. A question image is turn-local: it is NEVER indexed as a
|
||||
document, and prior turns' images are never replayed into the model's
|
||||
history (``history_to_messages`` is unchanged — the text of a prior
|
||||
turn that had an image stands alone, LOCKED A7).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -188,6 +209,7 @@ import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
@@ -229,6 +251,11 @@ from app.rag.retriever import (
|
||||
)
|
||||
from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter
|
||||
from app.rag.suggestions import derive_suggestions
|
||||
from app.rag.summarizer import (
|
||||
IMAGE_FALLBACK_MIME,
|
||||
IMAGE_MIMES,
|
||||
image_data_url,
|
||||
) # phase 123: the phase-122 mime map + the shared data-URL helper
|
||||
from app.schemas import (
|
||||
ChatDoneEvent,
|
||||
ChatErrorEvent,
|
||||
@@ -281,6 +308,37 @@ def sse_event(payload: dict[str, Any]) -> str:
|
||||
return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"
|
||||
|
||||
|
||||
def build_user_content(
|
||||
message: str,
|
||||
image_path: str | None,
|
||||
settings: Settings,
|
||||
) -> str | list[dict[str, Any]]:
|
||||
"""The current turn's user message content (phase 123, task 01).
|
||||
|
||||
*image_path* ``None`` (every text-only question) → the plain
|
||||
question string — byte-identical to pre-phase-123 (the multimodal
|
||||
branch is inert). Set → the OpenAI-compatible multimodal content
|
||||
list: the text part + the image part, a data URL built server-side
|
||||
from the stored bytes (``settings.chat_image_dir`` + the path's
|
||||
filename) and the phase-122 ``IMAGE_MIMES`` map (one map, one
|
||||
truth) through the shared :func:`app.rag.summarizer.image_data_url`
|
||||
helper (one construction, both call sites — the phase-122 describe
|
||||
path). The caller has already validated the path shape (the
|
||||
``ChatRequest.image`` schema guard) and the file's existence (the
|
||||
pre-stream turn gate). The image applies to the CURRENT turn only
|
||||
(LOCKED A7) — prior turns' images are never replayed.
|
||||
"""
|
||||
if image_path is None:
|
||||
return message
|
||||
filename = image_path.rsplit("/", 1)[-1]
|
||||
data = (Path(settings.chat_image_dir).expanduser() / filename).read_bytes()
|
||||
mime = IMAGE_MIMES.get(Path(filename).suffix.lower(), IMAGE_FALLBACK_MIME)
|
||||
return [
|
||||
{"type": "text", "text": message},
|
||||
{"type": "image_url", "image_url": {"url": image_data_url(data, mime)}},
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurnPlan:
|
||||
"""What one chat turn sends to the LLM and reports on ``done``."""
|
||||
@@ -494,6 +552,59 @@ async def chat(
|
||||
retries_used = 0 # phase 67: LLM requests restarted this turn (log line)
|
||||
try:
|
||||
settings = get_settings()
|
||||
# Phase 123 (TODO L6, task 01): the question's attached
|
||||
# image — validated BEFORE any model call (the embed is
|
||||
# a model call): a rejected turn calls nothing and
|
||||
# settles with the phase-114 error frame (the existing
|
||||
# error-path convention — no ``done``, no ``query_log``
|
||||
# row, no persisted record; the question is not saved).
|
||||
# Two gates, in order:
|
||||
# 1. the ``images`` toggle (``BOR_IMAGES``) — off with
|
||||
# an image set: the HINTED frame (the client's
|
||||
# banner shows the hint in place of its default
|
||||
# reachability copy, phase 114);
|
||||
# 2. the stored file — the schema already pinned the
|
||||
# path shape, but the file may have been deleted
|
||||
# out-of-band (the stale-path edge): the same frame
|
||||
# shape, no hint (the banner's default copy is the
|
||||
# honest fallback — there is nothing to point at).
|
||||
if request.image is not None:
|
||||
if not settings.images:
|
||||
logger.warning(
|
||||
"chat: image question rejected (images toggle "
|
||||
"off) question=%r image=%r",
|
||||
request.message,
|
||||
request.image,
|
||||
)
|
||||
settled = True # terminal: the error frame settles the turn
|
||||
yield sse_event(
|
||||
ChatErrorEvent(
|
||||
detail="Image support is turned off on this server.",
|
||||
hint=(
|
||||
"Enable BOR_IMAGES in the server's .env "
|
||||
"(and restart) to ask with an image."
|
||||
),
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
image_file = (
|
||||
Path(settings.chat_image_dir).expanduser()
|
||||
/ request.image.rsplit("/", 1)[-1]
|
||||
)
|
||||
if not image_file.is_file():
|
||||
logger.warning(
|
||||
"chat: image question rejected (stored file "
|
||||
"missing) question=%r image=%r",
|
||||
request.message,
|
||||
request.image,
|
||||
)
|
||||
settled = True # terminal: the error frame settles the turn
|
||||
yield sse_event(
|
||||
ChatErrorEvent(
|
||||
detail="That image is no longer available."
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
# Phase 74 (TODO L4): the client's prior turns, mapped ONCE
|
||||
# per turn — trimmed newest-first against the settings
|
||||
# budgets, assistant turns carrying their prior thinking as
|
||||
@@ -640,10 +751,21 @@ async def chat(
|
||||
).model_dump()
|
||||
)
|
||||
return
|
||||
# Phase 123 (task 01): the user message's content — the
|
||||
# plain question string (``image=None`` — byte-identical
|
||||
# to pre-phase) or the multimodal content list (text
|
||||
# part + image_url data URL; the image was validated
|
||||
# above). BOTH construction sites use the same build:
|
||||
# this deflected-branch list and the grounded branch's
|
||||
# ``run_agent`` call below (run_agent builds its own
|
||||
# ``[system, *history, user]`` — pinned there).
|
||||
user_content: str | list[dict[str, Any]] = build_user_content(
|
||||
request.message, request.image, settings
|
||||
)
|
||||
messages: list[dict[str, Any]] = [
|
||||
{"role": "system", "content": plan.system_prompt},
|
||||
*hist, # phase 74: the trimmed prior turns (empty by default)
|
||||
{"role": "user", "content": request.message},
|
||||
{"role": "user", "content": user_content},
|
||||
]
|
||||
|
||||
# 3. Stream the answer (grounded, or an honest deflection).
|
||||
@@ -689,7 +811,11 @@ async def chat(
|
||||
llm,
|
||||
db_factory, # SEC-14-04: session factory, not a long-lived session
|
||||
system_prompt=plan.system_prompt,
|
||||
user_message=request.message,
|
||||
# Phase 123 (task 01): the plain question string
|
||||
# (image=None) or the multimodal content list
|
||||
# (validated above) — run_agent builds its own
|
||||
# user message from this value (see its docstring).
|
||||
user_message=user_content,
|
||||
seed_docs=plan.suggested_docs, # phase 118 (A4): the suggestion tier
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
"""Question-image upload/serve pair (phase 123, TODO L6 — attach an
|
||||
image to a question).
|
||||
|
||||
A user attaches ONE image to a chat question (LOCKED A5): the bytes are
|
||||
stored server-side under ``chat_image_dir/<uuid4().hex>.<ext>`` — NOT
|
||||
base64 in saved/shared chats. ``POST /api/chat-images`` answers
|
||||
``{"path": "/api/chat-images/<uuid>.<ext>"}`` and that path is what
|
||||
``ChatRequest.image`` / ``ChatMessage.image`` carry (a stored PATH,
|
||||
never a data URL — the upload endpoint owns the size/mime
|
||||
enforcement). ``GET /api/chat-images/{filename}`` serves the bytes back
|
||||
so the user bubble, the refreshed page, and the shared chat can render
|
||||
the attachment. A question image is NEVER indexed as a document (no
|
||||
importer call) — it is turn-local storage, not a source.
|
||||
|
||||
Auth posture: the upload is user-gated exactly like the chat turn it
|
||||
feeds (``require_user`` — the question itself is user-gated, and an
|
||||
anonymous 10 MiB disk-fill would be a DoS); the serve route is PUBLIC
|
||||
like saved-chat content (phase 55 A1 — a saved chat's id is already its
|
||||
credential, and the image is part of that content; the filename is an
|
||||
unguessable ``uuid4().hex`` — no enumeration value).
|
||||
|
||||
The extension is the source of truth (the Content-Type header is a
|
||||
hint — the archive-uploader precedent): it must be in the phase-122
|
||||
image set (``settings.image_extension_set`` — the same frozenset the
|
||||
image-document walk uses), lowercased; total bytes are streamed with a
|
||||
``chat_image_max_mb`` cap (413, fixed detail naming the cap — never
|
||||
echoing the filename).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, UploadFile
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_user
|
||||
from app.rag.summarizer import IMAGE_FALLBACK_MIME, IMAGE_MIMES
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(tags=["chat-images"])
|
||||
|
||||
#: Stream receive chunk (the git-sources upload's 1 MiB pattern).
|
||||
_STREAM_CHUNK = 1 << 20
|
||||
|
||||
#: Stored question-image filename guard: ``<uuid4().hex>.<ext>`` — 32
|
||||
#: hex chars + one of the six image extensions (the ``ChatRequest.image``
|
||||
#: path pattern's filename part, kept in lockstep with it). Anything
|
||||
#: else 404s — no path traversal by construction (the route parameter
|
||||
#: cannot carry a ``/`` and the regex rejects everything but the
|
||||
#: upload endpoint's own naming).
|
||||
_CHAT_IMAGE_FILENAME_RE = re.compile(r"^[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$")
|
||||
|
||||
|
||||
@router.post("/chat-images")
|
||||
async def upload_chat_image(
|
||||
file: UploadFile = File(...), # noqa: B008
|
||||
_user: None = Depends(require_user), # noqa: B008
|
||||
) -> dict[str, str]:
|
||||
"""Store one question image (phase 123, task 01; LOCKED A5).
|
||||
|
||||
Gates, in order:
|
||||
|
||||
1. **extension** — the file name's extension (lowercased) must be in
|
||||
the phase-122 image set (``settings.image_extension_set``); the
|
||||
Content-Type header is a hint, never a source of truth (the
|
||||
archive-uploader precedent). A missing/unknown extension is a
|
||||
422 naming the accepted set — a fixed detail, no filename echo.
|
||||
2. **size** — the bytes are streamed (1 MiB chunks) with the
|
||||
``chat_image_max_mb`` cap; over-cap is a 413 naming the cap
|
||||
(fixed detail — never echoing the filename), the temp file is
|
||||
removed, and nothing is stored.
|
||||
|
||||
The file lands as ``<uuid4().hex>.<ext>`` in ``chat_image_dir``
|
||||
(created on demand; a dotfile temp is renamed into place, so a
|
||||
failed/partial receive never leaves a servable-looking file). The
|
||||
response is the served path — ``{"path":
|
||||
"/api/chat-images/<uuid>.<ext>"}`` — the value ``ChatRequest.image``
|
||||
accepts (never a data URL, never the on-disk location).
|
||||
"""
|
||||
settings = get_settings()
|
||||
filename = file.filename or ""
|
||||
ext = Path(filename).suffix.lower()
|
||||
if ext not in settings.image_extension_set:
|
||||
accepted = ", ".join(sorted(settings.image_extension_set))
|
||||
raise HTTPException(
|
||||
status_code=422, detail=f"only {accepted} images are accepted"
|
||||
)
|
||||
|
||||
root = Path(settings.chat_image_dir).expanduser()
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
name = f"{uuid.uuid4().hex}{ext}"
|
||||
temp = root / f".{name}.upload"
|
||||
max_bytes = settings.chat_image_max_mb * 1024 * 1024
|
||||
total = 0
|
||||
try:
|
||||
# Stream with the cap — the dotfile temp is hidden from any
|
||||
# listing of the store dir (the git-sources upload pattern).
|
||||
with open(temp, "wb") as out:
|
||||
while chunk := await file.read(_STREAM_CHUNK):
|
||||
total += len(chunk)
|
||||
if total > max_bytes:
|
||||
raise HTTPException(
|
||||
status_code=413,
|
||||
detail=(
|
||||
f"the image exceeds the "
|
||||
f"{settings.chat_image_max_mb} MB limit"
|
||||
),
|
||||
)
|
||||
out.write(chunk)
|
||||
temp.rename(root / name)
|
||||
except BaseException:
|
||||
# A failed receive (413, broken pipe, cancellation) leaves no
|
||||
# file behind — the final name was never created.
|
||||
temp.unlink(missing_ok=True)
|
||||
raise
|
||||
logger.info(
|
||||
"chat-image: uploaded name=%s bytes=%d", name, total
|
||||
)
|
||||
return {"path": f"/api/chat-images/{name}"}
|
||||
|
||||
|
||||
@router.get("/chat-images/{filename}", response_class=FileResponse)
|
||||
def serve_chat_image(filename: str) -> FileResponse:
|
||||
"""Serve one stored question image (phase 123, task 01).
|
||||
|
||||
PUBLIC (no auth dependency) — like saved-chat content: the saved
|
||||
chat's id is already its credential (phase 55 A1), the image is part
|
||||
of that content, and the filename is an unguessable ``uuid4().hex``
|
||||
(no enumeration value).
|
||||
|
||||
Every non-servable case is a 404 with the same fixed detail — a
|
||||
filename that does not match ``<uuid-hex>.<ext>`` (the regex guard:
|
||||
no path traversal by construction, no 422 that would hint at
|
||||
accepted shapes) and a matching name whose file is missing (the
|
||||
stale-path edge — the file was deleted out-of-band). Servable files
|
||||
stream the exact bytes with the phase-122 ``IMAGE_MIMES``
|
||||
``Content-Type`` (one map, one truth with the describe call's
|
||||
data-URL mime; the six-extension guard makes the fallback
|
||||
unreachable) and ``Cache-Control: private, max-age=3600`` (the
|
||||
phase-122 serve-route convention — the bytes are content-hashed
|
||||
uuids, bustable by re-upload).
|
||||
"""
|
||||
if _CHAT_IMAGE_FILENAME_RE.fullmatch(filename) is None:
|
||||
raise HTTPException(status_code=404, detail="chat image not found")
|
||||
path = Path(get_settings().chat_image_dir).expanduser() / filename
|
||||
if not path.is_file():
|
||||
raise HTTPException(status_code=404, detail="chat image not found")
|
||||
media_type = IMAGE_MIMES.get(path.suffix.lower(), IMAGE_FALLBACK_MIME)
|
||||
return FileResponse(
|
||||
path,
|
||||
media_type=media_type,
|
||||
headers={"Cache-Control": "private, max-age=3600"},
|
||||
)
|
||||
Reference in New Issue
Block a user