brain-of-reese/app/api/docs.py

"""GET /api/docs — the indexed document list (feeds the Sources page).

GET /api/documents/content — one indexed document's full content (feeds the
clickable document viewer, phase 10). DB-only by design: the (source, path)
pair is looked up as a row, so there is no filesystem access and no
path-traversal surface — ``../``-style values simply aren't rows (→ 404).

PATCH /api/documents/summary — the admin summary editor (phase 57):
update or clear ``documents.summary`` and re-embed the ``is_summary``
chunk (embed first, mutate second — a failed LLM call leaves the row and
chunk untouched; the content chunks are never re-embedded, D4).

PATCH /api/documents/date — the admin document-date editor (phase 106,
D7): set the owner's corrected ``documents.created_at`` (normalized
through ``app.rag.doc_dates.normalize_doc_date`` — a manually set future
date folds to today, D3) + flag it manual, or clear the manual flag
(null date — the stored date stands until the next sync). A pure DB
write: NO LLM/embedding call — a date is never embedded (the
deliberate contrast with the phase-57 ``is_summary`` re-embed).

GET /api/docs/tree — the admin's KB tree (phase 97, task 02; depth-
limited since phase 167, task 02): the same drill-down tree the agent's
``ls`` walks, with the file metadata the RAG view's rows and stat cards
need. The default ``depth=1`` ships the sources plus ONE level with
server-computed ``totals`` — a node whose children were not built
carries ``truncated: true`` (the TODO L5 2 MB payload fix; ``GET
/api/docs`` and the agent's ``ls`` path are untouched).

GET /api/docs/tree/node — ONE tree node's direct children (phase 167,
task 03, D3): the RAG view's per-level drill fetch — the same catalogue
composition as ``/api/docs/tree`` (ONE catalogue query per drill, no
N+1) resolved to a single ``KbTreeSource`` (``path=""``, the source
root) or ``KbTreeFolder``. Admin-gated like the tree; 404
``{"detail": "unknown node"}`` for a source that is neither registered
nor indexed, or a folder failing the phase-94 existence rule. Row-only
like every other route here: no request value ever reaches a
filesystem path — ``../``-style values simply are not folders (the
phase-10 traversal argument, reused).

Revalidation (phase 167, task 03, LOCKED A4 — a scoped extension of
A19): the three catalogue READ routes (``/api/docs``,
``/api/docs/tree``, ``/api/docs/tree/node``) serialize once and carry a
content-exact weak ``ETag`` + ``Cache-Control: private,
must-revalidate`` (the :mod:`app.core.http_cache` helpers), so a repeat
load costs a 304 with no body. The tag is a hash of the served bytes —
NOT ``sources_meta.version`` (bumping that would mark saved chats
stale, and the admin PATCHes do not bump it). ``sources_meta``, the
SSE stream and the mutating surface are untouched.

GET /api/documents/{doc_id}/image — the phase-122 (task 04) image
BYTES route: the persistent copy behind an ``is_image`` document's
``image_path``, served with the extension's ``Content-Type`` (the
``app.rag.summarizer.IMAGE_MIMES`` map — one map, one truth) and a
``Cache-Control: private, max-age=3600`` header (the bytes are
content-hashed — long enough, bustable by re-upload), plus — for the
vector ``image/svg+xml`` entry only, phase 181 LOCKED A3 — a
``Content-Security-Policy: default-src 'none'; sandbox`` so a DIRECT
navigation to the URL cannot run script from stored SVG markup (the
``SecurityHeadersMiddleware`` preserves that inner CSP; raster
responses gain nothing). PUBLIC (phase 124 — the phase-123 chat-image
precedent: the image is part of a shared chat's rendered content, and
the shared chat — the app's ONLY anonymous surface — must render the
answer figure faithfully for
guests; the unguessable ``uuid4`` document id IS the credential, no
enumeration value): a missing doc, a non-image doc, a doc whose
``image_path`` is NULL, or a row whose copy was lost all map to 404
``document not found`` (the router's unknown-document shape —
traversal/UUID-guessing has no filesystem surface to hit). The
document's TEXT (the content endpoint) stays user-gated — only the
bytes are public.

GET /api/documents/{doc_id}/pages/{n} — the phase-148 (task 02) PDF
page BYTES route: ONE rendered page PNG (``<doc-id>-p{N}.png`` in the
importer's ``pdf_page_dir`` copy — task 01) of a page-QUALIFYING
document (the phase-133 fallback class or the phase-140 pictorial
class). PUBLIC — the image route's phase-124 A4 trust model verbatim
(a shared chat can cite a page figure, so the bytes serve
ANONYMOUSLY; the unguessable ``uuid4`` document id IS the credential —
the document's TEXT stays user-gated). ``image/png`` +
``Cache-Control: private, max-age=3600`` (the image route's header);
404 ``document not found`` for EVERY non-servable case (an
unparseable id, an unknown doc, ``page_count`` NULL, ``n`` outside
``1..page_count``, or a copy lost on disk). The path is built from the
ROW's id + ``n`` only — no request value reaches a filesystem path.

PATCH /api/folders/summary — the admin folder-description editor
(phase 97, task 03): update / create / clear a stored
``folder_summaries`` row, marking every non-empty save
``manually_edited`` (from this point on the sync-time generator skips
the row and never prunes it — task 01). A pure DB write: NO
LLM/embedding call — a folder description is never embedded (no
chunk, no retrieval role beyond the ``ls`` line), the deliberate
contrast with the phase-57 ``is_summary`` re-embed above.
"""
from __future__ import annotations

import uuid
from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from pathlib import Path

from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session

from app.api.sync import _sanitize_error
from app.config import get_settings
from app.core.auth import require_admin, require_user
from app.core.http_cache import etag_matches, revalidate_headers, weak_etag
from app.db import get_db
from app.models import Chunk, Document, FolderSummary
from app.rag.agent import list_source_names
from app.rag.doc_dates import normalize_doc_date
from app.rag.folder_summaries import MIN_DOCS_PER_FOLDER, folder_of
from app.rag.git_sources import origin_for_source
from app.rag.importer import match_extension
from app.rag.llm import EmbeddingError, LLMClient
from app.rag.summarizer import IMAGE_FALLBACK_MIME, IMAGE_MIMES
from app.schemas import (
    DateResult,
    DateUpdate,
    DocContent,
    DocList,
    DocSummary,
    FolderSummaryResult,
    FolderSummaryUpdate,
    KbTree,
    KbTreeFile,
    KbTreeFolder,
    KbTreeSource,
    KbTreeTotals,
    SummaryResult,
    SummaryUpdate,
)

router = APIRouter(tags=["kb"])

#: The vector media type (phase 181, task 03) — read out of
#: :data:`app.rag.summarizer.IMAGE_MIMES` rather than restated, so the
#: guard below can never drift from the ``Content-Type`` the route
#: sends for an ``.svg`` (one map, one truth).
SVG_MEDIA_TYPE = IMAGE_MIMES[".svg"]

#: The CSP an SVG bytes response carries (phase 181, task 03, LOCKED
#: A3). ``sandbox`` — no scripting, and the document gets an OPAQUE
#: origin (no same-origin fetch/cookie/form capability);
#: ``default-src 'none'`` — no subresource of any kind. Scoped to the
#: vector type ON PURPOSE: an ``<img>`` never runs SVG script, but this
#: URL is directly navigable, and the bytes come from an untrusted
#: source tree, so a direct visit must not become a script-capable
#: same-origin document. Raster types are untouched (their header set
#: stays byte-identical). See :func:`get_document_image`.
SVG_SANDBOX_CSP = "default-src 'none'; sandbox"


def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
    """Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
    ``notes/deep.Markdown`` → ``markdown``) — returned **unconditionally**
    for a non-empty suffix (display never depends on the import list — an
    out-of-scope ``readme.rst`` still badges ``rst``); ``text`` when the
    path has no suffix **and its name is not a configured token** (phase
    102: a suffix-less ``Dockerfile`` badges ``dockerfile`` when its
    lowercased full filename is one of *extensions* — the importer's
    :func:`app.rag.importer.match_extension` rule). The value shown in
    the viewer's format badge; the default empty *extensions* keeps the
    pre-phase-102 suffix-only result for every path (byte-identical).
    """
    p = Path(path)
    suffix = p.suffix.lower().lstrip(".")
    if suffix:
        return suffix
    matched = match_extension(p, extensions)
    return matched if matched is not None else "text"


@router.get("/docs", response_model=DocList)
def list_indexed_documents(
    request: Request,
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> Response:
    """All indexed documents with per-document chunk counts.

    Admin-only (phase 16 — the catalog is what the admin sign-in gates;
    the document viewer below is user-gated since phase 79, the shared
    chats being the only anonymous surface). Anonymous callers get 403
    ``admin only`` and the Sources page renders its sign-in gate instead.
    An empty list means the knowledge base has not been imported yet —
    the Sources page renders its designed empty state in that case.

    Phase 167 (task 03, D3/A4): the response carries the content-exact
    weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair
    (see :mod:`app.core.http_cache`) — the client is ``fetch``, which
    serves a matched conditional request from its cache transparently,
    and ``must-revalidate`` guarantees a sync is seen on the next load
    (the tag changes because the body changes). The decorator keeps
    ``response_model=DocList`` for the OpenAPI schema only — a returned
    :class:`~fastapi.responses.Response` short-circuits FastAPI's
    serialization, so the model is dumped exactly ONCE here.
    """
    rows = db.execute(
        select(
            Document.id,
            Document.source,
            Document.path,
            Document.title,
            func.count(Chunk.id).label("chunks"),
            Document.created_at,
            Document.indexed_at,
        )
        .outerjoin(Chunk, Chunk.document_id == Document.id)
        .group_by(
            Document.id,
            Document.source,
            Document.path,
            Document.title,
            Document.created_at,
            Document.indexed_at,
        )
        .order_by(Document.source, Document.path)
    ).all()
    model = DocList(
        documents=[
            DocSummary(
                id=str(row.id),
                source=row.source,
                path=row.path,
                title=row.title,
                chunks=row.chunks,
                created_at=row.created_at.isoformat(),
                indexed_at=row.indexed_at.isoformat(),
            )
            for row in rows
        ]
    )
    return _revalidated_json(request, model)


@router.get("/documents/content", response_model=DocContent)
def get_document_content(
    source: str,
    path: str,
    db: Session = Depends(get_db),  # noqa: B008
    _user: None = Depends(require_user),  # noqa: B008  # phase 79: admin or live token
) -> DocContent:
    """Full content of one indexed document, looked up by ``(source, path)``.

    Stateless (A10) and database-only: unknown pairs — including traversal
    strings such as ``../../etc/passwd`` — are just non-existent rows and
    map to 404 ``{detail: "document not found"}``.

    User-gated (phase 79 — SUPERSEDES the phase-16 "deliberately PUBLIC
    (soft rule)" note, owner decision 2026-08-22): the viewer content is
    token-or-admin like the rest of the app surface — chat cites
    documents and a signed-in user (admin or token holder) opens a cited
    document by direct URL. The ONLY anonymous content left is the
    shared chats.

    Phase 145 (task 03): the payload carries the document's git origin
    when one resolves — ``origin_url`` (the sanitized repository URL)
    + ``origin_branch`` (only when the row pins one), computed by the
    SAME :func:`app.rag.git_sources.origin_for_source` resolver the
    agent's read branch uses (one mapping for the whole app: the
    source's directory basename → the ``kind='git'`` row whose URL
    basename matches). ``None`` — a non-git source, no row, or an
    ambiguous basename — keeps the payload byte-identical to pre-phase
    (both fields are dropped by the serializer, never ``null``).

    Phase 163 (task 01): a page-qualifying PDF's payload additionally
    carries ``page_strip_mode`` — ``"primary"`` (the phase-133 fallback
    class: the strip IS the view) or ``"additive"`` (the phase-140
    pictorial class: the text + Summary view with the strip behind a
    Text/Pages toggle). Both strip keys are built off ``page_count`` and
    dropped when it is NULL, so every non-qualifying payload stays
    byte-identical to pre-phase (and a pre-phase row, whose mode column
    is NULL, keeps the strip-only DOM the viewer assumes for an absent
    mode).
    """
    row = db.execute(
        select(Document, func.count(Chunk.id).label("chunks"))
        .outerjoin(Chunk, Chunk.document_id == Document.id)
        .where(Document.source == source, Document.path == path)
        .group_by(Document.id)
    ).first()
    if row is None:
        raise HTTPException(status_code=404, detail="document not found")
    doc, chunks = row
    # Phase 145 (task 03): the document's git origin — the SAME resolver
    # the read branch uses (one mapping for the whole app). ``None``
    # (non-git source, no row, ambiguous basename) → both fields are
    # dropped on serialization, never ``null``.
    origin = origin_for_source(db, doc.source)
    # Phase 122 (task 04): the image affordance — ``is_image`` is ALWAYS
    # present on the wire (text docs: false — the one new key); for an
    # image doc, ``image_url`` is the bytes route's path (absent for
    # text docs, and for an image row whose copy path is NULL — never
    # null, the ``DocContent`` omission rule).
    return DocContent(
        source=doc.source,
        path=doc.path,
        title=doc.title,
        format=doc_format(doc.path, get_settings().import_extension_set),
        summary=doc.summary,
        created_at=doc.created_at.isoformat(),
        content=doc.content,
        indexed_at=doc.indexed_at.isoformat(),
        chunks=chunks,
        is_image=doc.is_image,
        image_url=(
            f"/api/documents/{doc.id}/image" if doc.is_image and doc.image_path else None
        ),
        origin_url=origin[0] if origin else None,
        origin_branch=origin[1] if origin else None,
        # Phase 148 (task 02): the rendered-page BYTES routes for a
        # page-qualifying PDF — the exact 1-based list over
        # ``1..page_count`` (the viewer's strip source, task 03; a
        # per-page render failure leaves a GAP the route 404s). ABSENT
        # (never null — the ``image_url`` omission rule) for every
        # other document, so its payload is byte-identical to
        # pre-phase.
        page_urls=(
            [f"/api/documents/{doc.id}/pages/{i}" for i in range(1, doc.page_count + 1)]
            if doc.page_count is not None
            else None
        ),
        # Phase 163 (task 01): the strip CLASS (``"primary"`` = the
        # fallback class, the strip IS the view; ``"additive"`` = the
        # pictorial class, text + a Text/Pages toggle). The whole strip
        # block keys off ``page_count`` — a row with a stray mode but no
        # strip serializes NEITHER key — and a pre-phase row (mode NULL)
        # serializes the key ABSENT (never null), which the viewer reads
        # as ``"primary"`` (task 04).
        page_strip_mode=(
            doc.page_strip_mode if doc.page_count is not None else None
        ),
    )


@router.get("/documents/{doc_id}/image", response_class=FileResponse)
def get_document_image(
    doc_id: str,
    db: Session = Depends(get_db),  # noqa: B008
) -> FileResponse:
    """The phase-122 (task 04) image BYTES route — the persistent copy
    behind an ``is_image`` document's ``image_path``.

    PUBLIC (phase 124, LOCKED A4 — "visible … on the shared page"): a
    guest opening a shared chat sees the brain answer's image figure,
    so the bytes must serve ANONYMOUSLY. The trust model is the
    phase-123 chat-image precedent (the question attachment travels in
    the SAME shared content): the document id is an unguessable
    server-minted ``uuid4`` — IT IS the credential, with no
    enumeration value — exactly like the saved-chat id (phase 55) and
    the chat-image filename (phase 123). (The phase-122/79
    ``require_user`` posture was set before the image became part of a
    shared chat's rendered content; the shared page's figure — built
    on this route per the phase-124 design — 401'd and collapsed for
    every guest, contradicting the owner-locked A4. The document's
    TEXT stays user-gated: only the bytes are public.)

    404 ``document not found`` (the router's unknown-document shape) in
    every non-servable case — a missing id (an unparseable string maps
    here too, not to a 422 — a guessed id is an unknown document), a
    text doc, an image doc whose ``image_path`` is NULL, or a row whose
    copy was lost on disk (defensive — the row exists, the bytes
    don't). There is no path parameter to a filesystem value: the path
    comes from the ROW (the importer's ``image_dir`` copy), so there is
    no traversal surface.

    Servable rows stream the exact bytes with the extension's
    ``Content-Type`` (the ``IMAGE_MIMES`` map — one map, one truth with
    the describe call's data-URL mime; an unexpected extension takes
    ``application/octet-stream``) and ``Cache-Control: private,
    max-age=3600`` (the bytes are content-hashed — long enough to be
    useful, bustable by re-upload).

    Phase 181 (task 03, LOCKED A3): an ``image/svg+xml`` response — the
    vector entry of that same map, derived from the ROW's extension and
    NEVER from a client value — carries ONE extra header,
    :data:`SVG_SANDBOX_CSP` (``default-src 'none'; sandbox``). An
    ``<img>`` can never execute an SVG's script, which is why the
    figure surfaces are safe as they stand; but this URL is also
    DIRECTLY NAVIGABLE (a guest who pastes it into the address bar),
    and without the policy a stored SVG would render as a
    script-capable SAME-ORIGIN document — the ingestion path takes
    untrusted files from a source tree, so that is a stored-XSS seam
    scoped to the app's origin. ``sandbox`` blocks scripting AND makes
    the response an opaque origin (no same-origin fetches, cookies or
    form submission from the document); ``default-src 'none'`` blocks
    every subresource load, so the SVG cannot pull a tracker or a
    remote image either. Deliberately NOT applied to the raster types:
    their header set stays byte-identical (a PNG needs no policy
    change, and adding one would risk the phase-123/124 image
    surfaces), and no ``Content-Disposition: attachment`` is set — the
    graphic must render inline, and ``sandbox`` is the stronger, less
    intrusive control. The route sets the value on the response
    itself; :class:`app.core.security_headers.SecurityHeadersMiddleware`
    PRESERVES a CSP an inner layer already set, so this is the policy
    the browser sees (the app-wide ``CSP`` string would be wrong here —
    ``default-src 'self'`` would grant the SVG document the app's own
    origins), while the middleware's global
    ``X-Content-Type-Options: nosniff`` still applies.
    """
    try:
        uid = uuid.UUID(doc_id)
    except ValueError:
        raise HTTPException(status_code=404, detail="document not found") from None
    doc = db.scalar(select(Document).where(Document.id == uid))
    if doc is None or not doc.is_image or not doc.image_path:
        raise HTTPException(status_code=404, detail="document not found")
    image_file = Path(doc.image_path)
    if not image_file.is_file():
        # Defensive: the row exists but the copy was lost (the owner
        # cleaned the image dir, the disk was wiped) — the viewer's
        # onerror fallback renders the "image unavailable" note.
        raise HTTPException(status_code=404, detail="document not found")
    media_type = IMAGE_MIMES.get(image_file.suffix.lower(), IMAGE_FALLBACK_MIME)
    headers = {"Cache-Control": "private, max-age=3600"}
    if media_type == SVG_MEDIA_TYPE:
        # Phase 181 (LOCKED A3): vector bytes only — see the route
        # docstring. Raster responses keep the header set they had.
        headers["Content-Security-Policy"] = SVG_SANDBOX_CSP
    return FileResponse(
        image_file,
        media_type=media_type,
        headers=headers,
    )


@router.get("/documents/{doc_id}/pages/{n}", response_class=FileResponse)
def get_document_page(
    doc_id: str,
    n: int,
    db: Session = Depends(get_db),  # noqa: B008
) -> FileResponse:
    """The phase-148 (task 02) PDF page BYTES route — ONE rendered page
    PNG (``<doc-id>-p{N}.png`` in the importer's ``pdf_page_dir`` copy,
    task 01) of a page-QUALIFYING document.

    PUBLIC (the phase-124 A4 trust model verbatim — the image route's
    precedent): a shared chat can cite a page FIGURE (the viewer's
    strip, task 03), so the bytes serve ANONYMOUSLY. The document id
    is an unguessable server-minted ``uuid4`` — IT IS the credential,
    with no enumeration value — exactly like the image route and the
    saved-chat id (phase 55). The document's TEXT stays user-gated (the
    content endpoint): only the page bytes are public.

    404 ``document not found`` (the router's unknown-document shape) in
    EVERY non-servable case — an unparseable ``doc_id`` (the
    ``uuid.UUID`` try/except maps here, not to a 422 — the image
    route's mapping: a guessed id is an unknown document), an unknown
    doc, a doc whose ``page_count`` is NULL (a non-qualifying doc — the
    route knows nothing of its pages), ``n`` outside ``1..page_count``
    (out of range — a GAP, a page whose render failed, hits the
    missing-file arm below), or a file on disk that is missing
    (defensive — the row exists, the bytes don't; the viewer's onerror
    note covers it).

    ``n`` is an ``int`` path parameter (FastAPI parses it — a non-int
    422s at the router), and the path is built ONLY from the row's id
    + ``n`` (``pdf_page_dir/<doc.id>-p{n}.png``) — no request value
    reaches a filesystem path (no traversal surface, the image route's
    property).

    Servable pages stream the exact stored bytes as ``image/png``
    (``render_pdf_pages``'s ``tobytes("png")`` — a page is always a
    PNG) with ``Cache-Control: private, max-age=3600`` (the image
    route's header — the bytes are content-addressed by the document
    id + page number, bustable by re-index).
    """
    try:
        uid = uuid.UUID(doc_id)
    except ValueError:
        raise HTTPException(status_code=404, detail="document not found") from None
    doc = db.scalar(select(Document).where(Document.id == uid))
    if doc is None or doc.page_count is None:
        raise HTTPException(status_code=404, detail="document not found")
    if not 1 <= n <= doc.page_count:
        raise HTTPException(status_code=404, detail="document not found")
    page_file = Path(get_settings().pdf_page_dir).expanduser() / f"{doc.id}-p{n}.png"
    if not page_file.is_file():
        # Defensive: the row qualifies but the copy was lost (the owner
        # cleaned the page dir, the disk was wiped) — the viewer's
        # onerror note renders "Page N unavailable".
        raise HTTPException(status_code=404, detail="document not found")
    return FileResponse(
        page_file,
        media_type="image/png",
        headers={"Cache-Control": "private, max-age=3600"},
    )


@router.patch("/documents/summary", response_model=SummaryResult)
async def update_document_summary(
    payload: SummaryUpdate,
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> SummaryResult:
    """Update or clear a document's stored summary and re-embed it.

    Admin-only (phase 57, D4) — the document viewer itself stays
    PUBLIC (phase 16 owner decision); only this edit affordance is
    gated. The re-embed scope is the ``is_summary`` chunk only (D4):
    the summary is the only text that changed, so the document's
    content chunks keep their existing embeddings — the total chunk
    count is unchanged by an update.

    Fail-before-write (phase 57 locked decision): when the stripped
    text is non-empty it is embedded **before** any DB mutation — an
    embedding failure returns 503 with a sanitized ``detail`` naming
    the failure (the ``ModelUnavailableError`` handling of
    ``app/api/git_sources.py``) and leaves the row and chunk untouched.
    An empty/whitespace-only ``summary`` clears instead:
    ``documents.summary = NULL`` and the ``is_summary`` chunk (if any)
    is deleted.
    """
    doc = db.scalar(
        select(Document).where(
            Document.source == payload.source, Document.path == payload.path
        )
    )
    if doc is None:
        raise HTTPException(status_code=404, detail="document not found")

    summary_chunk = db.scalar(
        select(Chunk).where(Chunk.document_id == doc.id, Chunk.is_summary.is_(True))
    )
    text = payload.summary.strip()
    if text:
        # Embed first, mutate second — a failed LLM call must never
        # leave a half-updated row (phase 57 locked decision).
        llm = LLMClient()
        try:
            vector = (await llm.embed([text]))[0]
        except EmbeddingError as e:
            raise HTTPException(status_code=503, detail=_sanitize_error(str(e))) from None
        if summary_chunk is None:
            # Markdown doc, or a phase-30 fail-soft import that indexed
            # without a summary chunk — create the position −1 chunk.
            summary_chunk = Chunk(document_id=doc.id, position=-1, is_summary=True)
            db.add(summary_chunk)
        summary_chunk.content = text
        summary_chunk.embedding = vector
        doc.summary = text
    else:
        if summary_chunk is not None:
            db.delete(summary_chunk)
        doc.summary = None
    db.commit()
    chunks = db.scalar(
        select(func.count(Chunk.id))
        .select_from(Document)
        .outerjoin(Chunk, Chunk.document_id == Document.id)
        .where(Document.id == doc.id)
    ) or 0
    return SummaryResult(
        source=doc.source, path=doc.path, summary=doc.summary, chunks=chunks
    )


@router.patch("/documents/date", response_model=DateResult)
def update_document_date(
    payload: DateUpdate,
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> DateResult:
    """Set or clear the owner's correction of a document's creation date.

    Admin-only (phase 106, D7 — the phase-57 split): the document
    viewer itself stays user-gated (``/documents/content`` — admin OR
    token holder); only this edit affordance is admin-gated. DB-only
    (the ``/documents/content`` row-lookup rule): the ``(source, path)``
    pair is looked up as a row — an unknown pair, including traversal
    strings such as ``../../etc/passwd``, is simply not a row (→ 404
    ``document not found``), and there is no filesystem access. NO
    LLM/embedding call — a date is never embedded (no chunk, no
    retrieval role) — the deliberate contrast with the phase-57
    ``is_summary`` re-embed in :func:`update_document_summary`.

    * **Set** (``date`` present) — ``datetime.fromisoformat`` accepts a
      bare ``YYYY-MM-DD`` (midnight) and full ISO datetimes; a
      MALFORMED value 422s here (the model field is an unconstrained
      ``str | None`` on purpose, so the detail can name the field). The
      parse goes through
      :func:`app.rag.doc_dates.normalize_doc_date` (D3 — the single
      choke point: naive → UTC, aware → converted, a manually set
      FUTURE date also folds to today — consistency with the sourced
      path) and stores ``created_at`` + ``created_at_manual = True``
      (the owner's correction — the sync-time importer then SKIPS the
      refresh on this row, D1/D4).
    * **Clear** (``date`` null/absent — the "revert to sync"
      operation) — ``created_at_manual = False`` ONLY: the stored date
      stands until the next sync refreshes it (the API cannot
      re-read the source, D7).

    The response echoes the stored state — ``created_at`` (ISO-8601)
    + ``created_at_manual`` — the viewer re-renders its Created badge
    from it (no second fetch).
    """
    doc = db.scalar(
        select(Document).where(
            Document.source == payload.source, Document.path == payload.path
        )
    )
    if doc is None:
        raise HTTPException(status_code=404, detail="document not found")
    if payload.date:
        try:
            parsed = datetime.fromisoformat(payload.date)
        except ValueError:
            raise HTTPException(
                status_code=422,
                detail="date must be an ISO date or datetime (e.g. 2024-06-15)",
            ) from None
        doc.created_at = normalize_doc_date(parsed)
        doc.created_at_manual = True
    else:
        # The CLEAR (D7): drop the manual flag only — the stored date
        # stands until the next sync refreshes it.
        doc.created_at_manual = False
    db.commit()
    return DateResult(
        source=doc.source,
        path=doc.path,
        created_at=doc.created_at.isoformat(),
        created_at_manual=doc.created_at_manual,
    )


@router.patch("/folders/summary", response_model=FolderSummaryResult)
def update_folder_summary(
    payload: FolderSummaryUpdate,
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> FolderSummaryResult:
    """Update / create / clear a folder's stored description.

    Admin-only: the catalog is admin-only (phase 16) and this gate is
    the API-level defense in depth (the RAG view never renders for
    anonymous, the endpoint must not lean on that). ``folder_path =
    ""`` is the SOURCE ROOT (the phase-94 ``folder_summaries``
    convention — the top-level source summary).

    Checks, in order — DB-only (the ``/documents/content`` rule: no
    filesystem access at all, a traversal string such as ``../../etc``
    is simply not a prefix of any indexed path):

    * **Source** — registered (``list_source_names``) OR has indexed
      documents → else 404 ``{"detail": "source not found"}``.
    * **Folder** — ``""`` is valid for an allowed source; otherwise the
      phase-94 existence rule over the source's indexed paths (some
      ``Document.path`` starts with ``folder_path + "/"`` — a file
      merely sharing the folder's name is NOT a folder) → else 404
      ``{"detail": "folder not found"}``.

    Writes: a non-empty (after ``strip()``) ``summary`` upserts the row
    with ``summary = stripped text``, ``manually_edited = True``, and a
    fresh UTC ``updated_at`` — a manual description can be CREATED
    where no row exists (a < 2-document folder, or the generator's
    fail-soft miss), and from this save on the task-01 keep/keep-out
    rules apply (the generator skips the row and never prunes it).
    An empty/whitespace-only ``summary`` CLEARS instead: the row is
    ``db.delete``'d when present (the phase-57 analog — the row may be
    AI-written or manual, either way it is gone; the next KB-changing
    sync regenerates an AI row — the reset path). A clear with no row
    is a 200 no-op. The response echoes the stored text — ``summary``
    null after a clear.

    No LLM/embedding call on this path: a folder description is never
    embedded (no chunk, no retrieval role beyond the ``ls`` line) —
    the deliberate contrast with the phase-57 ``is_summary`` re-embed
    in :func:`update_document_summary`. The no-LLM contract is
    source-pinned in the test suite (the handler's source never names
    the LLM client).
    """
    source = payload.source
    folder = payload.folder_path
    paths = list(db.scalars(select(Document.path).where(Document.source == source)))
    if source not in list_source_names(db) and not paths:
        raise HTTPException(status_code=404, detail="source not found")
    if folder and not any(p.startswith(folder + "/") for p in paths):
        raise HTTPException(status_code=404, detail="folder not found")
    row = db.scalar(
        select(FolderSummary).where(
            FolderSummary.source == source, FolderSummary.folder_path == folder
        )
    )
    text = payload.summary.strip()
    if text:
        if row is None:
            row = FolderSummary(source=source, folder_path=folder)
            db.add(row)
        row.summary = text
        row.manually_edited = True
        row.updated_at = datetime.now(UTC)
    else:
        if row is not None:
            db.delete(row)
    db.commit()
    return FolderSummaryResult(source=source, folder_path=folder, summary=text or None)


#: One catalogue row the tree builder consumes:
#: ``(source, path, title, chunks, indexed_at, created_at)`` — the
#: ``GET /api/docs`` query's columns minus the document ``id`` (the
#: tree has no document ids), in the same ``(source, path)`` order;
#: ``indexed_at`` / ``created_at`` are the ISO-8601 strings the
#: endpoint converts (the builder stays pure over plain types —
#: unit-testable without a DB).
TreeDocRow = tuple[str, str, str, int, str, str]

#: One of a source's file rows, already source-scoped:
#: ``(path, title, chunks, indexed_at, created_at)``.
TreeFileRow = tuple[str, str, int, str, str]


def _folder_counts(
    rows: Sequence[TreeFileRow],
) -> tuple[set[str], dict[str, int], dict[str, str]]:
    """One source's folders (existence rule) + recursive counts + the
    recursive max-date map (all pure).

    The phase-94 rules, reused verbatim from
    :func:`app.rag.agent.group_folder_listing` (ONE concept end to end —
    the UI tree is the ``ls`` tree plus file metadata): a folder exists
    ⟺ some indexed path starts with ``folder + "/"`` (a slash-boundary
    prefix of an indexed path — a document's OWN path is never a
    folder; the folder's parent is :func:`folder_of`, the shared
    notion, never re-derived). The count of a folder is its recursive
    subtree — every path EQUAL to the folder (the file sharing its
    name counts, the ``path == folder`` arm) or starting with
    ``folder + "/"`` — exactly the set the sync-time folder summary
    describes.

    Returns ``(folders, counts, max_dates)`` — the folder-name set, the
    per-folder count map (every folder counts ≥ 1 by construction: its
    own descendants, or the file wearing its name, exist), and the
    per-folder RECURSIVE MAX document ``created_at`` map — the same
    subtree the count walks, over the same ISO-8601 strings (one
    ``isoformat()`` shape, so the lexicographic max is the
    chronological one — the phase-106 D9 rule).

    The max map is the phase-167 (task 02, D2) truncation-proof date
    source: a depth-limited node has NO built children, so
    :func:`_subtree_max` over its empty child list would read ``None``
    and silently lose the D9 column; the aggregate — computed from the
    rows in the same pass — keeps the date honest at every depth. The
    expanded path keeps deriving ``updated_at`` via
    :func:`_subtree_max` (one concept, two derivations — cross-pinned
    equal by a unit test).
    """
    folders: set[str] = set()
    for path, _title, _chunks, _indexed_at, _created_at in rows:
        folder = folder_of(path)
        while folder:
            folders.add(folder)
            folder = folder_of(folder)
    counts: dict[str, int] = {folder: 0 for folder in folders}
    max_dates: dict[str, str] = {}
    for path, _title, _chunks, _indexed_at, created_at in rows:
        if path in folders:
            counts[path] += 1
            if created_at > max_dates.get(path, ""):
                max_dates[path] = created_at
        folder = folder_of(path)
        while folder:
            counts[folder] += 1
            if created_at > max_dates.get(folder, ""):
                max_dates[folder] = created_at
            folder = folder_of(folder)
    return folders, counts, max_dates


#: One image-docs map value (phase 122, task 04):
#: ``(doc_id, summary)`` — the id the bytes route's URL is built from
#: and the summary the RAG view's thumbnail uses as ``alt`` (the vision
#: description; ``None`` = the fail-soft backfill corner).
ImageDocInfo = tuple[str, str]


def _level_children(
    source: str,
    folder: str,
    folders: set[str],
    counts: dict[str, int],
    max_dates: dict[str, str],
    rows: Sequence[TreeFileRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
) -> list[KbTreeFolder | KbTreeFile]:
    """One level's children (pure): subfolders in path order, then the
    direct files in input (catalog) order.

    *folder* is source-relative (``""`` = the source root). A folder
    *sub* appears here iff ``folder_of(sub) == folder`` (a DIRECT
    subfolder) and it exists (the :func:`_folder_counts` set — the
    phase-94 existence rule); a file appears iff
    ``folder_of(path) == folder`` (a DIRECT file). The subfolder
    order is the sorted (path) order and the file order is the input
    (catalog — ``GET /api/docs``) order, both matching
    :func:`app.rag.agent.group_folder_listing` level-for-level; the
    file list is NOT capped (the ``ls`` 50-line cap is a model-context
    budget — the UI is for humans). Recurses one level per call.

    *depth* (phase 167, task 02, D2/A2) is the REMAINING build budget
    for this level's subfolders: ``None`` = unlimited (the pre-phase
    recursion, byte-identical); a positive budget is spent here and
    ``depth - 1`` rides the recursion. When the budget is exhausted
    (``depth == 1`` reaching its subfolders) the subfolder nodes are
    TRUNCATED instead of recursed: ``children=[]``, ``truncated=True``,
    and every field the level row still renders — the recursive
    ``documents`` count, the recursive ``updated_at`` from *max_dates*
    (NOT ``_subtree_max`` over the empty child list — the D9 trap), the
    node's own ``summary`` (the row shows a SUBFOLDER's description),
    and ``summary_pending`` under the unchanged rule. The level's
    DIRECT files are always built — the budget limits DEPTH, never the
    row set of the levels it does send.

    Each folder node also carries the phase-98 D3 ``summary_pending``
    flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
    stored ``folder_summaries`` row for ``(source, sub)`` — the same
    rule the source node applies (see :func:`build_kb_tree`).

    And, since phase 106 (D9), each folder node carries ``updated_at``
    = the subtree's MAX document ``created_at``: the max over this
    folder's direct files' dates and its subfolder children's (already
    recursive) ``updated_at`` values, via :func:`_subtree_max`.

    Since phase 122 (task 04), a file node whose ``(source, path)`` is
    in *images* carries the thumbnail affordance (``is_image`` +
    ``image_url`` built from the mapped doc id + the mapped ``summary``
    — see :class:`app.schemas.KbTreeFile`); every other file node is
    the pre-phase shape (the omission rule keeps its wire shape
    byte-identical).
    """
    children: list[KbTreeFolder | KbTreeFile] = []
    expand = depth is None or depth > 1
    next_depth = None if depth is None else depth - 1
    for sub in sorted(g for g in folders if folder_of(g) == folder):
        if expand:
            sub_children = _level_children(
                source, sub, folders, counts, max_dates, rows, summaries, images, next_depth
            )
            children.append(
                KbTreeFolder(
                    path=sub,
                    documents=counts[sub],
                    updated_at=_subtree_max(sub_children),
                    summary=summaries.get((source, sub)),
                    summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
                    and (source, sub) not in summaries,
                    children=sub_children,
                )
            )
        else:
            # Phase 167 (D2): budget exhausted — the node rides with its
            # recursive aggregate fields and truncated=True; the client
            # drills via the node route (task 03).
            children.append(
                KbTreeFolder(
                    path=sub,
                    documents=counts[sub],
                    updated_at=max_dates[sub],
                    summary=summaries.get((source, sub)),
                    summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
                    and (source, sub) not in summaries,
                    children=[],
                    truncated=True,
                )
            )
    for path, title, chunks, indexed_at, created_at in rows:
        if folder_of(path) == folder:
            image_info = images.get((source, path)) if images else None
            if image_info is not None:
                # Phase 122 (task 04): the image-docs node — the RAG
                # view's Path cell renders the 48px thumbnail from the
                # bytes route's URL with ``alt = summary``.
                doc_id, doc_summary = image_info
                children.append(
                    KbTreeFile(
                        path=path,
                        title=title,
                        chunks=chunks,
                        created_at=created_at,
                        indexed_at=indexed_at,
                        is_image=True,
                        image_url=f"/api/documents/{doc_id}/image",
                        summary=doc_summary,
                    )
                )
            else:
                children.append(
                    KbTreeFile(
                        path=path,
                        title=title,
                        chunks=chunks,
                        created_at=created_at,
                        indexed_at=indexed_at,
                    )
                )
    return children


def _subtree_max(children: Sequence[KbTreeFolder | KbTreeFile]) -> str | None:
    """A node's ``updated_at`` (phase 106, D9): the subtree's MAX
    document ``created_at``, computed from the node's direct children —
    files contribute their ``created_at``, subfolder nodes contribute
    their (already recursive) ``updated_at``. UNCHANGED for the
    expanded path since phase 167 (the truncated path uses the
    :func:`_folder_counts` max map instead — one concept, two
    derivations, cross-pinned equal by a unit test).

    All values share the same UTC ``isoformat()`` shape (the endpoint
    converts every ``created_at`` before the builder runs), so the
    LEXICOGRAPHIC max is the chronological max — ISO-8601 strings of
    one offset order by instant. ``None`` when no child carries a date
    (a node with no documents at all — the 0-document source).
    """
    dates = [
        child.created_at if child.kind == "file" else child.updated_at
        for child in children
    ]
    dates = [d for d in dates if d is not None]
    return max(dates) if dates else None


def build_kb_tree(
    names: Sequence[str],
    doc_rows: Sequence[TreeDocRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
) -> list[KbTreeSource]:
    """The pure tree builder behind ``GET /api/docs/tree`` (phase 97,
    task 02) — module-level and DB-free so unit tests drive it
    directly (the house pattern; the endpoint composes the fetches).

    *names* — the registry source names in order (``app.rag.agent.
    list_source_names`` — deduped, registry order). *doc_rows* — the
    catalogue ``(source, path, title, chunks, indexed_at, created_at)``
    tuples (both stamps ISO-8601) in the ``GET /api/docs`` query order
    (``source, path``). *summaries* —
    ``{(source, folder_path): summary}`` over the stored
    ``folder_summaries`` rows (``folder_path = ""`` = the source root;
    rows for sources the tree does not list are simply never
    referenced). *images* (phase 122, task 04) —
    ``{(source, path): (doc_id, summary)}`` over the stored image docs
    (``is_image`` rows with a servable ``image_path`` — the endpoint
    composes the bounded select); the default ``None``/empty map keeps
    EVERY file node the pre-phase shape (byte-identical wire — a
    pre-phase KB has no image rows, so the endpoint's own map is empty
    for it).

    Shape, per the phase-97 ``00_phase.md`` "The tree endpoint":

    * **Sources** — the registry names first (each ALWAYS present — a
      registered 0-document source lists with ``documents: 0`` and no
      children), then the distinct indexed sources not in *names*
      (alphabetical — the superset rule: the catalog has never hidden
      an indexed document, while the agent's ``ls`` keeps listing
      registry sources only — unchanged). Every doc source is listed by
      construction; the registry order still leads.
    * **Folder nodes** — ``path`` source-relative (never ``""`` — the
      source node IS the root); direct subfolders only, in path
      (sorted) order, a folder existing only under the phase-94
      existence rule; ``documents`` = the recursive subtree count;
      ``updated_at`` (phase 106, D9) = the subtree's MAX document
      ``created_at`` — DERIVED as the builder recurses (the max of the
      direct files' dates and the children's ``updated_at`` values via
      :func:`_subtree_max`; the ISO-8601 strings share one
      ``isoformat()`` shape, so the lexicographic max is the
      chronological one), never stored — ``null`` when the node has no
      documents at all; ``summary`` = the stored row (AI OR manual —
      any row) or null; ``children`` = the folder's own subfolders +
      direct files, same shape.
    * **File nodes** — direct files only, in input (catalog) order;
      ``path`` source-relative; ``title`` / ``chunks`` / ``created_at``
      (phase 106) / ``indexed_at`` verbatim from the catalogue row.
      File nodes carry NO pending flag (the file table has no
      description column) and no ``updated_at`` (a file's date IS its
      ``created_at``).
    * **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
      node (phase 98, decision D3 — ONE concept): true iff the node's
      recursive ``documents`` count ≥
      :data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (1) AND it
      has NO stored ``folder_summaries`` row (AI or manual — any row;
      the builder sees stored rows only). That is EXACTLY
      :func:`app.rag.folder_summaries.missing_folder_summaries`'s
      candidate set (phase 96's gap-fill regenerates precisely those
      keys on the next sync — the marker is honest: "waiting to
      generate", and the integration cross-check pins the tree's
      pending set to that function so the marker can never drift from
      the gap-fill). A 0-document folder cannot exist (a folder is a
      catalogue prefix only) — every existing folder with no row is
      pending, single-file folders included — and a registered
      0-document source never is.

    *depth* (phase 167, task 02, D2/A2) bounds how deep the tree is
    BUILT: ``None`` (the default) = unlimited — every pre-phase call
    and expectation is byte-identical; a positive budget stops the
    recursion (see :func:`_level_children`) so the response carries the
    sources plus *depth* levels, every node whose children were not
    built flagged ``truncated=True`` with its recursive count, its
    recursive ``updated_at`` (the max-date map — D9 survives
    truncation), its ``summary`` and ``summary_pending``. A SOURCE node
    is never truncated at ``depth ≥ 1`` — the view's top level is
    source → level 1, so the source's direct children are always built.

    ONE concept end to end: the builder reuses
    :func:`app.rag.folder_summaries.folder_of` and the phase-94
    existence / count rules, so — for a single-source dataset — its
    level equals :func:`app.rag.agent.group_folder_listing`'s output
    (same subfolder ``(path, count, summary)`` triples in order, same
    file ``(source, path, title, date)`` 4-tuples in order — phase
    106, D5: the agent's file lines carry the appended ``date`` field
    and the tree's file nodes carry ``created_at``; the "UI shows
    what the agent sees" cross-check, unit-pinned at the root and a
    nested level, compares the extended shapes).

    D9 (phase 106): ``updated_at`` on every SOURCE and FOLDER node is
    the subtree's MAX document ``created_at`` — derived, never stored;
    ``None`` for a node with no documents at all (the registered
    0-document source).
    """
    by_source: dict[str, list[TreeFileRow]] = {}
    for source, path, title, chunks, indexed_at, created_at in doc_rows:
        by_source.setdefault(source, []).append((path, title, chunks, indexed_at, created_at))
    tree: list[KbTreeSource] = []
    listed: set[str] = set()
    for name in names:
        if name in listed:  # defensive: list_source_names dedupes
            continue
        listed.add(name)
        tree.append(
            _source_node(name, by_source.get(name, ()), summaries, images, depth)
        )
    for source in sorted(by_source):
        if source not in listed:
            tree.append(
                _source_node(source, by_source[source], summaries, images, depth)
            )
    return tree


def _source_node(
    source: str,
    rows: Sequence[TreeFileRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
) -> KbTreeSource:
    """One source node (pure): whole-source count + the source-root
    summary + the root level's children (direct subfolders + direct
    files).

    ``documents`` is ``len(rows)`` — the source's WHOLE recursive
    count (every one of its documents, the set its stored
    ``(source, "")`` summary describes). A source with no rows lists
    ``documents: 0`` and no children (the registered 0-document source
    — the phase-70/72 invariant, extended by the superset rule) and
    is never ``summary_pending`` (0 < the minimum).

    ``summary_pending`` (phase 98, D3): the whole-source count ≥
    :data:`MIN_DOCS_PER_FOLDER` AND no stored ``(source, "")`` row —
    the source-root arm of the rule :func:`build_kb_tree` documents.

    ``updated_at`` (phase 106, D9): the source's subtree max —
    :func:`_subtree_max` over the root level's children; ``None`` for a
    0-document source (no children, no dates). Correct at every depth:
    a TRUNCATED child still carries its recursive ``updated_at`` (the
    max-date map), so the source's max never silently drops the levels
    below the budget (phase 167, D2). The SOURCE node itself is never
    marked ``truncated`` at ``depth ≥ 1`` — its direct children are
    always built (the view's top level).
    """
    folders, counts, max_dates = _folder_counts(rows)
    children = _level_children(
        source, "", folders, counts, max_dates, rows, summaries, images, depth
    )
    return KbTreeSource(
        name=source,
        documents=len(rows),
        updated_at=_subtree_max(children),
        summary=summaries.get((source, "")),
        summary_pending=len(rows) >= MIN_DOCS_PER_FOLDER and (source, "") not in summaries,
        children=children,
    )


def _kb_totals(doc_rows: Sequence[TreeDocRow]) -> KbTreeTotals:
    """The catalogue totals beside the tree (phase 167, task 02, D2) —
    computed from the SAME rows the builder consumes (no extra query):
    ``documents = len(rows)``, ``chunks`` = the sum of the per-document
    chunk counts, ``last_indexed`` = the MAX ``indexed_at`` stamp
    (lexicographic max = chronological max over one ISO-8601 shape),
    ``None`` for an empty KB. The RAG view's stat cards read these
    because a depth-limited tree cannot be summed client-side (the
    phase-97 ``treeStats()`` walk only sees the sent levels).
    """
    return KbTreeTotals(
        documents=len(doc_rows),
        chunks=sum(chunks for _s, _p, _t, chunks, _i, _c in doc_rows),
        last_indexed=max((indexed for _s, _p, _t, _ch, indexed, _c in doc_rows), default=None),
    )


def _kb_catalogue(
    db: Session,
) -> tuple[
    list[str],
    list[TreeDocRow],
    dict[tuple[str, str], str],
    dict[tuple[str, str], ImageDocInfo],
]:
    """The catalogue composition the tree routes share (phase 167,
    task 03) — ONE outerjoin/grouped catalogue query plus the two
    bounded side selects, no N+1 anywhere (the drill pays exactly what
    the first paint pays). Returns ``(names, doc_rows, summaries,
    images)``:

    * *names* — the registry source names (``list_source_names`` — the
      superset rule's registry half, imported from ``app.rag.agent``
      exactly as ``app/api/chat.py`` does);
    * *doc_rows* — the SAME outerjoin/grouped catalogue query
      ``GET /api/docs`` runs (the document ``id`` excluded — the tree
      has no document ids), both stamps converted to ISO-8601 HERE so
      the pure builder stays DB-free;
    * *summaries* — ALL stored ``folder_summaries`` rows (a bounded
      select — one row per existing folder at the ≥ 1-doc rule; rows
      for sources the tree does not list are never referenced by the
      builder);
    * *images* — the phase-122 (task 04) image-docs map: a second
      bounded select over the ``is_image`` rows with a servable
      ``image_path`` only (a handful of rows at KB scale, never the
      whole catalog; the catalogue query above stays byte-identical).
      EMPTY for every pre-phase KB (no image rows), so the response
      stays byte-identical to pre-phase — the fields are row-driven,
      not toggle-driven (a surviving image doc keeps its thumbnail
      through a toggle-off sync, the prune guard's UX side).
    """
    names = list_source_names(db)
    rows = db.execute(
        select(
            Document.source,
            Document.path,
            Document.title,
            func.count(Chunk.id).label("chunks"),
            Document.created_at,
            Document.indexed_at,
        )
        .outerjoin(Chunk, Chunk.document_id == Document.id)
        .group_by(
            Document.id,
            Document.source,
            Document.path,
            Document.title,
            Document.created_at,
            Document.indexed_at,
        )
        .order_by(Document.source, Document.path)
    ).all()
    doc_rows: list[TreeDocRow] = [
        (source, path, title, chunks, indexed_at.isoformat(), created_at.isoformat())
        for source, path, title, chunks, created_at, indexed_at in rows
    ]
    summaries: dict[tuple[str, str], str] = {
        (source, folder_path): summary
        for source, folder_path, summary in db.execute(
            select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
        ).all()
    }
    images: dict[tuple[str, str], ImageDocInfo] = {
        (source, path): (str(doc_id), summary)
        for source, path, doc_id, summary in db.execute(
            select(
                Document.source,
                Document.path,
                Document.id,
                Document.summary,
            ).where(Document.is_image.is_(True), Document.image_path.is_not(None))
        ).all()
    }
    return names, doc_rows, summaries, images


def _revalidated_json(request: Request, model: BaseModel) -> Response:
    """Serialize once, hash the EXACT served bytes, answer 304 when the
    client already holds them (phase 167, task 03, D3/A4 — the
    :mod:`app.core.http_cache` pattern in ONE place for the three
    catalogue reads).

    The client is ``fetch``: a matched conditional request is served
    from its cache transparently (fetch never surfaces the 304), and
    ``must-revalidate`` guarantees a sync or an admin edit is seen on
    the next load — the tag changes because the body changes (the A4
    proof: a ``sources_meta.version``-derived tag would keep serving
    stale trees after the folder-summary / document-date PATCHes, which
    never bump it). ``model_dump_json`` runs exactly ONCE — the same
    string feeds the tag and the body, and the returned ``Response``
    short-circuits FastAPI's own serialization.
    """
    body = model.model_dump_json()
    etag = weak_etag(body)
    if etag_matches(request.headers.get("if-none-match"), etag):
        return Response(status_code=304, headers=revalidate_headers(etag))
    return Response(
        content=body, media_type="application/json", headers=revalidate_headers(etag)
    )


@router.get("/docs/tree", response_model=KbTree)
def list_kb_tree(
    request: Request,
    depth: int = Query(1, ge=1, le=8),
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> Response:
    """The KB tree, DEPTH-LIMITED by default (phase 97, task 02;
    phase 167, task 02, D2/A2 — the TODO L5 2 MB payload fix).

    ``depth`` bounds the levels built under each source:
    ``1`` (default) = sources + ONE level — the RAG view's first paint;
    ``8`` = the practical KB ceiling (the whole tree, the pre-phase
    shape plus the additive ``totals`` key). Every node whose children
    were NOT built carries ``truncated: true`` (with its recursive
    count, recursive ``updated_at``, ``summary`` and
    ``summary_pending`` intact) so the view can drill level by level;
    ``totals`` (documents / chunks / ``last_indexed``) is computed here
    from the full catalogue rows, so the stat cards stay honest at any
    depth.

    Admin-only, like ``GET /api/docs`` — anonymous callers get 403
    ``admin only`` (the RAG view's anonymous gate never fetches the
    tree). The phase-97 "one fetch, drill client-side" contract is
    deliberately superseded (LOCKED A3): one initial load plus one
    fetch per drilled level (task 03's node route).

    Composition: :func:`_kb_catalogue` (the registry source names, the
    SAME outerjoin/grouped catalogue query ``GET /api/docs`` runs, ALL
    stored ``folder_summaries`` rows, and the phase-122 image-docs map)
    through the pure :func:`build_kb_tree` + :func:`_kb_totals`.
    ``GET /api/docs`` itself is untouched.

    Phase 167 (task 03, D3/A4): the response carries the content-exact
    weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair via
    :func:`_revalidated_json` — a repeat load costs a 304 with no body
    (the client is ``fetch``, which serves a matched conditional request
    from its cache transparently). The decorator keeps
    ``response_model=KbTree`` for the OpenAPI schema only — a returned
    :class:`~fastapi.responses.Response` short-circuits FastAPI's
    serialization, so the model is dumped exactly ONCE here.
    """
    names, doc_rows, summaries, images = _kb_catalogue(db)
    model = KbTree(
        sources=build_kb_tree(names, doc_rows, summaries, images, depth),
        totals=_kb_totals(doc_rows),
    )
    return _revalidated_json(request, model)


@router.get("/docs/tree/node", response_model=None)
def get_kb_tree_node(
    request: Request,
    source: str = Query(...),
    path: str = Query(""),
    depth: int = Query(1, ge=1, le=8),
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
) -> KbTreeSource | KbTreeFolder | Response:
    """ONE tree node's direct children — the RAG view's per-level drill
    fetch (phase 167, task 03, D3/A3: the deliberate cost of the small
    first payload is one request per drilled level).

    ``source`` is the registry name OR an indexed source (the superset
    rule); ``path`` is SOURCE-RELATIVE, ``""`` (the default) = the
    source root. The node answers in the shape the full tree produces
    for the same node — a :class:`~app.schemas.KbTreeSource` for
    ``path == ""``, else a :class:`~app.schemas.KbTreeFolder` carrying
    the recursive ``documents`` count, the recursive ``updated_at``
    (the phase-106 D9 max — the truncation-proof max-date map), its
    ``summary`` and ``summary_pending`` (the unchanged phase-98 D3
    rule). The node's DIRECT children are always built — the *depth*
    budget only decides whether those children's subfolders recurse
    (the same rule that never truncates a SOURCE node at ``depth ≥ 1``,
    see :func:`_level_children`) — so the node itself is never
    ``truncated``: what arrived is what the level renders, and deeper
    levels ride the children's own flags.

    Admin-gated — the same :func:`~app.core.auth.require_admin` gate as
    ``/api/docs/tree`` (anonymous → 403 ``admin only``). 404
    ``{"detail": "unknown node"}`` when the source is NEITHER
    registered NOR indexed, or when a non-empty ``path`` fails the
    phase-94 existence rule (a folder exists ⟺ some indexed path
    starts with ``path + "/"`` — a document's OWN path is not a folder,
    unless a deeper path wears it).

    Row-only, like every route in this file: NOTHING here builds a
    filesystem path from ``path`` — the node is derived from catalogue
    ROWS alone, so a ``../``-style value is simply not a folder and 404s
    with no filesystem surface (the phase-10 traversal argument, reused
    by :func:`build_kb_tree`).

    Composition: the SAME catalogue query ``list_kb_tree`` runs (via
    :func:`_kb_catalogue` — ONE query per drill, no N+1; the tree's
    counts and recursive dates are only honest over the whole
    catalogue), then the pure helpers scoped to the one source. Carries
    the same ``ETag`` + ``private, must-revalidate`` pair as the other
    two catalogue reads (:func:`_revalidated_json`).
    ``response_model=None`` on the decorator: the return is a UNION
    annotated for the reader while the body is returned as a raw
    ``Response`` — the serializer must not run twice (once for the
    tag, once inside FastAPI).
    """
    names, doc_rows, summaries, images = _kb_catalogue(db)
    rows: list[TreeFileRow] = [
        (row_path, title, chunks, indexed_at, created_at)
        for row_source, row_path, title, chunks, indexed_at, created_at in doc_rows
        if row_source == source
    ]
    if source not in names and not rows:
        raise HTTPException(status_code=404, detail="unknown node")
    folders, counts, max_dates = _folder_counts(rows)
    if path and path not in folders:
        raise HTTPException(status_code=404, detail="unknown node")
    node: KbTreeSource | KbTreeFolder
    if path:
        children = _level_children(
            source, path, folders, counts, max_dates, rows, summaries, images, depth
        )
        node = KbTreeFolder(
            path=path,
            documents=counts[path],
            updated_at=max_dates[path],
            summary=summaries.get((source, path)),
            summary_pending=counts[path] >= MIN_DOCS_PER_FOLDER
            and (source, path) not in summaries,
            children=children,
        )
    else:
        # The source root — the very node the full tree builds.
        node = _source_node(source, rows, summaries, images, depth)
    return _revalidated_json(request, node)