phase: 167_api_payload_diet
All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
+362
-66
@@ -18,10 +18,34 @@ date folds to today, D3) + flag it manual, or clear the manual flag
|
||||
write: NO LLM/embedding call — a date is never embedded (the
|
||||
deliberate contrast with the phase-57 ``is_summary`` re-embed).
|
||||
|
||||
GET /api/docs/tree — the admin's full recursive KB tree in one fetch
|
||||
(phase 97, task 02): the same drill-down tree the agent's ``ls``
|
||||
walks, with the file metadata the RAG view's rows and stat cards need
|
||||
(the view drills client-side; ``GET /api/docs`` is untouched).
|
||||
GET /api/docs/tree — the admin's KB tree (phase 97, task 02; depth-
|
||||
limited since phase 167, task 02): the same drill-down tree the agent's
|
||||
``ls`` walks, with the file metadata the RAG view's rows and stat cards
|
||||
need. The default ``depth=1`` ships the sources plus ONE level with
|
||||
server-computed ``totals`` — a node whose children were not built
|
||||
carries ``truncated: true`` (the TODO L5 2 MB payload fix; ``GET
|
||||
/api/docs`` and the agent's ``ls`` path are untouched).
|
||||
|
||||
GET /api/docs/tree/node — ONE tree node's direct children (phase 167,
|
||||
task 03, D3): the RAG view's per-level drill fetch — the same catalogue
|
||||
composition as ``/api/docs/tree`` (ONE catalogue query per drill, no
|
||||
N+1) resolved to a single ``KbTreeSource`` (``path=""``, the source
|
||||
root) or ``KbTreeFolder``. Admin-gated like the tree; 404
|
||||
``{"detail": "unknown node"}`` for a source that is neither registered
|
||||
nor indexed, or a folder failing the phase-94 existence rule. Row-only
|
||||
like every other route here: no request value ever reaches a
|
||||
filesystem path — ``../``-style values simply are not folders (the
|
||||
phase-10 traversal argument, reused).
|
||||
|
||||
Revalidation (phase 167, task 03, LOCKED A4 — a scoped extension of
|
||||
A19): the three catalogue READ routes (``/api/docs``,
|
||||
``/api/docs/tree``, ``/api/docs/tree/node``) serialize once and carry a
|
||||
content-exact weak ``ETag`` + ``Cache-Control: private,
|
||||
must-revalidate`` (the :mod:`app.core.http_cache` helpers), so a repeat
|
||||
load costs a 304 with no body. The tag is a hash of the served bytes —
|
||||
NOT ``sources_meta.version`` (bumping that would mark saved chats
|
||||
stale, and the admin PATCHes do not bump it). ``sources_meta``, the
|
||||
SSE stream and the mutating surface are untouched.
|
||||
|
||||
GET /api/documents/{doc_id}/image — the phase-122 (task 04) image
|
||||
BYTES route: the persistent copy behind an ``is_image`` document's
|
||||
@@ -70,14 +94,16 @@ from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import FileResponse, Response
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.api.sync import _sanitize_error
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_admin, require_user
|
||||
from app.core.http_cache import etag_matches, revalidate_headers, weak_etag
|
||||
from app.db import get_db
|
||||
from app.models import Chunk, Document, FolderSummary
|
||||
from app.rag.agent import list_source_names
|
||||
@@ -99,6 +125,7 @@ from app.schemas import (
|
||||
KbTreeFile,
|
||||
KbTreeFolder,
|
||||
KbTreeSource,
|
||||
KbTreeTotals,
|
||||
SummaryResult,
|
||||
SummaryUpdate,
|
||||
)
|
||||
@@ -128,9 +155,10 @@ def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
|
||||
|
||||
@router.get("/docs", response_model=DocList)
|
||||
def list_indexed_documents(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
_admin: None = Depends(require_admin), # noqa: B008
|
||||
) -> DocList:
|
||||
) -> Response:
|
||||
"""All indexed documents with per-document chunk counts.
|
||||
|
||||
Admin-only (phase 16 — the catalog is what the admin sign-in gates;
|
||||
@@ -139,6 +167,16 @@ def list_indexed_documents(
|
||||
``admin only`` and the Sources page renders its sign-in gate instead.
|
||||
An empty list means the knowledge base has not been imported yet —
|
||||
the Sources page renders its designed empty state in that case.
|
||||
|
||||
Phase 167 (task 03, D3/A4): the response carries the content-exact
|
||||
weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair
|
||||
(see :mod:`app.core.http_cache`) — the client is ``fetch``, which
|
||||
serves a matched conditional request from its cache transparently,
|
||||
and ``must-revalidate`` guarantees a sync is seen on the next load
|
||||
(the tag changes because the body changes). The decorator keeps
|
||||
``response_model=DocList`` for the OpenAPI schema only — a returned
|
||||
:class:`~fastapi.responses.Response` short-circuits FastAPI's
|
||||
serialization, so the model is dumped exactly ONCE here.
|
||||
"""
|
||||
rows = db.execute(
|
||||
select(
|
||||
@@ -161,7 +199,7 @@ def list_indexed_documents(
|
||||
)
|
||||
.order_by(Document.source, Document.path)
|
||||
).all()
|
||||
return DocList(
|
||||
model = DocList(
|
||||
documents=[
|
||||
DocSummary(
|
||||
id=str(row.id),
|
||||
@@ -175,6 +213,7 @@ def list_indexed_documents(
|
||||
for row in rows
|
||||
]
|
||||
)
|
||||
return _revalidated_json(request, model)
|
||||
|
||||
|
||||
@router.get("/documents/content", response_model=DocContent)
|
||||
@@ -625,8 +664,9 @@ TreeFileRow = tuple[str, str, int, str, str]
|
||||
|
||||
def _folder_counts(
|
||||
rows: Sequence[TreeFileRow],
|
||||
) -> tuple[set[str], dict[str, int]]:
|
||||
"""One source's folders (existence rule) + recursive counts (pure).
|
||||
) -> tuple[set[str], dict[str, int], dict[str, str]]:
|
||||
"""One source's folders (existence rule) + recursive counts + the
|
||||
recursive max-date map (all pure).
|
||||
|
||||
The phase-94 rules, reused verbatim from
|
||||
:func:`app.rag.agent.group_folder_listing` (ONE concept end to end —
|
||||
@@ -640,9 +680,22 @@ def _folder_counts(
|
||||
``folder + "/"`` — exactly the set the sync-time folder summary
|
||||
describes.
|
||||
|
||||
Returns ``(folders, counts)`` — the folder-name set and the
|
||||
Returns ``(folders, counts, max_dates)`` — the folder-name set, the
|
||||
per-folder count map (every folder counts ≥ 1 by construction: its
|
||||
own descendants, or the file wearing its name, exist).
|
||||
own descendants, or the file wearing its name, exist), and the
|
||||
per-folder RECURSIVE MAX document ``created_at`` map — the same
|
||||
subtree the count walks, over the same ISO-8601 strings (one
|
||||
``isoformat()`` shape, so the lexicographic max is the
|
||||
chronological one — the phase-106 D9 rule).
|
||||
|
||||
The max map is the phase-167 (task 02, D2) truncation-proof date
|
||||
source: a depth-limited node has NO built children, so
|
||||
:func:`_subtree_max` over its empty child list would read ``None``
|
||||
and silently lose the D9 column; the aggregate — computed from the
|
||||
rows in the same pass — keeps the date honest at every depth. The
|
||||
expanded path keeps deriving ``updated_at`` via
|
||||
:func:`_subtree_max` (one concept, two derivations — cross-pinned
|
||||
equal by a unit test).
|
||||
"""
|
||||
folders: set[str] = set()
|
||||
for path, _title, _chunks, _indexed_at, _created_at in rows:
|
||||
@@ -651,14 +704,19 @@ def _folder_counts(
|
||||
folders.add(folder)
|
||||
folder = folder_of(folder)
|
||||
counts: dict[str, int] = {folder: 0 for folder in folders}
|
||||
for path, _title, _chunks, _indexed_at, _created_at in rows:
|
||||
max_dates: dict[str, str] = {}
|
||||
for path, _title, _chunks, _indexed_at, created_at in rows:
|
||||
if path in folders:
|
||||
counts[path] += 1
|
||||
if created_at > max_dates.get(path, ""):
|
||||
max_dates[path] = created_at
|
||||
folder = folder_of(path)
|
||||
while folder:
|
||||
counts[folder] += 1
|
||||
if created_at > max_dates.get(folder, ""):
|
||||
max_dates[folder] = created_at
|
||||
folder = folder_of(folder)
|
||||
return folders, counts
|
||||
return folders, counts, max_dates
|
||||
|
||||
|
||||
#: One image-docs map value (phase 122, task 04):
|
||||
@@ -673,9 +731,11 @@ def _level_children(
|
||||
folder: str,
|
||||
folders: set[str],
|
||||
counts: dict[str, int],
|
||||
max_dates: dict[str, str],
|
||||
rows: Sequence[TreeFileRow],
|
||||
summaries: Mapping[tuple[str, str], str],
|
||||
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
|
||||
depth: int | None = None,
|
||||
) -> list[KbTreeFolder | KbTreeFile]:
|
||||
"""One level's children (pure): subfolders in path order, then the
|
||||
direct files in input (catalog) order.
|
||||
@@ -691,6 +751,20 @@ def _level_children(
|
||||
file list is NOT capped (the ``ls`` 50-line cap is a model-context
|
||||
budget — the UI is for humans). Recurses one level per call.
|
||||
|
||||
*depth* (phase 167, task 02, D2/A2) is the REMAINING build budget
|
||||
for this level's subfolders: ``None`` = unlimited (the pre-phase
|
||||
recursion, byte-identical); a positive budget is spent here and
|
||||
``depth - 1`` rides the recursion. When the budget is exhausted
|
||||
(``depth == 1`` reaching its subfolders) the subfolder nodes are
|
||||
TRUNCATED instead of recursed: ``children=[]``, ``truncated=True``,
|
||||
and every field the level row still renders — the recursive
|
||||
``documents`` count, the recursive ``updated_at`` from *max_dates*
|
||||
(NOT ``_subtree_max`` over the empty child list — the D9 trap), the
|
||||
node's own ``summary`` (the row shows a SUBFOLDER's description),
|
||||
and ``summary_pending`` under the unchanged rule. The level's
|
||||
DIRECT files are always built — the budget limits DEPTH, never the
|
||||
row set of the levels it does send.
|
||||
|
||||
Each folder node also carries the phase-98 D3 ``summary_pending``
|
||||
flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
|
||||
stored ``folder_summaries`` row for ``(source, sub)`` — the same
|
||||
@@ -709,21 +783,40 @@ def _level_children(
|
||||
byte-identical).
|
||||
"""
|
||||
children: list[KbTreeFolder | KbTreeFile] = []
|
||||
expand = depth is None or depth > 1
|
||||
next_depth = None if depth is None else depth - 1
|
||||
for sub in sorted(g for g in folders if folder_of(g) == folder):
|
||||
sub_children = _level_children(
|
||||
source, sub, folders, counts, rows, summaries, images
|
||||
)
|
||||
children.append(
|
||||
KbTreeFolder(
|
||||
path=sub,
|
||||
documents=counts[sub],
|
||||
updated_at=_subtree_max(sub_children),
|
||||
summary=summaries.get((source, sub)),
|
||||
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
|
||||
and (source, sub) not in summaries,
|
||||
children=sub_children,
|
||||
if expand:
|
||||
sub_children = _level_children(
|
||||
source, sub, folders, counts, max_dates, rows, summaries, images, next_depth
|
||||
)
|
||||
children.append(
|
||||
KbTreeFolder(
|
||||
path=sub,
|
||||
documents=counts[sub],
|
||||
updated_at=_subtree_max(sub_children),
|
||||
summary=summaries.get((source, sub)),
|
||||
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
|
||||
and (source, sub) not in summaries,
|
||||
children=sub_children,
|
||||
)
|
||||
)
|
||||
else:
|
||||
# Phase 167 (D2): budget exhausted — the node rides with its
|
||||
# recursive aggregate fields and truncated=True; the client
|
||||
# drills via the node route (task 03).
|
||||
children.append(
|
||||
KbTreeFolder(
|
||||
path=sub,
|
||||
documents=counts[sub],
|
||||
updated_at=max_dates[sub],
|
||||
summary=summaries.get((source, sub)),
|
||||
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
|
||||
and (source, sub) not in summaries,
|
||||
children=[],
|
||||
truncated=True,
|
||||
)
|
||||
)
|
||||
)
|
||||
for path, title, chunks, indexed_at, created_at in rows:
|
||||
if folder_of(path) == folder:
|
||||
image_info = images.get((source, path)) if images else None
|
||||
@@ -761,7 +854,10 @@ def _subtree_max(children: Sequence[KbTreeFolder | KbTreeFile]) -> str | None:
|
||||
"""A node's ``updated_at`` (phase 106, D9): the subtree's MAX
|
||||
document ``created_at``, computed from the node's direct children —
|
||||
files contribute their ``created_at``, subfolder nodes contribute
|
||||
their (already recursive) ``updated_at``.
|
||||
their (already recursive) ``updated_at``. UNCHANGED for the
|
||||
expanded path since phase 167 (the truncated path uses the
|
||||
:func:`_folder_counts` max map instead — one concept, two
|
||||
derivations, cross-pinned equal by a unit test).
|
||||
|
||||
All values share the same UTC ``isoformat()`` shape (the endpoint
|
||||
converts every ``created_at`` before the builder runs), so the
|
||||
@@ -782,6 +878,7 @@ def build_kb_tree(
|
||||
doc_rows: Sequence[TreeDocRow],
|
||||
summaries: Mapping[tuple[str, str], str],
|
||||
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
|
||||
depth: int | None = None,
|
||||
) -> list[KbTreeSource]:
|
||||
"""The pure tree builder behind ``GET /api/docs/tree`` (phase 97,
|
||||
task 02) — module-level and DB-free so unit tests drive it
|
||||
@@ -847,6 +944,17 @@ def build_kb_tree(
|
||||
pending, single-file folders included — and a registered
|
||||
0-document source never is.
|
||||
|
||||
*depth* (phase 167, task 02, D2/A2) bounds how deep the tree is
|
||||
BUILT: ``None`` (the default) = unlimited — every pre-phase call
|
||||
and expectation is byte-identical; a positive budget stops the
|
||||
recursion (see :func:`_level_children`) so the response carries the
|
||||
sources plus *depth* levels, every node whose children were not
|
||||
built flagged ``truncated=True`` with its recursive count, its
|
||||
recursive ``updated_at`` (the max-date map — D9 survives
|
||||
truncation), its ``summary`` and ``summary_pending``. A SOURCE node
|
||||
is never truncated at ``depth ≥ 1`` — the view's top level is
|
||||
source → level 1, so the source's direct children are always built.
|
||||
|
||||
ONE concept end to end: the builder reuses
|
||||
:func:`app.rag.folder_summaries.folder_of` and the phase-94
|
||||
existence / count rules, so — for a single-source dataset — its
|
||||
@@ -872,10 +980,14 @@ def build_kb_tree(
|
||||
if name in listed: # defensive: list_source_names dedupes
|
||||
continue
|
||||
listed.add(name)
|
||||
tree.append(_source_node(name, by_source.get(name, ()), summaries, images))
|
||||
tree.append(
|
||||
_source_node(name, by_source.get(name, ()), summaries, images, depth)
|
||||
)
|
||||
for source in sorted(by_source):
|
||||
if source not in listed:
|
||||
tree.append(_source_node(source, by_source[source], summaries, images))
|
||||
tree.append(
|
||||
_source_node(source, by_source[source], summaries, images, depth)
|
||||
)
|
||||
return tree
|
||||
|
||||
|
||||
@@ -884,6 +996,7 @@ def _source_node(
|
||||
rows: Sequence[TreeFileRow],
|
||||
summaries: Mapping[tuple[str, str], str],
|
||||
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
|
||||
depth: int | None = None,
|
||||
) -> KbTreeSource:
|
||||
"""One source node (pure): whole-source count + the source-root
|
||||
summary + the root level's children (direct subfolders + direct
|
||||
@@ -902,10 +1015,17 @@ def _source_node(
|
||||
|
||||
``updated_at`` (phase 106, D9): the source's subtree max —
|
||||
:func:`_subtree_max` over the root level's children; ``None`` for a
|
||||
0-document source (no children, no dates).
|
||||
0-document source (no children, no dates). Correct at every depth:
|
||||
a TRUNCATED child still carries its recursive ``updated_at`` (the
|
||||
max-date map), so the source's max never silently drops the levels
|
||||
below the budget (phase 167, D2). The SOURCE node itself is never
|
||||
marked ``truncated`` at ``depth ≥ 1`` — its direct children are
|
||||
always built (the view's top level).
|
||||
"""
|
||||
folders, counts = _folder_counts(rows)
|
||||
children = _level_children(source, "", folders, counts, rows, summaries, images)
|
||||
folders, counts, max_dates = _folder_counts(rows)
|
||||
children = _level_children(
|
||||
source, "", folders, counts, max_dates, rows, summaries, images, depth
|
||||
)
|
||||
return KbTreeSource(
|
||||
name=source,
|
||||
documents=len(rows),
|
||||
@@ -916,32 +1036,56 @@ def _source_node(
|
||||
)
|
||||
|
||||
|
||||
@router.get("/docs/tree", response_model=KbTree)
|
||||
def list_kb_tree(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
_admin: None = Depends(require_admin), # noqa: B008
|
||||
) -> KbTree:
|
||||
"""The full recursive KB tree in ONE fetch (phase 97, task 02).
|
||||
def _kb_totals(doc_rows: Sequence[TreeDocRow]) -> KbTreeTotals:
|
||||
"""The catalogue totals beside the tree (phase 167, task 02, D2) —
|
||||
computed from the SAME rows the builder consumes (no extra query):
|
||||
``documents = len(rows)``, ``chunks`` = the sum of the per-document
|
||||
chunk counts, ``last_indexed`` = the MAX ``indexed_at`` stamp
|
||||
(lexicographic max = chronological max over one ISO-8601 shape),
|
||||
``None`` for an empty KB. The RAG view's stat cards read these
|
||||
because a depth-limited tree cannot be summed client-side (the
|
||||
phase-97 ``treeStats()`` walk only sees the sent levels).
|
||||
"""
|
||||
return KbTreeTotals(
|
||||
documents=len(doc_rows),
|
||||
chunks=sum(chunks for _s, _p, _t, chunks, _i, _c in doc_rows),
|
||||
last_indexed=max((indexed for _s, _p, _t, _ch, indexed, _c in doc_rows), default=None),
|
||||
)
|
||||
|
||||
Admin-only, like ``GET /api/docs`` — anonymous callers get 403
|
||||
``admin only`` (the RAG view's anonymous gate never fetches the
|
||||
tree). The RAG view drills CLIENT-side: this is the view's single
|
||||
fetch, zero per-level requests (the ``00_phase.md`` "The tree
|
||||
endpoint" contract).
|
||||
|
||||
Composition: the registry source names (``list_source_names`` —
|
||||
the superset rule's registry half, imported from ``app.rag.agent``
|
||||
exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped
|
||||
catalogue query ``GET /api/docs`` runs (the document ``id``
|
||||
excluded — the tree has no document ids) + ALL stored
|
||||
``folder_summaries`` rows (a bounded select — one row per
|
||||
existing folder at the ≥ 1-doc rule; rows for sources the tree
|
||||
does not list are never referenced by the builder) + the phase-122
|
||||
(task 04) image-docs map (a second bounded select over the
|
||||
``is_image`` rows with a servable ``image_path`` — empty for every
|
||||
pre-phase KB, so the response stays byte-identical to pre-phase)
|
||||
— through the pure :func:`build_kb_tree`. ``GET /api/docs`` itself
|
||||
is untouched.
|
||||
def _kb_catalogue(
|
||||
db: Session,
|
||||
) -> tuple[
|
||||
list[str],
|
||||
list[TreeDocRow],
|
||||
dict[tuple[str, str], str],
|
||||
dict[tuple[str, str], ImageDocInfo],
|
||||
]:
|
||||
"""The catalogue composition the tree routes share (phase 167,
|
||||
task 03) — ONE outerjoin/grouped catalogue query plus the two
|
||||
bounded side selects, no N+1 anywhere (the drill pays exactly what
|
||||
the first paint pays). Returns ``(names, doc_rows, summaries,
|
||||
images)``:
|
||||
|
||||
* *names* — the registry source names (``list_source_names`` — the
|
||||
superset rule's registry half, imported from ``app.rag.agent``
|
||||
exactly as ``app/api/chat.py`` does);
|
||||
* *doc_rows* — the SAME outerjoin/grouped catalogue query
|
||||
``GET /api/docs`` runs (the document ``id`` excluded — the tree
|
||||
has no document ids), both stamps converted to ISO-8601 HERE so
|
||||
the pure builder stays DB-free;
|
||||
* *summaries* — ALL stored ``folder_summaries`` rows (a bounded
|
||||
select — one row per existing folder at the ≥ 1-doc rule; rows
|
||||
for sources the tree does not list are never referenced by the
|
||||
builder);
|
||||
* *images* — the phase-122 (task 04) image-docs map: a second
|
||||
bounded select over the ``is_image`` rows with a servable
|
||||
``image_path`` only (a handful of rows at KB scale, never the
|
||||
whole catalog; the catalogue query above stays byte-identical).
|
||||
EMPTY for every pre-phase KB (no image rows), so the response
|
||||
stays byte-identical to pre-phase — the fields are row-driven,
|
||||
not toggle-driven (a surviving image doc keeps its thumbnail
|
||||
through a toggle-off sync, the prune guard's UX side).
|
||||
"""
|
||||
names = list_source_names(db)
|
||||
rows = db.execute(
|
||||
@@ -974,14 +1118,7 @@ def list_kb_tree(
|
||||
select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
|
||||
).all()
|
||||
}
|
||||
# Phase 122 (task 04): the image-docs affordance map — a bounded
|
||||
# select over the ``is_image`` rows only (a handful of rows at KB
|
||||
# scale, never the whole catalog; the catalogue query above stays
|
||||
# byte-identical). EMPTY for every pre-phase KB (no image rows), so
|
||||
# the response stays byte-identical to pre-phase — the fields are
|
||||
# row-driven, not toggle-driven (a surviving image doc keeps its
|
||||
# thumbnail through a toggle-off sync, the prune guard's UX side).
|
||||
images: dict[tuple[str, str], tuple[str, str]] = {
|
||||
images: dict[tuple[str, str], ImageDocInfo] = {
|
||||
(source, path): (str(doc_id), summary)
|
||||
for source, path, doc_id, summary in db.execute(
|
||||
select(
|
||||
@@ -992,4 +1129,163 @@ def list_kb_tree(
|
||||
).where(Document.is_image.is_(True), Document.image_path.is_not(None))
|
||||
).all()
|
||||
}
|
||||
return KbTree(sources=build_kb_tree(names, doc_rows, summaries, images))
|
||||
return names, doc_rows, summaries, images
|
||||
|
||||
|
||||
def _revalidated_json(request: Request, model: BaseModel) -> Response:
|
||||
"""Serialize once, hash the EXACT served bytes, answer 304 when the
|
||||
client already holds them (phase 167, task 03, D3/A4 — the
|
||||
:mod:`app.core.http_cache` pattern in ONE place for the three
|
||||
catalogue reads).
|
||||
|
||||
The client is ``fetch``: a matched conditional request is served
|
||||
from its cache transparently (fetch never surfaces the 304), and
|
||||
``must-revalidate`` guarantees a sync or an admin edit is seen on
|
||||
the next load — the tag changes because the body changes (the A4
|
||||
proof: a ``sources_meta.version``-derived tag would keep serving
|
||||
stale trees after the folder-summary / document-date PATCHes, which
|
||||
never bump it). ``model_dump_json`` runs exactly ONCE — the same
|
||||
string feeds the tag and the body, and the returned ``Response``
|
||||
short-circuits FastAPI's own serialization.
|
||||
"""
|
||||
body = model.model_dump_json()
|
||||
etag = weak_etag(body)
|
||||
if etag_matches(request.headers.get("if-none-match"), etag):
|
||||
return Response(status_code=304, headers=revalidate_headers(etag))
|
||||
return Response(
|
||||
content=body, media_type="application/json", headers=revalidate_headers(etag)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/docs/tree", response_model=KbTree)
|
||||
def list_kb_tree(
|
||||
request: Request,
|
||||
depth: int = Query(1, ge=1, le=8),
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
_admin: None = Depends(require_admin), # noqa: B008
|
||||
) -> Response:
|
||||
"""The KB tree, DEPTH-LIMITED by default (phase 97, task 02;
|
||||
phase 167, task 02, D2/A2 — the TODO L5 2 MB payload fix).
|
||||
|
||||
``depth`` bounds the levels built under each source:
|
||||
``1`` (default) = sources + ONE level — the RAG view's first paint;
|
||||
``8`` = the practical KB ceiling (the whole tree, the pre-phase
|
||||
shape plus the additive ``totals`` key). Every node whose children
|
||||
were NOT built carries ``truncated: true`` (with its recursive
|
||||
count, recursive ``updated_at``, ``summary`` and
|
||||
``summary_pending`` intact) so the view can drill level by level;
|
||||
``totals`` (documents / chunks / ``last_indexed``) is computed here
|
||||
from the full catalogue rows, so the stat cards stay honest at any
|
||||
depth.
|
||||
|
||||
Admin-only, like ``GET /api/docs`` — anonymous callers get 403
|
||||
``admin only`` (the RAG view's anonymous gate never fetches the
|
||||
tree). The phase-97 "one fetch, drill client-side" contract is
|
||||
deliberately superseded (LOCKED A3): one initial load plus one
|
||||
fetch per drilled level (task 03's node route).
|
||||
|
||||
Composition: :func:`_kb_catalogue` (the registry source names, the
|
||||
SAME outerjoin/grouped catalogue query ``GET /api/docs`` runs, ALL
|
||||
stored ``folder_summaries`` rows, and the phase-122 image-docs map)
|
||||
through the pure :func:`build_kb_tree` + :func:`_kb_totals`.
|
||||
``GET /api/docs`` itself is untouched.
|
||||
|
||||
Phase 167 (task 03, D3/A4): the response carries the content-exact
|
||||
weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair via
|
||||
:func:`_revalidated_json` — a repeat load costs a 304 with no body
|
||||
(the client is ``fetch``, which serves a matched conditional request
|
||||
from its cache transparently). The decorator keeps
|
||||
``response_model=KbTree`` for the OpenAPI schema only — a returned
|
||||
:class:`~fastapi.responses.Response` short-circuits FastAPI's
|
||||
serialization, so the model is dumped exactly ONCE here.
|
||||
"""
|
||||
names, doc_rows, summaries, images = _kb_catalogue(db)
|
||||
model = KbTree(
|
||||
sources=build_kb_tree(names, doc_rows, summaries, images, depth),
|
||||
totals=_kb_totals(doc_rows),
|
||||
)
|
||||
return _revalidated_json(request, model)
|
||||
|
||||
|
||||
@router.get("/docs/tree/node", response_model=None)
|
||||
def get_kb_tree_node(
|
||||
request: Request,
|
||||
source: str = Query(...),
|
||||
path: str = Query(""),
|
||||
depth: int = Query(1, ge=1, le=8),
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
_admin: None = Depends(require_admin), # noqa: B008
|
||||
) -> KbTreeSource | KbTreeFolder | Response:
|
||||
"""ONE tree node's direct children — the RAG view's per-level drill
|
||||
fetch (phase 167, task 03, D3/A3: the deliberate cost of the small
|
||||
first payload is one request per drilled level).
|
||||
|
||||
``source`` is the registry name OR an indexed source (the superset
|
||||
rule); ``path`` is SOURCE-RELATIVE, ``""`` (the default) = the
|
||||
source root. The node answers in the shape the full tree produces
|
||||
for the same node — a :class:`~app.schemas.KbTreeSource` for
|
||||
``path == ""``, else a :class:`~app.schemas.KbTreeFolder` carrying
|
||||
the recursive ``documents`` count, the recursive ``updated_at``
|
||||
(the phase-106 D9 max — the truncation-proof max-date map), its
|
||||
``summary`` and ``summary_pending`` (the unchanged phase-98 D3
|
||||
rule). The node's DIRECT children are always built — the *depth*
|
||||
budget only decides whether those children's subfolders recurse
|
||||
(the same rule that never truncates a SOURCE node at ``depth ≥ 1``,
|
||||
see :func:`_level_children`) — so the node itself is never
|
||||
``truncated``: what arrived is what the level renders, and deeper
|
||||
levels ride the children's own flags.
|
||||
|
||||
Admin-gated — the same :func:`~app.core.auth.require_admin` gate as
|
||||
``/api/docs/tree`` (anonymous → 403 ``admin only``). 404
|
||||
``{"detail": "unknown node"}`` when the source is NEITHER
|
||||
registered NOR indexed, or when a non-empty ``path`` fails the
|
||||
phase-94 existence rule (a folder exists ⟺ some indexed path
|
||||
starts with ``path + "/"`` — a document's OWN path is not a folder,
|
||||
unless a deeper path wears it).
|
||||
|
||||
Row-only, like every route in this file: NOTHING here builds a
|
||||
filesystem path from ``path`` — the node is derived from catalogue
|
||||
ROWS alone, so a ``../``-style value is simply not a folder and 404s
|
||||
with no filesystem surface (the phase-10 traversal argument, reused
|
||||
by :func:`build_kb_tree`).
|
||||
|
||||
Composition: the SAME catalogue query ``list_kb_tree`` runs (via
|
||||
:func:`_kb_catalogue` — ONE query per drill, no N+1; the tree's
|
||||
counts and recursive dates are only honest over the whole
|
||||
catalogue), then the pure helpers scoped to the one source. Carries
|
||||
the same ``ETag`` + ``private, must-revalidate`` pair as the other
|
||||
two catalogue reads (:func:`_revalidated_json`).
|
||||
``response_model=None`` on the decorator: the return is a UNION
|
||||
annotated for the reader while the body is returned as a raw
|
||||
``Response`` — the serializer must not run twice (once for the
|
||||
tag, once inside FastAPI).
|
||||
"""
|
||||
names, doc_rows, summaries, images = _kb_catalogue(db)
|
||||
rows: list[TreeFileRow] = [
|
||||
(row_path, title, chunks, indexed_at, created_at)
|
||||
for row_source, row_path, title, chunks, indexed_at, created_at in doc_rows
|
||||
if row_source == source
|
||||
]
|
||||
if source not in names and not rows:
|
||||
raise HTTPException(status_code=404, detail="unknown node")
|
||||
folders, counts, max_dates = _folder_counts(rows)
|
||||
if path and path not in folders:
|
||||
raise HTTPException(status_code=404, detail="unknown node")
|
||||
node: KbTreeSource | KbTreeFolder
|
||||
if path:
|
||||
children = _level_children(
|
||||
source, path, folders, counts, max_dates, rows, summaries, images, depth
|
||||
)
|
||||
node = KbTreeFolder(
|
||||
path=path,
|
||||
documents=counts[path],
|
||||
updated_at=max_dates[path],
|
||||
summary=summaries.get((source, path)),
|
||||
summary_pending=counts[path] >= MIN_DOCS_PER_FOLDER
|
||||
and (source, path) not in summaries,
|
||||
children=children,
|
||||
)
|
||||
else:
|
||||
# The source root — the very node the full tree builds.
|
||||
node = _source_node(source, rows, summaries, images, depth)
|
||||
return _revalidated_json(request, node)
|
||||
|
||||
Reference in New Issue
Block a user