Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 167_api_payload_diet
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
ducoterra
2026-10-08 21:23:49 -04:00
parent ab9391d019
commit 2ac37455f0
73 changed files with 7313 additions and 228 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/api/docs.py
+362 -66
View File
@@ -18,10 +18,34 @@ date folds to today, D3) + flag it manual, or clear the manual flag
write: NO LLM/embedding call — a date is never embedded (the
deliberate contrast with the phase-57 ``is_summary`` re-embed).
GET /api/docs/tree — the admin's full recursive KB tree in one fetch
(phase 97, task 02): the same drill-down tree the agent's ``ls``
walks, with the file metadata the RAG view's rows and stat cards need
(the view drills client-side; ``GET /api/docs`` is untouched).
GET /api/docs/tree — the admin's KB tree (phase 97, task 02; depth-
limited since phase 167, task 02): the same drill-down tree the agent's
``ls`` walks, with the file metadata the RAG view's rows and stat cards
need. The default ``depth=1`` ships the sources plus ONE level with
server-computed ``totals`` — a node whose children were not built
carries ``truncated: true`` (the TODO L5 2 MB payload fix; ``GET
/api/docs`` and the agent's ``ls`` path are untouched).
GET /api/docs/tree/node — ONE tree node's direct children (phase 167,
task 03, D3): the RAG view's per-level drill fetch — the same catalogue
composition as ``/api/docs/tree`` (ONE catalogue query per drill, no
N+1) resolved to a single ``KbTreeSource`` (``path=""``, the source
root) or ``KbTreeFolder``. Admin-gated like the tree; 404
``{"detail": "unknown node"}`` for a source that is neither registered
nor indexed, or a folder failing the phase-94 existence rule. Row-only
like every other route here: no request value ever reaches a
filesystem path — ``../``-style values simply are not folders (the
phase-10 traversal argument, reused).
Revalidation (phase 167, task 03, LOCKED A4 — a scoped extension of
A19): the three catalogue READ routes (``/api/docs``,
``/api/docs/tree``, ``/api/docs/tree/node``) serialize once and carry a
content-exact weak ``ETag`` + ``Cache-Control: private,
must-revalidate`` (the :mod:`app.core.http_cache` helpers), so a repeat
load costs a 304 with no body. The tag is a hash of the served bytes —
NOT ``sources_meta.version`` (bumping that would mark saved chats
stale, and the admin PATCHes do not bump it). ``sources_meta``, the
SSE stream and the mutating surface are untouched.
GET /api/documents/{doc_id}/image — the phase-122 (task 04) image
BYTES route: the persistent copy behind an ``is_image`` document's
@@ -70,14 +94,16 @@ from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response
from pydantic import BaseModel
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from app.api.sync import _sanitize_error
from app.config import get_settings
from app.core.auth import require_admin, require_user
from app.core.http_cache import etag_matches, revalidate_headers, weak_etag
from app.db import get_db
from app.models import Chunk, Document, FolderSummary
from app.rag.agent import list_source_names
@@ -99,6 +125,7 @@ from app.schemas import (
KbTreeFile,
KbTreeFolder,
KbTreeSource,
KbTreeTotals,
SummaryResult,
SummaryUpdate,
)
@@ -128,9 +155,10 @@ def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
@router.get("/docs", response_model=DocList)
def list_indexed_documents(
request: Request,
db: Session = Depends(get_db), # noqa: B008
_admin: None = Depends(require_admin), # noqa: B008
) -> DocList:
) -> Response:
"""All indexed documents with per-document chunk counts.
Admin-only (phase 16 — the catalog is what the admin sign-in gates;
@@ -139,6 +167,16 @@ def list_indexed_documents(
``admin only`` and the Sources page renders its sign-in gate instead.
An empty list means the knowledge base has not been imported yet —
the Sources page renders its designed empty state in that case.
Phase 167 (task 03, D3/A4): the response carries the content-exact
weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair
(see :mod:`app.core.http_cache`) — the client is ``fetch``, which
serves a matched conditional request from its cache transparently,
and ``must-revalidate`` guarantees a sync is seen on the next load
(the tag changes because the body changes). The decorator keeps
``response_model=DocList`` for the OpenAPI schema only — a returned
:class:`~fastapi.responses.Response` short-circuits FastAPI's
serialization, so the model is dumped exactly ONCE here.
"""
rows = db.execute(
select(
@@ -161,7 +199,7 @@ def list_indexed_documents(
)
.order_by(Document.source, Document.path)
).all()
return DocList(
model = DocList(
documents=[
DocSummary(
id=str(row.id),
@@ -175,6 +213,7 @@ def list_indexed_documents(
for row in rows
]
)
return _revalidated_json(request, model)
@router.get("/documents/content", response_model=DocContent)
@@ -625,8 +664,9 @@ TreeFileRow = tuple[str, str, int, str, str]
def _folder_counts(
rows: Sequence[TreeFileRow],
) -> tuple[set[str], dict[str, int]]:
"""One source's folders (existence rule) + recursive counts (pure).
) -> tuple[set[str], dict[str, int], dict[str, str]]:
"""One source's folders (existence rule) + recursive counts + the
recursive max-date map (all pure).
The phase-94 rules, reused verbatim from
:func:`app.rag.agent.group_folder_listing` (ONE concept end to end —
@@ -640,9 +680,22 @@ def _folder_counts(
``folder + "/"`` — exactly the set the sync-time folder summary
describes.
Returns ``(folders, counts)`` — the folder-name set and the
Returns ``(folders, counts, max_dates)`` — the folder-name set, the
per-folder count map (every folder counts ≥ 1 by construction: its
own descendants, or the file wearing its name, exist).
own descendants, or the file wearing its name, exist), and the
per-folder RECURSIVE MAX document ``created_at`` map — the same
subtree the count walks, over the same ISO-8601 strings (one
``isoformat()`` shape, so the lexicographic max is the
chronological one — the phase-106 D9 rule).
The max map is the phase-167 (task 02, D2) truncation-proof date
source: a depth-limited node has NO built children, so
:func:`_subtree_max` over its empty child list would read ``None``
and silently lose the D9 column; the aggregate — computed from the
rows in the same pass — keeps the date honest at every depth. The
expanded path keeps deriving ``updated_at`` via
:func:`_subtree_max` (one concept, two derivations — cross-pinned
equal by a unit test).
"""
folders: set[str] = set()
for path, _title, _chunks, _indexed_at, _created_at in rows:
@@ -651,14 +704,19 @@ def _folder_counts(
folders.add(folder)
folder = folder_of(folder)
counts: dict[str, int] = {folder: 0 for folder in folders}
for path, _title, _chunks, _indexed_at, _created_at in rows:
max_dates: dict[str, str] = {}
for path, _title, _chunks, _indexed_at, created_at in rows:
if path in folders:
counts[path] += 1
if created_at > max_dates.get(path, ""):
max_dates[path] = created_at
folder = folder_of(path)
while folder:
counts[folder] += 1
if created_at > max_dates.get(folder, ""):
max_dates[folder] = created_at
folder = folder_of(folder)
return folders, counts
return folders, counts, max_dates
#: One image-docs map value (phase 122, task 04):
@@ -673,9 +731,11 @@ def _level_children(
folder: str,
folders: set[str],
counts: dict[str, int],
max_dates: dict[str, str],
rows: Sequence[TreeFileRow],
summaries: Mapping[tuple[str, str], str],
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
depth: int | None = None,
) -> list[KbTreeFolder | KbTreeFile]:
"""One level's children (pure): subfolders in path order, then the
direct files in input (catalog) order.
@@ -691,6 +751,20 @@ def _level_children(
file list is NOT capped (the ``ls`` 50-line cap is a model-context
budget — the UI is for humans). Recurses one level per call.
*depth* (phase 167, task 02, D2/A2) is the REMAINING build budget
for this level's subfolders: ``None`` = unlimited (the pre-phase
recursion, byte-identical); a positive budget is spent here and
``depth - 1`` rides the recursion. When the budget is exhausted
(``depth == 1`` reaching its subfolders) the subfolder nodes are
TRUNCATED instead of recursed: ``children=[]``, ``truncated=True``,
and every field the level row still renders — the recursive
``documents`` count, the recursive ``updated_at`` from *max_dates*
(NOT ``_subtree_max`` over the empty child list — the D9 trap), the
node's own ``summary`` (the row shows a SUBFOLDER's description),
and ``summary_pending`` under the unchanged rule. The level's
DIRECT files are always built — the budget limits DEPTH, never the
row set of the levels it does send.
Each folder node also carries the phase-98 D3 ``summary_pending``
flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
stored ``folder_summaries`` row for ``(source, sub)`` — the same
@@ -709,21 +783,40 @@ def _level_children(
byte-identical).
"""
children: list[KbTreeFolder | KbTreeFile] = []
expand = depth is None or depth > 1
next_depth = None if depth is None else depth - 1
for sub in sorted(g for g in folders if folder_of(g) == folder):
sub_children = _level_children(
source, sub, folders, counts, rows, summaries, images
)
children.append(
KbTreeFolder(
path=sub,
documents=counts[sub],
updated_at=_subtree_max(sub_children),
summary=summaries.get((source, sub)),
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
and (source, sub) not in summaries,
children=sub_children,
if expand:
sub_children = _level_children(
source, sub, folders, counts, max_dates, rows, summaries, images, next_depth
)
children.append(
KbTreeFolder(
path=sub,
documents=counts[sub],
updated_at=_subtree_max(sub_children),
summary=summaries.get((source, sub)),
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
and (source, sub) not in summaries,
children=sub_children,
)
)
else:
# Phase 167 (D2): budget exhausted — the node rides with its
# recursive aggregate fields and truncated=True; the client
# drills via the node route (task 03).
children.append(
KbTreeFolder(
path=sub,
documents=counts[sub],
updated_at=max_dates[sub],
summary=summaries.get((source, sub)),
summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
and (source, sub) not in summaries,
children=[],
truncated=True,
)
)
)
for path, title, chunks, indexed_at, created_at in rows:
if folder_of(path) == folder:
image_info = images.get((source, path)) if images else None
@@ -761,7 +854,10 @@ def _subtree_max(children: Sequence[KbTreeFolder | KbTreeFile]) -> str | None:
"""A node's ``updated_at`` (phase 106, D9): the subtree's MAX
document ``created_at``, computed from the node's direct children —
files contribute their ``created_at``, subfolder nodes contribute
their (already recursive) ``updated_at``.
their (already recursive) ``updated_at``. UNCHANGED for the
expanded path since phase 167 (the truncated path uses the
:func:`_folder_counts` max map instead — one concept, two
derivations, cross-pinned equal by a unit test).
All values share the same UTC ``isoformat()`` shape (the endpoint
converts every ``created_at`` before the builder runs), so the
@@ -782,6 +878,7 @@ def build_kb_tree(
doc_rows: Sequence[TreeDocRow],
summaries: Mapping[tuple[str, str], str],
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
depth: int | None = None,
) -> list[KbTreeSource]:
"""The pure tree builder behind ``GET /api/docs/tree`` (phase 97,
task 02) — module-level and DB-free so unit tests drive it
@@ -847,6 +944,17 @@ def build_kb_tree(
pending, single-file folders included — and a registered
0-document source never is.
*depth* (phase 167, task 02, D2/A2) bounds how deep the tree is
BUILT: ``None`` (the default) = unlimited — every pre-phase call
and expectation is byte-identical; a positive budget stops the
recursion (see :func:`_level_children`) so the response carries the
sources plus *depth* levels, every node whose children were not
built flagged ``truncated=True`` with its recursive count, its
recursive ``updated_at`` (the max-date map — D9 survives
truncation), its ``summary`` and ``summary_pending``. A SOURCE node
is never truncated at ``depth ≥ 1`` — the view's top level is
source → level 1, so the source's direct children are always built.
ONE concept end to end: the builder reuses
:func:`app.rag.folder_summaries.folder_of` and the phase-94
existence / count rules, so — for a single-source dataset — its
@@ -872,10 +980,14 @@ def build_kb_tree(
if name in listed: # defensive: list_source_names dedupes
continue
listed.add(name)
tree.append(_source_node(name, by_source.get(name, ()), summaries, images))
tree.append(
_source_node(name, by_source.get(name, ()), summaries, images, depth)
)
for source in sorted(by_source):
if source not in listed:
tree.append(_source_node(source, by_source[source], summaries, images))
tree.append(
_source_node(source, by_source[source], summaries, images, depth)
)
return tree
@@ -884,6 +996,7 @@ def _source_node(
rows: Sequence[TreeFileRow],
summaries: Mapping[tuple[str, str], str],
images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
depth: int | None = None,
) -> KbTreeSource:
"""One source node (pure): whole-source count + the source-root
summary + the root level's children (direct subfolders + direct
@@ -902,10 +1015,17 @@ def _source_node(
``updated_at`` (phase 106, D9): the source's subtree max —
:func:`_subtree_max` over the root level's children; ``None`` for a
0-document source (no children, no dates).
0-document source (no children, no dates). Correct at every depth:
a TRUNCATED child still carries its recursive ``updated_at`` (the
max-date map), so the source's max never silently drops the levels
below the budget (phase 167, D2). The SOURCE node itself is never
marked ``truncated`` at ``depth ≥ 1`` — its direct children are
always built (the view's top level).
"""
folders, counts = _folder_counts(rows)
children = _level_children(source, "", folders, counts, rows, summaries, images)
folders, counts, max_dates = _folder_counts(rows)
children = _level_children(
source, "", folders, counts, max_dates, rows, summaries, images, depth
)
return KbTreeSource(
name=source,
documents=len(rows),
@@ -916,32 +1036,56 @@ def _source_node(
)
@router.get("/docs/tree", response_model=KbTree)
def list_kb_tree(
db: Session = Depends(get_db), # noqa: B008
_admin: None = Depends(require_admin), # noqa: B008
) -> KbTree:
"""The full recursive KB tree in ONE fetch (phase 97, task 02).
def _kb_totals(doc_rows: Sequence[TreeDocRow]) -> KbTreeTotals:
"""The catalogue totals beside the tree (phase 167, task 02, D2) —
computed from the SAME rows the builder consumes (no extra query):
``documents = len(rows)``, ``chunks`` = the sum of the per-document
chunk counts, ``last_indexed`` = the MAX ``indexed_at`` stamp
(lexicographic max = chronological max over one ISO-8601 shape),
``None`` for an empty KB. The RAG view's stat cards read these
because a depth-limited tree cannot be summed client-side (the
phase-97 ``treeStats()`` walk only sees the sent levels).
"""
return KbTreeTotals(
documents=len(doc_rows),
chunks=sum(chunks for _s, _p, _t, chunks, _i, _c in doc_rows),
last_indexed=max((indexed for _s, _p, _t, _ch, indexed, _c in doc_rows), default=None),
)
Admin-only, like ``GET /api/docs`` — anonymous callers get 403
``admin only`` (the RAG view's anonymous gate never fetches the
tree). The RAG view drills CLIENT-side: this is the view's single
fetch, zero per-level requests (the ``00_phase.md`` "The tree
endpoint" contract).
Composition: the registry source names (``list_source_names`` —
the superset rule's registry half, imported from ``app.rag.agent``
exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped
catalogue query ``GET /api/docs`` runs (the document ``id``
excluded — the tree has no document ids) + ALL stored
``folder_summaries`` rows (a bounded select — one row per
existing folder at the ≥ 1-doc rule; rows for sources the tree
does not list are never referenced by the builder) + the phase-122
(task 04) image-docs map (a second bounded select over the
``is_image`` rows with a servable ``image_path`` — empty for every
pre-phase KB, so the response stays byte-identical to pre-phase)
— through the pure :func:`build_kb_tree`. ``GET /api/docs`` itself
is untouched.
def _kb_catalogue(
db: Session,
) -> tuple[
list[str],
list[TreeDocRow],
dict[tuple[str, str], str],
dict[tuple[str, str], ImageDocInfo],
]:
"""The catalogue composition the tree routes share (phase 167,
task 03) — ONE outerjoin/grouped catalogue query plus the two
bounded side selects, no N+1 anywhere (the drill pays exactly what
the first paint pays). Returns ``(names, doc_rows, summaries,
images)``:
* *names* — the registry source names (``list_source_names`` — the
superset rule's registry half, imported from ``app.rag.agent``
exactly as ``app/api/chat.py`` does);
* *doc_rows* — the SAME outerjoin/grouped catalogue query
``GET /api/docs`` runs (the document ``id`` excluded — the tree
has no document ids), both stamps converted to ISO-8601 HERE so
the pure builder stays DB-free;
* *summaries* — ALL stored ``folder_summaries`` rows (a bounded
select — one row per existing folder at the ≥ 1-doc rule; rows
for sources the tree does not list are never referenced by the
builder);
* *images* — the phase-122 (task 04) image-docs map: a second
bounded select over the ``is_image`` rows with a servable
``image_path`` only (a handful of rows at KB scale, never the
whole catalog; the catalogue query above stays byte-identical).
EMPTY for every pre-phase KB (no image rows), so the response
stays byte-identical to pre-phase — the fields are row-driven,
not toggle-driven (a surviving image doc keeps its thumbnail
through a toggle-off sync, the prune guard's UX side).
"""
names = list_source_names(db)
rows = db.execute(
@@ -974,14 +1118,7 @@ def list_kb_tree(
select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
).all()
}
# Phase 122 (task 04): the image-docs affordance map — a bounded
# select over the ``is_image`` rows only (a handful of rows at KB
# scale, never the whole catalog; the catalogue query above stays
# byte-identical). EMPTY for every pre-phase KB (no image rows), so
# the response stays byte-identical to pre-phase — the fields are
# row-driven, not toggle-driven (a surviving image doc keeps its
# thumbnail through a toggle-off sync, the prune guard's UX side).
images: dict[tuple[str, str], tuple[str, str]] = {
images: dict[tuple[str, str], ImageDocInfo] = {
(source, path): (str(doc_id), summary)
for source, path, doc_id, summary in db.execute(
select(
@@ -992,4 +1129,163 @@ def list_kb_tree(
).where(Document.is_image.is_(True), Document.image_path.is_not(None))
).all()
}
return KbTree(sources=build_kb_tree(names, doc_rows, summaries, images))
return names, doc_rows, summaries, images
def _revalidated_json(request: Request, model: BaseModel) -> Response:
"""Serialize once, hash the EXACT served bytes, answer 304 when the
client already holds them (phase 167, task 03, D3/A4 — the
:mod:`app.core.http_cache` pattern in ONE place for the three
catalogue reads).
The client is ``fetch``: a matched conditional request is served
from its cache transparently (fetch never surfaces the 304), and
``must-revalidate`` guarantees a sync or an admin edit is seen on
the next load — the tag changes because the body changes (the A4
proof: a ``sources_meta.version``-derived tag would keep serving
stale trees after the folder-summary / document-date PATCHes, which
never bump it). ``model_dump_json`` runs exactly ONCE — the same
string feeds the tag and the body, and the returned ``Response``
short-circuits FastAPI's own serialization.
"""
body = model.model_dump_json()
etag = weak_etag(body)
if etag_matches(request.headers.get("if-none-match"), etag):
return Response(status_code=304, headers=revalidate_headers(etag))
return Response(
content=body, media_type="application/json", headers=revalidate_headers(etag)
)
@router.get("/docs/tree", response_model=KbTree)
def list_kb_tree(
request: Request,
depth: int = Query(1, ge=1, le=8),
db: Session = Depends(get_db), # noqa: B008
_admin: None = Depends(require_admin), # noqa: B008
) -> Response:
"""The KB tree, DEPTH-LIMITED by default (phase 97, task 02;
phase 167, task 02, D2/A2 — the TODO L5 2 MB payload fix).
``depth`` bounds the levels built under each source:
``1`` (default) = sources + ONE level — the RAG view's first paint;
``8`` = the practical KB ceiling (the whole tree, the pre-phase
shape plus the additive ``totals`` key). Every node whose children
were NOT built carries ``truncated: true`` (with its recursive
count, recursive ``updated_at``, ``summary`` and
``summary_pending`` intact) so the view can drill level by level;
``totals`` (documents / chunks / ``last_indexed``) is computed here
from the full catalogue rows, so the stat cards stay honest at any
depth.
Admin-only, like ``GET /api/docs`` — anonymous callers get 403
``admin only`` (the RAG view's anonymous gate never fetches the
tree). The phase-97 "one fetch, drill client-side" contract is
deliberately superseded (LOCKED A3): one initial load plus one
fetch per drilled level (task 03's node route).
Composition: :func:`_kb_catalogue` (the registry source names, the
SAME outerjoin/grouped catalogue query ``GET /api/docs`` runs, ALL
stored ``folder_summaries`` rows, and the phase-122 image-docs map)
through the pure :func:`build_kb_tree` + :func:`_kb_totals`.
``GET /api/docs`` itself is untouched.
Phase 167 (task 03, D3/A4): the response carries the content-exact
weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair via
:func:`_revalidated_json` — a repeat load costs a 304 with no body
(the client is ``fetch``, which serves a matched conditional request
from its cache transparently). The decorator keeps
``response_model=KbTree`` for the OpenAPI schema only — a returned
:class:`~fastapi.responses.Response` short-circuits FastAPI's
serialization, so the model is dumped exactly ONCE here.
"""
names, doc_rows, summaries, images = _kb_catalogue(db)
model = KbTree(
sources=build_kb_tree(names, doc_rows, summaries, images, depth),
totals=_kb_totals(doc_rows),
)
return _revalidated_json(request, model)
@router.get("/docs/tree/node", response_model=None)
def get_kb_tree_node(
request: Request,
source: str = Query(...),
path: str = Query(""),
depth: int = Query(1, ge=1, le=8),
db: Session = Depends(get_db), # noqa: B008
_admin: None = Depends(require_admin), # noqa: B008
) -> KbTreeSource | KbTreeFolder | Response:
"""ONE tree node's direct children — the RAG view's per-level drill
fetch (phase 167, task 03, D3/A3: the deliberate cost of the small
first payload is one request per drilled level).
``source`` is the registry name OR an indexed source (the superset
rule); ``path`` is SOURCE-RELATIVE, ``""`` (the default) = the
source root. The node answers in the shape the full tree produces
for the same node — a :class:`~app.schemas.KbTreeSource` for
``path == ""``, else a :class:`~app.schemas.KbTreeFolder` carrying
the recursive ``documents`` count, the recursive ``updated_at``
(the phase-106 D9 max — the truncation-proof max-date map), its
``summary`` and ``summary_pending`` (the unchanged phase-98 D3
rule). The node's DIRECT children are always built — the *depth*
budget only decides whether those children's subfolders recurse
(the same rule that never truncates a SOURCE node at ``depth ≥ 1``,
see :func:`_level_children`) — so the node itself is never
``truncated``: what arrived is what the level renders, and deeper
levels ride the children's own flags.
Admin-gated — the same :func:`~app.core.auth.require_admin` gate as
``/api/docs/tree`` (anonymous → 403 ``admin only``). 404
``{"detail": "unknown node"}`` when the source is NEITHER
registered NOR indexed, or when a non-empty ``path`` fails the
phase-94 existence rule (a folder exists ⟺ some indexed path
starts with ``path + "/"`` — a document's OWN path is not a folder,
unless a deeper path wears it).
Row-only, like every route in this file: NOTHING here builds a
filesystem path from ``path`` — the node is derived from catalogue
ROWS alone, so a ``../``-style value is simply not a folder and 404s
with no filesystem surface (the phase-10 traversal argument, reused
by :func:`build_kb_tree`).
Composition: the SAME catalogue query ``list_kb_tree`` runs (via
:func:`_kb_catalogue` — ONE query per drill, no N+1; the tree's
counts and recursive dates are only honest over the whole
catalogue), then the pure helpers scoped to the one source. Carries
the same ``ETag`` + ``private, must-revalidate`` pair as the other
two catalogue reads (:func:`_revalidated_json`).
``response_model=None`` on the decorator: the return is a UNION
annotated for the reader while the body is returned as a raw
``Response`` — the serializer must not run twice (once for the
tag, once inside FastAPI).
"""
names, doc_rows, summaries, images = _kb_catalogue(db)
rows: list[TreeFileRow] = [
(row_path, title, chunks, indexed_at, created_at)
for row_source, row_path, title, chunks, indexed_at, created_at in doc_rows
if row_source == source
]
if source not in names and not rows:
raise HTTPException(status_code=404, detail="unknown node")
folders, counts, max_dates = _folder_counts(rows)
if path and path not in folders:
raise HTTPException(status_code=404, detail="unknown node")
node: KbTreeSource | KbTreeFolder
if path:
children = _level_children(
source, path, folders, counts, max_dates, rows, summaries, images, depth
)
node = KbTreeFolder(
path=path,
documents=counts[path],
updated_at=max_dates[path],
summary=summaries.get((source, path)),
summary_pending=counts[path] >= MIN_DOCS_PER_FOLDER
and (source, path) not in summaries,
children=children,
)
else:
# The source root — the very node the full tree builds.
node = _source_node(source, rows, summaries, images, depth)
return _revalidated_json(request, node)
app/config.py
+52
View File
@@ -46,6 +46,35 @@ class Settings(BaseSettings):
log_level: str = "INFO"
static_dir: str = "frontend"
# --- HTTP response compression (phase 167, TODO L5) ---
#: Byte floor for the selective gzip layer
#: (``app/core/compression.py``, ``BOR_COMPRESS_MIN_BYTES``): responses
#: of at least this many bytes are gzipped when the client accepts it
#: (Starlette's ``GZipMiddleware`` owns the negotiation, the
#: already-encoded skip and the ``Vary`` append). ``0`` = the feature
#: is OFF — the middleware is not added at all (the kill switch);
#: ``> 0`` = the Starlette minimum-size floor, default 860 — the
#: classic nginx default, below which compression can INFLATE a body
#: instead of shrinking it. A negative value is a typo — the validator
#: fails startup loudly (the ``agent_max_rounds`` pattern). The SSE
#: stream (``POST /api/chat``, A15) is never compressed at any value.
compress_min_bytes: int = 860
# --- Response byte budget (phase 167, TODO L5) ---
#: INFO/DEBUG cut for the per-response ``api method=… bytes=…`` line
#: (``app/core/response_budget.py``, ``BOR_LOG_LARGE_RESPONSE_BYTES``).
#: ``/api/*`` responses of at least this many bytes (the bytes the ROUTE
#: produced, before compression) log at INFO; below it the line is
#: DEBUG, i.e. invisible at the default ``BOR_LOG_LEVEL=INFO`` so the
#: log stays readable while "which endpoint is heavy, right now" is
#: still one ``grep ' :: api '`` away. Default 65536 (64 KB) — the
#: "fine" bar of the phase-167 audit; the audit's 256 KB budget is what
#: marks an offender (LOCKED A5: a log line and a script, never a 413).
#: ``0`` = log EVERY ``/api/*`` response at INFO (the loud mode, for a
#: one-off investigation); a negative value is a typo — the validator
#: fails startup loudly (the ``agent_max_rounds`` pattern).
log_large_response_bytes: int = 65536
# --- UI customization (phase 62, TODO L3) ---
# Defaults are the phase-61 neutral copy — UNSET => byte-identical UI.
# (Phase 91, task 03: the retired CSS-file theme env var is gone —
@@ -768,6 +797,29 @@ class Settings(BaseSettings):
raise ValueError("chat_max_concurrent must be >= 1")
return v
@field_validator("compress_min_bytes")
@classmethod
def _compress_min_bytes_non_negative(cls, v: int) -> int:
"""``0`` is the compression kill switch (the middleware is not
added at all) — a negative value is a typo (the
``agent_max_rounds`` pattern, phase 167)."""
if v < 0:
raise ValueError("compress_min_bytes must be >= 0 (0 = compression disabled)")
return v
@field_validator("log_large_response_bytes")
@classmethod
def _log_large_response_bytes_non_negative(cls, v: int) -> int:
"""``0`` is the loud mode (every ``/api/*`` response logs its byte
line at INFO) — a negative value is a typo that would make the
comparison constant, so it fails startup naming the field (the
``agent_max_rounds`` pattern, phase 167)."""
if v < 0:
raise ValueError(
"log_large_response_bytes must be >= 0 (0 = log every /api/* response)"
)
return v
@field_validator("docs_branch", "docs_base_branch")
@classmethod
def _docs_branch_tokens(cls, v: str, info: ValidationInfo) -> str:
app/core/compression.py
+179
View File
@@ -0,0 +1,179 @@
"""Selective gzip for every HTTP response but the SSE chat stream (phase 167, D1, TODO L5).
The defect (TODO L5): ``GET /api/docs/tree`` loads ~2 MB in production and
the Sources page pays that on every sync and every shell boot. Nothing in
this stack compressed — ``app/main.py`` added only Session / caching /
security-headers, and no response ever carried ``Content-Encoding``. KB
JSON is extremely repetitive (key names, paths, ISO timestamps), so
gzipping it lands at roughly **8-12× smaller** on the wire: the same bytes
the client already knows how to read, ~one order of magnitude faster.
The SSE stream is EXCLUDED (A15, phase 167 LOCKED A1): ``POST /api/chat``
is a ``text/event-stream`` whose whole value is that frames arrive *while
the model is still talking*. Any middleware that buffers the response —
``BaseHTTPMiddleware`` is the classic offender, and the reason this module
is **pure ASGI** — would deliver the entire turn at the end, silently
breaking the typing animation, the tool lines, and the cancel surface. So
the skip is by EXACT path (never a prefix: ``/api/chat-images/…`` keeps
its compression) and the skipped scope is handed to the app untouched.
The compression math is deliberately NOT re-implemented here. Starlette's
:class:`~starlette.middleware.gzip.GZipMiddleware` owns ``Accept-Encoding``
negotiation (a client that does not speak gzip gets identity bytes), the
``minimum_size`` floor, the skip when a response ALREADY sets
``content-encoding``, the ``Vary: Accept-Encoding`` append, and the
streaming ``compressobj`` + final flush. It is also what makes the stack
cooperate with a fronting proxy: **nginx may also gzip, but an
already-encoded body is never double-encoded** — Starlette preserves the
upstream ``Content-Encoding`` verbatim, and so does nginx.
One seam IS patched here, and it is framing, not math: the
``CachingMiddleware`` below us is a ``BaseHTTPMiddleware``, and starlette
1.x re-frames every response it emits upstream (``_StreamingResponse``
ships each body chunk with ``more_body=True`` plus an empty terminator),
so Starlette's buffered small-response exemption can never fire for
anything that travelled through it — a 33-byte ``/api/health`` would
stream into the compressor and come back INFLATED, exactly what the
``minimum_size`` floor exists to prevent. :class:`_RebufferSmallResponses`
sits under the responder and hands it back the single buffered body
message the exemption was written for, bounded to the floor.
"""
from __future__ import annotations
from starlette.datastructures import Headers
from starlette.middleware.gzip import GZipMiddleware
from starlette.types import ASGIApp, Message, Receive, Scope, Send
#: Paths that are NEVER compressed, matched EXACTLY (never by prefix).
#: ``POST /api/chat`` is the SSE stream (A15): frame-by-frame delivery is
#: the entire point of the endpoint, and buffering the response to feed it
#: through a compressor would deliver the whole turn at completion (and a
#: never-ending stream never completes on a cancel). The exact match is
#: load-bearing — ``/api/chat-images/…`` shares the prefix and must keep
#: routing through the compression layer.
DEFAULT_SKIP_PATHS: tuple[str, ...] = ("/api/chat",)
class _RebufferSmallResponses:
"""Hold a declared-small body so Starlette's floor exemption fires.
When ``http.response.start`` carries ``content-length`` below
``minimum_size`` — byte-for-byte the condition Starlette's buffered
path checks — the messages are held (bounded by the floor) and
re-emitted as ONE ``http.response.body`` with ``more_body=False``,
undoing the ``BaseHTTPMiddleware`` re-framing so the responder's
small-response exemption sees what it was written to see and the
bytes stay identity (no ``Content-Encoding``, no ``Vary`` append).
A lying header can only cost the buffer, never correctness: if the
body outgrows the floor despite the header, everything held is
forwarded as it came and the rest streams through the normal
(compressing) path. Non-body messages while holding (e.g.
``http.response.pathsend``) also flush the hold untouched.
"""
def __init__(self, app: ASGIApp, *, minimum_size: int) -> None:
self.app = app
self.minimum_size = minimum_size
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
holding = False
armed = True # the start decision happens at most once per response
held: list[Message] = []
total = 0
async def send_filter(message: Message) -> None:
nonlocal holding, armed, total
message_type = message["type"]
if message_type == "http.response.start" and armed:
armed = False
length = Headers(scope=message).get("content-length")
if (
length is not None
and length.isdigit()
and int(length) < self.minimum_size
):
held.append(message)
holding = True
return
await send(message)
return
if holding and message_type == "http.response.body":
held.append(message)
total += len(message.get("body", b""))
if not message.get("more_body", False):
# Whole body in hand, still under the floor: re-emit
# it as the single buffered message the exemption wants.
start = held[0]
merged = b"".join(m.get("body", b"") for m in held[1:])
holding = False
held.clear()
await send(start)
await send({"type": "http.response.body", "body": merged, "more_body": False})
elif total >= self.minimum_size:
# The content-length lied — stop holding, stream on.
holding = False
for held_message in held:
await send(held_message)
held.clear()
return
if holding:
# pathsend or anything unexpected mid-hold: flush as-is.
holding = False
for held_message in held:
await send(held_message)
held.clear()
await send(message)
await self.app(scope, receive, send_filter)
class SelectiveGZipMiddleware:
"""Pure-ASGI selective gzip (phase 167, D1 — no ``BaseHTTPMiddleware``).
Holds the wrapped ``app`` and a composed
:class:`~starlette.middleware.gzip.GZipMiddleware` built over the SAME
app (via the :class:`_RebufferSmallResponses` seam, so the floor still
works under the caching layer's re-framing); ``__call__`` delegates to
the gzip path unless the scope is not HTTP (websocket / lifespan pass
straight through) or the request path is one of ``skip_paths`` — in
which case the app is called DIRECTLY, with the original
``receive``/``send``, so a streaming response reaches the client frame
by frame, uncompressed.
``minimum_size`` is the Starlette floor (bytes): responses smaller than
it are forwarded without compression because gzip can make them
bigger. It is supplied by the caller (``Settings.compress_min_bytes``;
``0`` removes this middleware entirely at wiring time — see
``app/main.py``). The SSE stream (``POST /api/chat``, A15) is never
compressed at any value.
"""
def __init__(
self,
app: ASGIApp,
*,
minimum_size: int,
skip_paths: tuple[str, ...] = DEFAULT_SKIP_PATHS,
) -> None:
self.app = app
self.minimum_size = minimum_size
self.skip_paths = tuple(skip_paths)
# The compression itself stays Starlette's: negotiation, floor,
# already-encoded skip, Vary append, streaming compressobj + flush.
self.gzip = GZipMiddleware(
_RebufferSmallResponses(app, minimum_size=minimum_size),
minimum_size=minimum_size,
)
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http" or scope["path"] in self.skip_paths:
# Non-HTTP scopes have no response to compress; the SSE stream
# must never see this layer (A15 — see DEFAULT_SKIP_PATHS).
await self.app(scope, receive, send)
return
await self.gzip(scope, receive, send)
app/core/http_cache.py
+112
View File
@@ -0,0 +1,112 @@
"""HTTP revalidation validators for the read-only catalogue reads
(phase 167, task 03, D3 — LOCKED A4).
Three pure functions, no I/O, ONE place for the pattern (the routes
compose them; the alternative was a copy per route):
* :func:`weak_etag` — the validator: a weak ``ETag`` over the EXACT
bytes being served,
* :func:`etag_matches` — the tolerant ``If-None-Match`` evaluator,
* :func:`revalidate_headers` — the header pair a validated response
must carry.
**Why a content hash, not ``sources_meta.version``** (LOCKED A4):
bumping that generation marks every saved chat ``stale`` (phase 53) —
the wrong tool for a payload revalidation — and the admin PATCHes
(``/api/documents/summary``, ``/api/documents/date``,
``/api/folders/summary``) mutate the tree WITHOUT bumping it, so a
version-derived tag would serve stale trees. A hash of the serialized
body is content-exact by construction: the tag changes ⟺ the body
changes, it needs no migration, and the tree is built anyway (the hash
costs microseconds next to the query). The tag is over the EXACT served
bytes — the endpoint serializes once and hands those bytes to both the
hash and the ``Response``.
**Why a truncation to 32 hex chars is safe**: 32 hex chars of SHA-256
is this file's truncation convention. Collisions are not a correctness
risk for a revalidated cache: a false match only ever costs one stale
repaint on a ~2 MB namespace (the next unconditional load corrects it),
while 128 bits of digest make even that astronomically unlikely.
A19 SCOPE (:func:`revalidate_headers`, LOCKED A4 as a recorded, scoped
extension of A19): HTML pages stay ``no-cache`` with NO validators (the
caching middleware strips inbound conditionals and outbound validators —
the phase-54 contract), ``/assets/*`` stay ``immutable`` for a year
(URL-versioned), and this module governs the READ-ONLY API arm only —
today the two KB-tree routes and ``GET /api/docs``. The SSE stream and
every mutating route are untouched.
"""
from __future__ import annotations
import hashlib
#: Hex chars of the SHA-256 digest the tag carries (see the module
#: docstring's collision argument).
_TAG_HEX_CHARS = 32
#: The ``Cache-Control`` paired with every validator here: ``private``
#: (no shared cache may store the admin-gated catalogue) +
#: ``must-revalidate`` (a sync or an admin edit is seen on the NEXT
#: load — the client may never serve these bytes without asking, which
#: is what distinguishes this from the ``/assets/*`` posture).
REVALIDATE_CACHE_CONTROL = "private, must-revalidate"
def weak_etag(body: str | bytes) -> str:
"""Weak validator over the exact served bytes: ``W/"<sha256 hex[:32]>"``.
Content-exact by construction (the endpoint hashes the SAME string
it returns — see the module docstring for the A4 argument and the
truncation convention). ``str`` bodies are hashed as UTF-8, so the
str and its ``bytes`` form produce the SAME tag. Weak (``W/``)
because the comparison is byte-equivalence under revalidation, not
bit-identity — a differently gzipped encoding of the same content
must still match (RFC 9110 §8.8.3).
"""
raw = body.encode("utf-8") if isinstance(body, str) else body
return f'W/"{hashlib.sha256(raw).hexdigest()[:_TAG_HEX_CHARS]}"'
def _strip_weakness(candidate: str) -> str:
"""One comparable form: trimmed, with the ``W/`` weakness marker gone
(weak/strong equivalence — ``W/"x"`` matches ``"x"`` in BOTH
directions, RFC 9110 §8.8.3.2's weak comparison rule applied in both
directions on purpose: our tag is over the exact bytes, so a client
that stored either form HAS these bytes)."""
value = candidate.strip()
if value[:2].upper() == "W/":
value = value[2:].strip()
return value
def etag_matches(header: str | None, tag: str) -> bool:
"""Does an inbound ``If-None-Match`` header value match our ``tag``?
Tolerant the way real clients send it: ``*`` matches (RFC 9110
§13.3.3 — the client holds nothing it trusts), a comma-separated
candidate list matches on ANY entry, weak/strong forms are
equivalent in both directions, and surrounding whitespace is
ignored. ``None`` or an empty/whitespace-only header never matches
(a request with no conditional header is an unconditional GET → 200).
"""
if not header or not header.strip():
return False
candidates = header.split(",")
if any(candidate.strip() == "*" for candidate in candidates):
return True
target = _strip_weakness(tag)
return any(_strip_weakness(candidate) == target for candidate in candidates)
def revalidate_headers(etag: str) -> dict[str, str]:
"""The ONE header pair every validated catalogue response carries —
on the 200 AND on the 304 (RFC 9110 §8.8.3: a 304 must repeat the
validator, or the client cannot extend the stored entry's life).
A19 scope (module docstring): HTML pages stay ``no-cache`` with no
validators, ``/assets/*`` stay ``immutable``; this pair is the
read-only API arm — ``private, must-revalidate`` so a sync or an
admin edit is seen on the next load, because the tag changes exactly
when the body changes.
"""
return {"ETag": etag, "Cache-Control": REVALIDATE_CACHE_CONTROL}
app/core/response_budget.py
+211
View File
@@ -0,0 +1,211 @@
"""The byte budget: ONE log line per ``/api/*`` response (phase 167, task 05, D5, LOCKED A5).
TODO L5 asked "which endpoint is heavy, right now, in production?" — and
before this phase the app could not answer it. The 2 MB
``GET /api/docs/tree`` defect was found by a human eyeballing a browser
network tab; every other candidate endpoint was a guess. This module makes
the answer a ``grep`` away:
.. code-block:: text
api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41
* ``bytes`` is the size of the body the ROUTE produced — the sum of the
``http.response.body`` chunk lengths as they pass through, uncompressed
even when the wire carried gzip (the compression layer above us encodes
AFTER we counted). ``enc`` reports what actually went on the wire
(``content-encoding``, else ``identity``). The two together answer "how
fat is it, and how fat did the client see it".
* level: ``INFO`` when ``bytes >= settings.log_large_response_bytes``
(``BOR_LOG_LARGE_RESPONSE_BYTES``, default 65536; ``0`` = every ``/api/*``
response at INFO), ``DEBUG`` below it — so the default log stays quiet
and an offender is loud without a re-deploy.
* a ``304`` logs ``bytes=0`` deliberately: the revalidation win this phase
bought is visible in the log, not hidden behind a missing line.
* `` stream=1`` is appended for ``text/event-stream`` responses, where
``bytes`` is the streamed total and ``duration_ms`` is the stream's LIFE.
Why **pure ASGI** (the phase-167 rule for anything on this path): the
middleware must never buffer. It only reads message lengths, so the SSE
chat stream (A15) passes through byte-identical — a
``BaseHTTPMiddleware`` here would deliver the whole turn at the end,
which is exactly the failure mode phase 167 exists to avoid elsewhere.
Logged on the ``http.response.body`` message with ``more_body=False`` —
for a stream that is when the **stream ends**, so ``duration_ms`` is the
stream's whole life. A stream the client walks away from (a cancelled SSE
turn) writes exactly one line too, with the bytes that actually got out:
the app unwinds without a completing body message and the ``finally`` path
emits it. Half a response happened, so half a count is the honest number —
and no line at all would make a chatty, abandoned poll look free.
An app exception is handled the same way (one line, bytes as counted, status
500 when the app never got to answer) and re-raises UNCHANGED; non-HTTP
scopes and non-``/api/`` paths pass through silently (pages, assets and the
static catch-all are not the catalogue surface this budget governs).
"""
from __future__ import annotations
import logging
import time
from starlette.datastructures import Headers
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from app.config import get_settings
#: The logger behind every line — renders as ``… :: api method=…`` with the
#: §9 format, so ``grep ' :: api '`` isolates the budget log.
logger = logging.getLogger("app.response_budget")
#: Only paths under this prefix are measured. The admin catalogue (and
#: whatever new read route joins it) is the surface TODO L5's "any any
#: other endpoints that could load a ton of data" is about; pages, assets
#: and the static catch-all are not.
API_PATH_PREFIX = "/api/"
#: The media type whose line carries `` stream=1`` (A15 SSE): the frame
#: stream is where ``bytes`` is a total over a life rather than one body.
STREAM_MEDIA_TYPE = "text/event-stream"
#: Status logged when the app died before it could send
#: ``http.response.start`` — the exception is propagating and Starlette's
#: ``ServerErrorMiddleware`` (outermost) answers exactly that.
UNKNOWN_STATUS = 500
class _Tally:
"""What one measured response accumulates: status, the start message,
bytes.
``start`` holds the ``http.response.start`` MESSAGE (the dict), not a
snapshot of its headers, and its headers are read at emit time — after
the final body chunk has been awaited. That is how ``enc`` reports the
wire encoding from a layer sitting UNDER the compressor (the position
that gives an uncompressed ``bytes=``): Starlette's gzip responder holds
that same start message while it decides what to do with the body and
decorates it in place (``MutableHeaders(raw=initial_message["headers"])``
shares the list), so by the time the body completes the
``content-encoding`` the client sees is on it. Holding the message
rather than the list is not pedantry — ``Headers(scope=message)``
REBINDS ``message["headers"]`` to a fresh list (the compression
responder and this stack's ``_RebufferSmallResponses`` both call it
that way), so a captured list would be a stale one.
The dependency is documented and pinned, not assumed:
``tests/integration/test_response_budget_api.py`` reads the real
middleware stack for both ``enc=gzip`` and ``enc=identity``, so a
future layer that copies the message instead of decorating it fails a
test rather than quietly mislabeling the log (the byte count stays
exact either way — it is the number the budget compares).
"""
__slots__ = ("bytes", "logged", "start", "status")
def __init__(self) -> None:
self.bytes = 0
self.status = UNKNOWN_STATUS
self.start: Message | None = None
self.logged = False
def line(self, scope: Scope, started: float) -> str:
"""The §9-style single line — the exact shape README documents.
``duration_ms`` is rounded from the monotonic clock (never
wall-clock: an NTP step must not produce a negative duration).
"""
headers = Headers(raw=list(self.start["headers"]) if self.start else [])
enc = (headers.get("content-encoding") or "identity").strip() or "identity"
media = (headers.get("content-type") or "").partition(";")[0].strip().lower()
duration_ms = int(round((time.monotonic() - started) * 1000))
line = (
f"api method={scope.get('method', '')} path={scope.get('path', '')} "
f"status={self.status} bytes={self.bytes} enc={enc} "
f"duration_ms={duration_ms}"
)
if media == STREAM_MEDIA_TYPE:
line += " stream=1"
return line
def emit(self, scope: Scope, started: float, budget: int) -> None:
"""Write the line ONCE, at the level the budget decides.
Idempotent on purpose: the completion path and the ``finally``
path describe the same response and must never double-log it.
"""
if self.logged:
return
self.logged = True
line = self.line(scope, started)
if self.bytes >= budget:
logger.info(line)
else:
logger.debug(line)
class ResponseBudgetMiddleware:
"""Pure-ASGI response-size logger — zero buffering, SSE-safe.
Wraps ``send`` only (never ``receive``): the timing runs from
``time.monotonic()`` at entry to the ``more_body=False`` completion,
so ``duration_ms`` is the response's (or the stream's) whole life
without the middleware needing to see one inbound message. Every
``http.response.body`` chunk's length adds to the count and nothing is
held, so a stream still flows frame by frame.
``threshold`` overrides the INFO/DEBUG cut for this instance; the
default (``None``) reads ``Settings.log_large_response_bytes`` per
response, so ``app/main.py`` wires it with no arguments and
``BOR_LOG_LARGE_RESPONSE_BYTES`` stays the operator's knob (``0`` = log
every ``/api/*`` response at INFO, negative refuses startup in the
validator). LOCKED A5: this is a log line and a script, never a 413 —
no endpoint starts rejecting large payloads here.
"""
def __init__(self, app: ASGIApp, *, threshold: int | None = None) -> None:
self.app = app
self.threshold = threshold
def _budget(self) -> int:
if self.threshold is not None:
return self.threshold
return get_settings().log_large_response_bytes
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http" or not str(scope.get("path", "")).startswith(
API_PATH_PREFIX
):
# Nothing to measure: not HTTP (websocket/lifespan) or not the
# API surface. Silent pass-through — the original send.
await self.app(scope, receive, send)
return
started = time.monotonic()
tally = _Tally()
async def send_counting(message: Message) -> None:
message_type = message["type"]
if message_type == "http.response.start":
tally.status = int(message["status"])
# The MESSAGE, not a header snapshot — see _Tally: the
# layers above decorate this same dict in place, so reading
# it after the body completes yields the wire encoding.
tally.start = message
elif message_type == "http.response.body":
tally.bytes += len(message.get("body", b""))
if not message.get("more_body", False):
# Await FIRST: the layers above decorate the start
# headers while handling this final chunk, so `enc`
# reads what actually went on the wire.
await send(message)
tally.emit(scope, started, self._budget())
return
await send(message)
try:
await self.app(scope, receive, send_counting)
finally:
# A normal response already emitted at its completion; this
# covers an app that raised (or returned) without finishing a
# body. The exception itself re-raises UNCHANGED.
tally.emit(scope, started, self._budget())
app/main.py
+30
View File
@@ -45,8 +45,10 @@ from app.api.ui_settings import router as ui_settings_router
from app.config import get_settings
from app.core.auth import ensure_admin_configured
from app.core.caching import configure_caching
from app.core.compression import SelectiveGZipMiddleware
from app.core.debugging import configure_debugging
from app.core.logging import configure_logging
from app.core.response_budget import ResponseBudgetMiddleware
from app.core.security_headers import SecurityHeadersMiddleware
configure_logging()
@@ -145,6 +147,34 @@ def create_app() -> FastAPI:
# stream in particular — passes through untouched.
configure_caching(app)
# Phase 167 (task 05, TODO L5): the byte budget — ONE
# ``api method=… path=… status=… bytes=… enc=… duration_ms=…`` line per
# /api/* response (INFO at/over BOR_LOG_LARGE_RESPONSE_BYTES, DEBUG
# below), so "which endpoint is heavy, right now" is a
# ``grep ' :: api '`` away. Position: added BEFORE the gzip call below,
# i.e. INSIDE it (SecurityHeaders → gzip → response-budget → caching →
# session → routes), so ``bytes=`` is the bytes the ROUTE produced
# (uncompressed) while ``enc=`` reports what went on the wire — the two
# together answer "how fat is it, and how fat did the client see it".
# Pure ASGI (never BaseHTTPMiddleware): the SSE stream is counted as it
# passes and logged once, when the stream ends.
app.add_middleware(ResponseBudgetMiddleware)
# Phase 167 (TODO L5): selective gzip — every response except the
# POST /api/chat SSE stream (skipped by EXACT path, A15) compresses
# when it is >= compress_min_bytes (the 2 MB /api/docs/tree defect).
# 0 = the feature is OFF: no middleware is added at all (the
# kill-switch convention). Position: added AFTER configure_caching but
# BEFORE the SecurityHeaders line below, so SecurityHeaders stays
# OUTERMOST (A20, the last add_middleware is outermost) and gzip sits
# outside the caching layer — an already-rewritten HTML page compresses
# too, and /api/* still passes the caching layer byte-identically
# before compression.
if settings.compress_min_bytes > 0:
app.add_middleware(
SelectiveGZipMiddleware, minimum_size=settings.compress_min_bytes
)
# Phase 82: security headers (CSP + no-framing + nosniff) — outermost on
# purpose (last add_middleware): every response carries them, including
# the static catch-all's 404s below (audit SEC-04).
app/schemas.py
+85 -3
View File
@@ -522,6 +522,17 @@ class KbTreeFolder(BaseModel):
A 0-document folder cannot exist (a folder is a catalogue prefix
only), so every existing folder with no stored row is pending —
single-file folders included.
``truncated`` (phase 167, D2/A2): true iff the depth-limited build
STOPPED here — the node's own children were not sent, so the RAG
view drills via ``GET /api/docs/tree/node`` instead of reading
``children``. A truncated node still carries every field the level
row renders: the recursive ``documents`` count, the recursive
``updated_at`` (the phase-106 D9 max — kept correct without the
children), its ``summary``, and ``summary_pending``. Omitted from
the wire when ``False`` (the :func:`_drop_image_fields` omission
precedent) — an untruncated node's wire shape stays byte-identical
to pre-phase-167.
"""
kind: Literal["folder"] = "folder"
@@ -534,6 +545,13 @@ class KbTreeFolder(BaseModel):
summary: str | None = None
summary_pending: bool = False
children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
#: Phase 167 (task 02, D2/A2) — the children were not built at this
#: depth (see the class docstring); OMITTED when ``False``.
truncated: bool = False
@model_serializer(mode="wrap")
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
return _drop_truncated_when_false(self, handler)
class KbTreeSource(BaseModel):
@@ -560,6 +578,14 @@ class KbTreeSource(BaseModel):
A registered 0-document source is never pending (there is nothing
to summarize); a single-document source IS pending while its root
row is absent.
``truncated`` (phase 167, D2/A2) exists on the shape for symmetry
with :class:`KbTreeFolder` (and the task-03 node route), but the
depth budget never marks a SOURCE node truncated at ``depth ≥ 1``:
the top level of the RAG view is source → level 1, so the source's
direct children are always built. Omitted from the wire when
``False`` — the source node's wire shape stays byte-identical to
pre-phase-167.
"""
name: str
@@ -571,14 +597,52 @@ class KbTreeSource(BaseModel):
summary: str | None = None
summary_pending: bool = False
children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
#: Phase 167 (task 02, D2/A2) — never true at ``depth ≥ 1`` (see the
#: class docstring); OMITTED from the wire when ``False``.
truncated: bool = False
@model_serializer(mode="wrap")
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
return _drop_truncated_when_false(self, handler)
class KbTreeTotals(BaseModel):
"""Catalogue-wide totals the RAG view's three stat cards read
(phase 167, task 02, D2/A2).
A depth-limited tree CANNOT be summed client-side — the phase-97
``treeStats()`` walk only sees the sent levels, so a truncated
tree would under-count every card. The endpoint therefore computes
the totals from the FULL catalogue rows it already fetched (no
extra query) and ships them beside the tree.
"""
#: Every indexed document (the whole catalogue, not one level).
documents: int = Field(default=0, ge=0)
#: Sum of the per-document chunk counts (content + ``is_summary``
#: chunks — the same numbers ``GET /api/docs`` reports).
chunks: int = Field(default=0, ge=0)
#: The newest ``indexed_at`` stamp across the catalogue (ISO-8601),
#: ``None`` for an empty KB.
last_indexed: str | None = None
class KbTree(BaseModel):
"""Response of ``GET /api/docs/tree`` (phase 97, task 02).
The FULL recursive tree in ONE fetch — the RAG view (admin) drills
client-side, zero per-level fetches (the ``00_phase.md`` "The tree
endpoint" contract).
Since phase 167 (task 02, LOCKED A2/A3 — supersedes the phase-97
"full recursive tree in one fetch, drill client-side" contract) the
endpoint is DEPTH-LIMITED by default (``depth=1``: the sources plus
ONE level); a node whose children were not built carries
``truncated: true`` and the client drills level by level. An
explicit ``depth`` up to 8 still returns the whole tree, and the
pure builder's ``depth=None`` default keeps every pre-phase
response byte-identical.
``totals`` (phase 167, additive top-level key) is the catalogue's
true documents/chunks/last-indexed — the stat cards stay honest at
every depth because the server, not the (partial) tree walk,
computes them.
Every SOURCE and FOLDER node carries ``summary_pending`` (phase
98, D3): true iff its recursive document count ≥
@@ -592,6 +656,9 @@ class KbTree(BaseModel):
"""
sources: list[KbTreeSource]
#: Phase 167 (task 02): the server-computed catalogue totals beside
#: any depth of tree (see :class:`KbTreeTotals`).
totals: KbTreeTotals = Field(default_factory=KbTreeTotals)
class DocContent(BaseModel):
@@ -1560,6 +1627,21 @@ def _drop_image_fields(model: KbTreeFile, handler: SerializerFunctionWrapHandler
return data
def _drop_truncated_when_false(
model: KbTreeFolder | KbTreeSource, handler: SerializerFunctionWrapHandler
) -> Any:
"""The phase-167 (task 02, D2) truncation-omission rule for the
:class:`KbTreeFolder` / :class:`KbTreeSource` nodes: only a node whose
children were NOT built adds the ``truncated: true`` key — an
untruncated node's wire shape stays byte-identical to pre-phase-167
(every existing wire pin holds), the deliberate echo of the
:func:`_drop_image_fields` omission idiom."""
data = handler(model)
if not model.truncated:
data.pop("truncated", None)
return data
def _drop_absent_share_url(model: BaseModel, handler: SerializerFunctionWrapHandler) -> Any:
"""The ``share_url`` omission rule (phase 51, task 02): ``None`` →
ABSENT from the JSON (not ``"share_url": null``) — an unshared chat