phase: 167_api_payload_diet
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 14s

All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
2026-10-08 21:23:49 -04:00
parent ab9391d019
commit 2ac37455f0
73 changed files with 7313 additions and 228 deletions
+179
View File
@@ -0,0 +1,179 @@
"""Selective gzip for every HTTP response but the SSE chat stream (phase 167, D1, TODO L5).
The defect (TODO L5): ``GET /api/docs/tree`` loads ~2 MB in production and
the Sources page pays that on every sync and every shell boot. Nothing in
this stack compressed — ``app/main.py`` added only Session / caching /
security-headers, and no response ever carried ``Content-Encoding``. KB
JSON is extremely repetitive (key names, paths, ISO timestamps), so
gzipping it lands at roughly **8-12× smaller** on the wire: the same bytes
the client already knows how to read, ~one order of magnitude faster.
The SSE stream is EXCLUDED (A15, phase 167 LOCKED A1): ``POST /api/chat``
is a ``text/event-stream`` whose whole value is that frames arrive *while
the model is still talking*. Any middleware that buffers the response —
``BaseHTTPMiddleware`` is the classic offender, and the reason this module
is **pure ASGI** — would deliver the entire turn at the end, silently
breaking the typing animation, the tool lines, and the cancel surface. So
the skip is by EXACT path (never a prefix: ``/api/chat-images/…`` keeps
its compression) and the skipped scope is handed to the app untouched.
The compression math is deliberately NOT re-implemented here. Starlette's
:class:`~starlette.middleware.gzip.GZipMiddleware` owns ``Accept-Encoding``
negotiation (a client that does not speak gzip gets identity bytes), the
``minimum_size`` floor, the skip when a response ALREADY sets
``content-encoding``, the ``Vary: Accept-Encoding`` append, and the
streaming ``compressobj`` + final flush. It is also what makes the stack
cooperate with a fronting proxy: **nginx may also gzip, but an
already-encoded body is never double-encoded** — Starlette preserves the
upstream ``Content-Encoding`` verbatim, and so does nginx.
One seam IS patched here, and it is framing, not math: the
``CachingMiddleware`` below us is a ``BaseHTTPMiddleware``, and starlette
1.x re-frames every response it emits upstream (``_StreamingResponse``
ships each body chunk with ``more_body=True`` plus an empty terminator),
so Starlette's buffered small-response exemption can never fire for
anything that travelled through it — a 33-byte ``/api/health`` would
stream into the compressor and come back INFLATED, exactly what the
``minimum_size`` floor exists to prevent. :class:`_RebufferSmallResponses`
sits under the responder and hands it back the single buffered body
message the exemption was written for, bounded to the floor.
"""
from __future__ import annotations
from starlette.datastructures import Headers
from starlette.middleware.gzip import GZipMiddleware
from starlette.types import ASGIApp, Message, Receive, Scope, Send
#: Paths that are NEVER compressed, matched EXACTLY (never by prefix).
#: ``POST /api/chat`` is the SSE stream (A15): frame-by-frame delivery is
#: the entire point of the endpoint, and buffering the response to feed it
#: through a compressor would deliver the whole turn at completion (and a
#: never-ending stream never completes on a cancel). The exact match is
#: load-bearing — ``/api/chat-images/…`` shares the prefix and must keep
#: routing through the compression layer.
DEFAULT_SKIP_PATHS: tuple[str, ...] = ("/api/chat",)
class _RebufferSmallResponses:
"""Hold a declared-small body so Starlette's floor exemption fires.
When ``http.response.start`` carries ``content-length`` below
``minimum_size`` — byte-for-byte the condition Starlette's buffered
path checks — the messages are held (bounded by the floor) and
re-emitted as ONE ``http.response.body`` with ``more_body=False``,
undoing the ``BaseHTTPMiddleware`` re-framing so the responder's
small-response exemption sees what it was written to see and the
bytes stay identity (no ``Content-Encoding``, no ``Vary`` append).
A lying header can only cost the buffer, never correctness: if the
body outgrows the floor despite the header, everything held is
forwarded as it came and the rest streams through the normal
(compressing) path. Non-body messages while holding (e.g.
``http.response.pathsend``) also flush the hold untouched.
"""
def __init__(self, app: ASGIApp, *, minimum_size: int) -> None:
self.app = app
self.minimum_size = minimum_size
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
holding = False
armed = True # the start decision happens at most once per response
held: list[Message] = []
total = 0
async def send_filter(message: Message) -> None:
nonlocal holding, armed, total
message_type = message["type"]
if message_type == "http.response.start" and armed:
armed = False
length = Headers(scope=message).get("content-length")
if (
length is not None
and length.isdigit()
and int(length) < self.minimum_size
):
held.append(message)
holding = True
return
await send(message)
return
if holding and message_type == "http.response.body":
held.append(message)
total += len(message.get("body", b""))
if not message.get("more_body", False):
# Whole body in hand, still under the floor: re-emit
# it as the single buffered message the exemption wants.
start = held[0]
merged = b"".join(m.get("body", b"") for m in held[1:])
holding = False
held.clear()
await send(start)
await send({"type": "http.response.body", "body": merged, "more_body": False})
elif total >= self.minimum_size:
# The content-length lied — stop holding, stream on.
holding = False
for held_message in held:
await send(held_message)
held.clear()
return
if holding:
# pathsend or anything unexpected mid-hold: flush as-is.
holding = False
for held_message in held:
await send(held_message)
held.clear()
await send(message)
await self.app(scope, receive, send_filter)
class SelectiveGZipMiddleware:
"""Pure-ASGI selective gzip (phase 167, D1 — no ``BaseHTTPMiddleware``).
Holds the wrapped ``app`` and a composed
:class:`~starlette.middleware.gzip.GZipMiddleware` built over the SAME
app (via the :class:`_RebufferSmallResponses` seam, so the floor still
works under the caching layer's re-framing); ``__call__`` delegates to
the gzip path unless the scope is not HTTP (websocket / lifespan pass
straight through) or the request path is one of ``skip_paths`` — in
which case the app is called DIRECTLY, with the original
``receive``/``send``, so a streaming response reaches the client frame
by frame, uncompressed.
``minimum_size`` is the Starlette floor (bytes): responses smaller than
it are forwarded without compression because gzip can make them
bigger. It is supplied by the caller (``Settings.compress_min_bytes``;
``0`` removes this middleware entirely at wiring time — see
``app/main.py``). The SSE stream (``POST /api/chat``, A15) is never
compressed at any value.
"""
def __init__(
self,
app: ASGIApp,
*,
minimum_size: int,
skip_paths: tuple[str, ...] = DEFAULT_SKIP_PATHS,
) -> None:
self.app = app
self.minimum_size = minimum_size
self.skip_paths = tuple(skip_paths)
# The compression itself stays Starlette's: negotiation, floor,
# already-encoded skip, Vary append, streaming compressobj + flush.
self.gzip = GZipMiddleware(
_RebufferSmallResponses(app, minimum_size=minimum_size),
minimum_size=minimum_size,
)
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http" or scope["path"] in self.skip_paths:
# Non-HTTP scopes have no response to compress; the SSE stream
# must never see this layer (A15 — see DEFAULT_SKIP_PATHS).
await self.app(scope, receive, send)
return
await self.gzip(scope, receive, send)
+112
View File
@@ -0,0 +1,112 @@
"""HTTP revalidation validators for the read-only catalogue reads
(phase 167, task 03, D3 — LOCKED A4).
Three pure functions, no I/O, ONE place for the pattern (the routes
compose them; the alternative was a copy per route):
* :func:`weak_etag` — the validator: a weak ``ETag`` over the EXACT
bytes being served,
* :func:`etag_matches` — the tolerant ``If-None-Match`` evaluator,
* :func:`revalidate_headers` — the header pair a validated response
must carry.
**Why a content hash, not ``sources_meta.version``** (LOCKED A4):
bumping that generation marks every saved chat ``stale`` (phase 53) —
the wrong tool for a payload revalidation — and the admin PATCHes
(``/api/documents/summary``, ``/api/documents/date``,
``/api/folders/summary``) mutate the tree WITHOUT bumping it, so a
version-derived tag would serve stale trees. A hash of the serialized
body is content-exact by construction: the tag changes ⟺ the body
changes, it needs no migration, and the tree is built anyway (the hash
costs microseconds next to the query). The tag is over the EXACT served
bytes — the endpoint serializes once and hands those bytes to both the
hash and the ``Response``.
**Why a truncation to 32 hex chars is safe**: 32 hex chars of SHA-256
is this file's truncation convention. Collisions are not a correctness
risk for a revalidated cache: a false match only ever costs one stale
repaint on a ~2 MB namespace (the next unconditional load corrects it),
while 128 bits of digest make even that astronomically unlikely.
A19 SCOPE (:func:`revalidate_headers`, LOCKED A4 as a recorded, scoped
extension of A19): HTML pages stay ``no-cache`` with NO validators (the
caching middleware strips inbound conditionals and outbound validators —
the phase-54 contract), ``/assets/*`` stay ``immutable`` for a year
(URL-versioned), and this module governs the READ-ONLY API arm only —
today the two KB-tree routes and ``GET /api/docs``. The SSE stream and
every mutating route are untouched.
"""
from __future__ import annotations
import hashlib
#: Hex chars of the SHA-256 digest the tag carries (see the module
#: docstring's collision argument).
_TAG_HEX_CHARS = 32
#: The ``Cache-Control`` paired with every validator here: ``private``
#: (no shared cache may store the admin-gated catalogue) +
#: ``must-revalidate`` (a sync or an admin edit is seen on the NEXT
#: load — the client may never serve these bytes without asking, which
#: is what distinguishes this from the ``/assets/*`` posture).
REVALIDATE_CACHE_CONTROL = "private, must-revalidate"
def weak_etag(body: str | bytes) -> str:
"""Weak validator over the exact served bytes: ``W/"<sha256 hex[:32]>"``.
Content-exact by construction (the endpoint hashes the SAME string
it returns — see the module docstring for the A4 argument and the
truncation convention). ``str`` bodies are hashed as UTF-8, so the
str and its ``bytes`` form produce the SAME tag. Weak (``W/``)
because the comparison is byte-equivalence under revalidation, not
bit-identity — a differently gzipped encoding of the same content
must still match (RFC 9110 §8.8.3).
"""
raw = body.encode("utf-8") if isinstance(body, str) else body
return f'W/"{hashlib.sha256(raw).hexdigest()[:_TAG_HEX_CHARS]}"'
def _strip_weakness(candidate: str) -> str:
"""One comparable form: trimmed, with the ``W/`` weakness marker gone
(weak/strong equivalence — ``W/"x"`` matches ``"x"`` in BOTH
directions, RFC 9110 §8.8.3.2's weak comparison rule applied in both
directions on purpose: our tag is over the exact bytes, so a client
that stored either form HAS these bytes)."""
value = candidate.strip()
if value[:2].upper() == "W/":
value = value[2:].strip()
return value
def etag_matches(header: str | None, tag: str) -> bool:
"""Does an inbound ``If-None-Match`` header value match our ``tag``?
Tolerant the way real clients send it: ``*`` matches (RFC 9110
§13.3.3 — the client holds nothing it trusts), a comma-separated
candidate list matches on ANY entry, weak/strong forms are
equivalent in both directions, and surrounding whitespace is
ignored. ``None`` or an empty/whitespace-only header never matches
(a request with no conditional header is an unconditional GET → 200).
"""
if not header or not header.strip():
return False
candidates = header.split(",")
if any(candidate.strip() == "*" for candidate in candidates):
return True
target = _strip_weakness(tag)
return any(_strip_weakness(candidate) == target for candidate in candidates)
def revalidate_headers(etag: str) -> dict[str, str]:
"""The ONE header pair every validated catalogue response carries —
on the 200 AND on the 304 (RFC 9110 §8.8.3: a 304 must repeat the
validator, or the client cannot extend the stored entry's life).
A19 scope (module docstring): HTML pages stay ``no-cache`` with no
validators, ``/assets/*`` stay ``immutable``; this pair is the
read-only API arm — ``private, must-revalidate`` so a sync or an
admin edit is seen on the next load, because the tag changes exactly
when the body changes.
"""
return {"ETag": etag, "Cache-Control": REVALIDATE_CACHE_CONTROL}
+211
View File
@@ -0,0 +1,211 @@
"""The byte budget: ONE log line per ``/api/*`` response (phase 167, task 05, D5, LOCKED A5).
TODO L5 asked "which endpoint is heavy, right now, in production?" — and
before this phase the app could not answer it. The 2 MB
``GET /api/docs/tree`` defect was found by a human eyeballing a browser
network tab; every other candidate endpoint was a guess. This module makes
the answer a ``grep`` away:
.. code-block:: text
api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41
* ``bytes`` is the size of the body the ROUTE produced — the sum of the
``http.response.body`` chunk lengths as they pass through, uncompressed
even when the wire carried gzip (the compression layer above us encodes
AFTER we counted). ``enc`` reports what actually went on the wire
(``content-encoding``, else ``identity``). The two together answer "how
fat is it, and how fat did the client see it".
* level: ``INFO`` when ``bytes >= settings.log_large_response_bytes``
(``BOR_LOG_LARGE_RESPONSE_BYTES``, default 65536; ``0`` = every ``/api/*``
response at INFO), ``DEBUG`` below it — so the default log stays quiet
and an offender is loud without a re-deploy.
* a ``304`` logs ``bytes=0`` deliberately: the revalidation win this phase
bought is visible in the log, not hidden behind a missing line.
* `` stream=1`` is appended for ``text/event-stream`` responses, where
``bytes`` is the streamed total and ``duration_ms`` is the stream's LIFE.
Why **pure ASGI** (the phase-167 rule for anything on this path): the
middleware must never buffer. It only reads message lengths, so the SSE
chat stream (A15) passes through byte-identical — a
``BaseHTTPMiddleware`` here would deliver the whole turn at the end,
which is exactly the failure mode phase 167 exists to avoid elsewhere.
Logged on the ``http.response.body`` message with ``more_body=False`` —
for a stream that is when the **stream ends**, so ``duration_ms`` is the
stream's whole life. A stream the client walks away from (a cancelled SSE
turn) writes exactly one line too, with the bytes that actually got out:
the app unwinds without a completing body message and the ``finally`` path
emits it. Half a response happened, so half a count is the honest number —
and no line at all would make a chatty, abandoned poll look free.
An app exception is handled the same way (one line, bytes as counted, status
500 when the app never got to answer) and re-raises UNCHANGED; non-HTTP
scopes and non-``/api/`` paths pass through silently (pages, assets and the
static catch-all are not the catalogue surface this budget governs).
"""
from __future__ import annotations
import logging
import time
from starlette.datastructures import Headers
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from app.config import get_settings
#: The logger behind every line — renders as ``… :: api method=…`` with the
#: §9 format, so ``grep ' :: api '`` isolates the budget log.
logger = logging.getLogger("app.response_budget")
#: Only paths under this prefix are measured. The admin catalogue (and
#: whatever new read route joins it) is the surface TODO L5's "any any
#: other endpoints that could load a ton of data" is about; pages, assets
#: and the static catch-all are not.
API_PATH_PREFIX = "/api/"
#: The media type whose line carries `` stream=1`` (A15 SSE): the frame
#: stream is where ``bytes`` is a total over a life rather than one body.
STREAM_MEDIA_TYPE = "text/event-stream"
#: Status logged when the app died before it could send
#: ``http.response.start`` — the exception is propagating and Starlette's
#: ``ServerErrorMiddleware`` (outermost) answers exactly that.
UNKNOWN_STATUS = 500
class _Tally:
"""What one measured response accumulates: status, the start message,
bytes.
``start`` holds the ``http.response.start`` MESSAGE (the dict), not a
snapshot of its headers, and its headers are read at emit time — after
the final body chunk has been awaited. That is how ``enc`` reports the
wire encoding from a layer sitting UNDER the compressor (the position
that gives an uncompressed ``bytes=``): Starlette's gzip responder holds
that same start message while it decides what to do with the body and
decorates it in place (``MutableHeaders(raw=initial_message["headers"])``
shares the list), so by the time the body completes the
``content-encoding`` the client sees is on it. Holding the message
rather than the list is not pedantry — ``Headers(scope=message)``
REBINDS ``message["headers"]`` to a fresh list (the compression
responder and this stack's ``_RebufferSmallResponses`` both call it
that way), so a captured list would be a stale one.
The dependency is documented and pinned, not assumed:
``tests/integration/test_response_budget_api.py`` reads the real
middleware stack for both ``enc=gzip`` and ``enc=identity``, so a
future layer that copies the message instead of decorating it fails a
test rather than quietly mislabeling the log (the byte count stays
exact either way — it is the number the budget compares).
"""
__slots__ = ("bytes", "logged", "start", "status")
def __init__(self) -> None:
self.bytes = 0
self.status = UNKNOWN_STATUS
self.start: Message | None = None
self.logged = False
def line(self, scope: Scope, started: float) -> str:
"""The §9-style single line — the exact shape README documents.
``duration_ms`` is rounded from the monotonic clock (never
wall-clock: an NTP step must not produce a negative duration).
"""
headers = Headers(raw=list(self.start["headers"]) if self.start else [])
enc = (headers.get("content-encoding") or "identity").strip() or "identity"
media = (headers.get("content-type") or "").partition(";")[0].strip().lower()
duration_ms = int(round((time.monotonic() - started) * 1000))
line = (
f"api method={scope.get('method', '')} path={scope.get('path', '')} "
f"status={self.status} bytes={self.bytes} enc={enc} "
f"duration_ms={duration_ms}"
)
if media == STREAM_MEDIA_TYPE:
line += " stream=1"
return line
def emit(self, scope: Scope, started: float, budget: int) -> None:
"""Write the line ONCE, at the level the budget decides.
Idempotent on purpose: the completion path and the ``finally``
path describe the same response and must never double-log it.
"""
if self.logged:
return
self.logged = True
line = self.line(scope, started)
if self.bytes >= budget:
logger.info(line)
else:
logger.debug(line)
class ResponseBudgetMiddleware:
"""Pure-ASGI response-size logger — zero buffering, SSE-safe.
Wraps ``send`` only (never ``receive``): the timing runs from
``time.monotonic()`` at entry to the ``more_body=False`` completion,
so ``duration_ms`` is the response's (or the stream's) whole life
without the middleware needing to see one inbound message. Every
``http.response.body`` chunk's length adds to the count and nothing is
held, so a stream still flows frame by frame.
``threshold`` overrides the INFO/DEBUG cut for this instance; the
default (``None``) reads ``Settings.log_large_response_bytes`` per
response, so ``app/main.py`` wires it with no arguments and
``BOR_LOG_LARGE_RESPONSE_BYTES`` stays the operator's knob (``0`` = log
every ``/api/*`` response at INFO, negative refuses startup in the
validator). LOCKED A5: this is a log line and a script, never a 413 —
no endpoint starts rejecting large payloads here.
"""
def __init__(self, app: ASGIApp, *, threshold: int | None = None) -> None:
self.app = app
self.threshold = threshold
def _budget(self) -> int:
if self.threshold is not None:
return self.threshold
return get_settings().log_large_response_bytes
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http" or not str(scope.get("path", "")).startswith(
API_PATH_PREFIX
):
# Nothing to measure: not HTTP (websocket/lifespan) or not the
# API surface. Silent pass-through — the original send.
await self.app(scope, receive, send)
return
started = time.monotonic()
tally = _Tally()
async def send_counting(message: Message) -> None:
message_type = message["type"]
if message_type == "http.response.start":
tally.status = int(message["status"])
# The MESSAGE, not a header snapshot — see _Tally: the
# layers above decorate this same dict in place, so reading
# it after the body completes yields the wire encoding.
tally.start = message
elif message_type == "http.response.body":
tally.bytes += len(message.get("body", b""))
if not message.get("more_body", False):
# Await FIRST: the layers above decorate the start
# headers while handling this final chunk, so `enc`
# reads what actually went on the wire.
await send(message)
tally.emit(scope, started, self._budget())
return
await send(message)
try:
await self.app(scope, receive, send_counting)
finally:
# A normal response already emitted at its completion; this
# covers an app that raised (or returned) without finishing a
# body. The exception itself re-raises UNCHANGED.
tally.emit(scope, started, self._budget())