All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
113 lines
5.2 KiB
Python
113 lines
5.2 KiB
Python
"""HTTP revalidation validators for the read-only catalogue reads
|
|
(phase 167, task 03, D3 — LOCKED A4).
|
|
|
|
Three pure functions, no I/O, ONE place for the pattern (the routes
|
|
compose them; the alternative was a copy per route):
|
|
|
|
* :func:`weak_etag` — the validator: a weak ``ETag`` over the EXACT
|
|
bytes being served,
|
|
* :func:`etag_matches` — the tolerant ``If-None-Match`` evaluator,
|
|
* :func:`revalidate_headers` — the header pair a validated response
|
|
must carry.
|
|
|
|
**Why a content hash, not ``sources_meta.version``** (LOCKED A4):
|
|
bumping that generation marks every saved chat ``stale`` (phase 53) —
|
|
the wrong tool for a payload revalidation — and the admin PATCHes
|
|
(``/api/documents/summary``, ``/api/documents/date``,
|
|
``/api/folders/summary``) mutate the tree WITHOUT bumping it, so a
|
|
version-derived tag would serve stale trees. A hash of the serialized
|
|
body is content-exact by construction: the tag changes ⟺ the body
|
|
changes, it needs no migration, and the tree is built anyway (the hash
|
|
costs microseconds next to the query). The tag is over the EXACT served
|
|
bytes — the endpoint serializes once and hands those bytes to both the
|
|
hash and the ``Response``.
|
|
|
|
**Why a truncation to 32 hex chars is safe**: 32 hex chars of SHA-256
|
|
is this file's truncation convention. Collisions are not a correctness
|
|
risk for a revalidated cache: a false match only ever costs one stale
|
|
repaint on a ~2 MB namespace (the next unconditional load corrects it),
|
|
while 128 bits of digest make even that astronomically unlikely.
|
|
|
|
A19 SCOPE (:func:`revalidate_headers`, LOCKED A4 as a recorded, scoped
|
|
extension of A19): HTML pages stay ``no-cache`` with NO validators (the
|
|
caching middleware strips inbound conditionals and outbound validators —
|
|
the phase-54 contract), ``/assets/*`` stay ``immutable`` for a year
|
|
(URL-versioned), and this module governs the READ-ONLY API arm only —
|
|
today the two KB-tree routes and ``GET /api/docs``. The SSE stream and
|
|
every mutating route are untouched.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
|
|
#: Hex chars of the SHA-256 digest the tag carries (see the module
|
|
#: docstring's collision argument).
|
|
_TAG_HEX_CHARS = 32
|
|
|
|
#: The ``Cache-Control`` paired with every validator here: ``private``
|
|
#: (no shared cache may store the admin-gated catalogue) +
|
|
#: ``must-revalidate`` (a sync or an admin edit is seen on the NEXT
|
|
#: load — the client may never serve these bytes without asking, which
|
|
#: is what distinguishes this from the ``/assets/*`` posture).
|
|
REVALIDATE_CACHE_CONTROL = "private, must-revalidate"
|
|
|
|
|
|
def weak_etag(body: str | bytes) -> str:
|
|
"""Weak validator over the exact served bytes: ``W/"<sha256 hex[:32]>"``.
|
|
|
|
Content-exact by construction (the endpoint hashes the SAME string
|
|
it returns — see the module docstring for the A4 argument and the
|
|
truncation convention). ``str`` bodies are hashed as UTF-8, so the
|
|
str and its ``bytes`` form produce the SAME tag. Weak (``W/``)
|
|
because the comparison is byte-equivalence under revalidation, not
|
|
bit-identity — a differently gzipped encoding of the same content
|
|
must still match (RFC 9110 §8.8.3).
|
|
"""
|
|
raw = body.encode("utf-8") if isinstance(body, str) else body
|
|
return f'W/"{hashlib.sha256(raw).hexdigest()[:_TAG_HEX_CHARS]}"'
|
|
|
|
|
|
def _strip_weakness(candidate: str) -> str:
|
|
"""One comparable form: trimmed, with the ``W/`` weakness marker gone
|
|
(weak/strong equivalence — ``W/"x"`` matches ``"x"`` in BOTH
|
|
directions, RFC 9110 §8.8.3.2's weak comparison rule applied in both
|
|
directions on purpose: our tag is over the exact bytes, so a client
|
|
that stored either form HAS these bytes)."""
|
|
value = candidate.strip()
|
|
if value[:2].upper() == "W/":
|
|
value = value[2:].strip()
|
|
return value
|
|
|
|
|
|
def etag_matches(header: str | None, tag: str) -> bool:
|
|
"""Does an inbound ``If-None-Match`` header value match our ``tag``?
|
|
|
|
Tolerant the way real clients send it: ``*`` matches (RFC 9110
|
|
§13.3.3 — the client holds nothing it trusts), a comma-separated
|
|
candidate list matches on ANY entry, weak/strong forms are
|
|
equivalent in both directions, and surrounding whitespace is
|
|
ignored. ``None`` or an empty/whitespace-only header never matches
|
|
(a request with no conditional header is an unconditional GET → 200).
|
|
"""
|
|
if not header or not header.strip():
|
|
return False
|
|
candidates = header.split(",")
|
|
if any(candidate.strip() == "*" for candidate in candidates):
|
|
return True
|
|
target = _strip_weakness(tag)
|
|
return any(_strip_weakness(candidate) == target for candidate in candidates)
|
|
|
|
|
|
def revalidate_headers(etag: str) -> dict[str, str]:
|
|
"""The ONE header pair every validated catalogue response carries —
|
|
on the 200 AND on the 304 (RFC 9110 §8.8.3: a 304 must repeat the
|
|
validator, or the client cannot extend the stored entry's life).
|
|
|
|
A19 scope (module docstring): HTML pages stay ``no-cache`` with no
|
|
validators, ``/assets/*`` stay ``immutable``; this pair is the
|
|
read-only API arm — ``private, must-revalidate`` so a sync or an
|
|
admin edit is seen on the next load, because the tag changes exactly
|
|
when the body changes.
|
|
"""
|
|
return {"ETag": etag, "Cache-Control": REVALIDATE_CACHE_CONTROL}
|