phase: 167_api_payload_diet

All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.

README.md

@@ -159,6 +159,30 @@ document's date from the document viewer. The date also feeds the small
 recency boost in retrieval ranking (`BOR_RECENCY_BOOST`). Admin-only —
 anonymous visitors see a sign-in gate instead.
 
+The tree payload is bounded (phase 167): `GET /api/docs/tree` builds only
+`depth` levels (default **1** — the sources plus one level, the page's first
+paint; `depth=2..8` builds deeper, anything else is a 422) instead of the old
+~2 MB full recursive dump. A node whose children were not built carries
+`"truncated": true` and ships with its **recursive** document count, updated
+date, description and pending marker (`children: []`); untruncated nodes are
+byte-for-byte what they were before. The three stat cards read the
+server-computed top-level `totals` (`documents`, `chunks`, `last_indexed`) so
+they stay honest whatever depth was requested. Responses also ship gzipped
+(see *Caching & deploys*). Deeper levels load on demand through
+`GET /api/docs/tree/node?source=X&path=Y&depth=N` (admin-only, phase 167):
+one node — the source root for `path=""` — answered in exactly the shape the
+full tree produces for it (unknown source or folder → 404 `unknown node`),
+built from the same single catalogue query. The RAG view fetches each drilled
+level once and caches it (re-opening a folder costs no network), showing one
+*Loading…* row while a level arrives and a *Couldn't load … — Retry* line on
+failure with the current level staying rendered; a folder that vanished since
+it was listed (the node 404) resets the view to the top level. The
+breadcrumb, the bounded scroll panel, and the description editors behave
+exactly as before. All three catalogue reads
+(`/api/docs`, `/api/docs/tree`, `/api/docs/tree/node`) also carry a weak
+`ETag` + `Cache-Control: private, must-revalidate`, so reloading a page you
+just looked at costs a bodiless 304 (see *Caching & deploys*).
+
 ### Git sources (`/git-sources.html`)
 
 The admin-managed source registry: add or remove git repositories, upload
@@ -942,8 +966,13 @@ uploaded archives):
 ## Caching & deploys
 
 A deploy is a commit — and the browser must see it **without a hard refresh**.
-One Starlette middleware (`app/core/caching.py`) applies the rule at the
-transport layer:
+The first three bullets below are that A19 rule, applied at the transport layer
+by `app/core/caching.py`; the last two are phase 167 — gzip from
+`app/core/compression.py` and per-route validators from `app/core/http_cache.py`.
+Outermost to innermost, the stack is security-headers → gzip (only when
+`BOR_COMPRESS_MIN_BYTES > 0`) → the byte-budget logger → caching → routes, so
+security headers land on every response and `bytes=` in the budget line is the
+route's own, uncompressed payload (see *Operational notes*):
 
 - **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
   `last-modified`** — each visit always gets a fresh 200 body.
@@ -954,6 +983,26 @@ transport layer:
   `HEAD` — so every commit/deploy flips the token and the versioned asset
   URLs change with it. A checkout without `.git` falls back to a stable
   content hash of the `frontend/` tree.
+- **Responses are gzipped (phase 167).** Every HTTP response except the
+  `POST /api/chat` SSE stream goes through selective gzip
+  (`app/core/compression.py`): when the client accepts gzip and the body
+  is at least `BOR_COMPRESS_MIN_BYTES` (default `860`; `0` = the feature
+  is OFF), KB JSON ships ~8-12x smaller — the fix for the ~2 MB
+  `/api/docs/tree` load. The SSE stream is skipped by EXACT path (A15:
+  frames keep arriving one by one, uncompressed), small responses stay
+  identity, and an already-encoded body is never double-encoded, so a
+  fronting nginx may gzip too.
+- **Catalogue reads revalidate (phase 167).** `GET /api/docs`,
+  `GET /api/docs/tree` and `GET /api/docs/tree/node` ship a content-exact
+  weak `ETag` (SHA-256 of the served body, `app/core/http_cache.py`) with
+  `Cache-Control: private, must-revalidate` — a repeat load costs a 304
+  with no body, and a sync or an admin edit (folder summary, document
+  date) changes the body, hence the tag, so it is seen on the next load.
+  The tag is deliberately NOT `sources_meta.version` (bumping that marks
+  saved chats stale, and those edits never bump it). This is a scoped
+  extension of the rule above: validators live on these three read-only
+  routes only — HTML pages keep publishing none, and the SSE stream and
+  every mutating route are untouched.
 
 No CDN, no new services, no build-step change: the middleware rewrites the
 asset references of the known pages in flight.
@@ -962,6 +1011,61 @@ asset references of the known pages in flight.
 > page visit, so the browser revalidates the HTML once and starts requesting
 > the versioned assets; every commit after that is picked up automatically.
 
+### Operational notes: the response byte budget (phase 167)
+
+Every `/api/*` response writes ONE parseable line, so "which endpoint is
+heavy, right now?" is answerable from the app's own log:
+
+```
+2026-10-08 07:40:57 INFO     app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
+```
+
+```bash
+grep ' :: api ' server.log                    # every measured response
+grep ' :: api ' server.log | grep 'bytes=[0-9]\{6,\}'   # the six-figure ones
+```
+
+- **`bytes=`** is what the ROUTE produced (uncompressed), **`enc=`** is what
+  went on the wire (`gzip` or `identity`) — "how fat is it, and how fat did
+  the client see it". A revalidated `304` logs `bytes=0` on purpose: that is
+  the win you want to see.
+- **Level:** `INFO` at or over `BOR_LOG_LARGE_RESPONSE_BYTES` (default
+  `65536`), `DEBUG` below it, so the default log stays readable.
+  `BOR_LOG_LARGE_RESPONSE_BYTES=0` logs EVERY `/api/*` response at INFO —
+  the mode for a one-off investigation. A negative value refuses startup.
+- **`stream=1`** marks the SSE turn (`POST /api/chat`, A15): `bytes` is the
+  streamed total and `duration_ms` the stream's life. One line per stream,
+  written when it ends — including a stream the client cancels (with the
+  bytes that got out). The middleware is pure ASGI and never buffers.
+
+**Measuring the whole API** (the audit behind the fix):
+
+```bash
+# the corpus that matches production's order of magnitude
+#   (destructive to the local DB — see the restore-test-db skill)
+uv run uvicorn app.main:app --port 8000
+uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
+```
+
+It signs in with `BOR_ADMIN_PASSWORD` (or `--password`; the single-admin
+login has no username), measures every candidate read route in BOTH
+`identity` and `gzip` forms plus one conditional revalidation, prints the
+corpus it measured, and marks anything over `--budget` (default `262144`)
+with `LOOK`. It is **read-only** (safe against the deployed app) and always
+exits 0 when it got its numbers — it is a report, not a gate (LOCKED A5:
+nothing in this phase rejects a large payload). Exit `2` means the app was
+unreachable or the login was rejected. The measured before/after tables and
+the verdict per endpoint live in the phase record — `167_api_payload_diet/AUDIT.md`
+under `.agents/phases/`.
+
+> **Deploy note (compression):** the app gzips its own responses, so nginx
+> `gzip on` is **optional** — leaving it off ships the same bytes, and
+> leaving it on is fine too: an already-encoded body is never double-encoded
+> by either side. Passing the client's header through untouched
+> (`proxy_set_header Accept-Encoding $http_accept_encoding;`, or simply not
+> overriding it) is correct; unsetting it would hand every client
+> uncompressed JSON.
+
 ---
 
 ## Configuration reference
@@ -982,6 +1086,8 @@ to `.env` and adjust.
 | `BOR_ENVIRONMENT` | `development` | `development` or `production` (the prod compose profile sets `production`) |
 | `BOR_LOG_LEVEL` | `INFO` | App log level |
 | `BOR_STATIC_DIR` | `frontend` | Where static files are served from (the container image sets `/app/static`) |
+| `BOR_COMPRESS_MIN_BYTES` | `860` | Selective-gzip floor (phase 167): responses ≥ this size compress when the client accepts gzip; `0` = OFF (no middleware), negative refuses startup. `POST /api/chat` (SSE) is never compressed |
+| `BOR_LOG_LARGE_RESPONSE_BYTES` | `65536` | INFO/DEBUG cut for the per-response `api method=… bytes=… enc=…` byte-budget line (phase 167, `grep ' :: api '`); `0` = log every `/api/*` response at INFO, negative refuses startup. See *Operational notes* |
 
 ### Database & concurrency