phase: 167_api_payload_diet

All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.

app/config.py

@@ -46,6 +46,35 @@ class Settings(BaseSettings):
     log_level: str = "INFO"
     static_dir: str = "frontend"
 
+    # --- HTTP response compression (phase 167, TODO L5) ---
+    #: Byte floor for the selective gzip layer
+    #: (``app/core/compression.py``, ``BOR_COMPRESS_MIN_BYTES``): responses
+    #: of at least this many bytes are gzipped when the client accepts it
+    #: (Starlette's ``GZipMiddleware`` owns the negotiation, the
+    #: already-encoded skip and the ``Vary`` append). ``0`` = the feature
+    #: is OFF — the middleware is not added at all (the kill switch);
+    #: ``> 0`` = the Starlette minimum-size floor, default 860 — the
+    #: classic nginx default, below which compression can INFLATE a body
+    #: instead of shrinking it. A negative value is a typo — the validator
+    #: fails startup loudly (the ``agent_max_rounds`` pattern). The SSE
+    #: stream (``POST /api/chat``, A15) is never compressed at any value.
+    compress_min_bytes: int = 860
+
+    # --- Response byte budget (phase 167, TODO L5) ---
+    #: INFO/DEBUG cut for the per-response ``api method=… bytes=…`` line
+    #: (``app/core/response_budget.py``, ``BOR_LOG_LARGE_RESPONSE_BYTES``).
+    #: ``/api/*`` responses of at least this many bytes (the bytes the ROUTE
+    #: produced, before compression) log at INFO; below it the line is
+    #: DEBUG, i.e. invisible at the default ``BOR_LOG_LEVEL=INFO`` so the
+    #: log stays readable while "which endpoint is heavy, right now" is
+    #: still one ``grep ' :: api '`` away. Default 65536 (64 KB) — the
+    #: "fine" bar of the phase-167 audit; the audit's 256 KB budget is what
+    #: marks an offender (LOCKED A5: a log line and a script, never a 413).
+    #: ``0`` = log EVERY ``/api/*`` response at INFO (the loud mode, for a
+    #: one-off investigation); a negative value is a typo — the validator
+    #: fails startup loudly (the ``agent_max_rounds`` pattern).
+    log_large_response_bytes: int = 65536
+
     # --- UI customization (phase 62, TODO L3) ---
     # Defaults are the phase-61 neutral copy — UNSET => byte-identical UI.
     # (Phase 91, task 03: the retired CSS-file theme env var is gone —
@@ -768,6 +797,29 @@ class Settings(BaseSettings):
             raise ValueError("chat_max_concurrent must be >= 1")
         return v
 
+    @field_validator("compress_min_bytes")
+    @classmethod
+    def _compress_min_bytes_non_negative(cls, v: int) -> int:
+        """``0`` is the compression kill switch (the middleware is not
+        added at all) — a negative value is a typo (the
+        ``agent_max_rounds`` pattern, phase 167)."""
+        if v < 0:
+            raise ValueError("compress_min_bytes must be >= 0 (0 = compression disabled)")
+        return v
+
+    @field_validator("log_large_response_bytes")
+    @classmethod
+    def _log_large_response_bytes_non_negative(cls, v: int) -> int:
+        """``0`` is the loud mode (every ``/api/*`` response logs its byte
+        line at INFO) — a negative value is a typo that would make the
+        comparison constant, so it fails startup naming the field (the
+        ``agent_max_rounds`` pattern, phase 167)."""
+        if v < 0:
+            raise ValueError(
+                "log_large_response_bytes must be >= 0 (0 = log every /api/* response)"
+            )
+        return v
+
     @field_validator("docs_branch", "docs_base_branch")
     @classmethod
     def _docs_branch_tokens(cls, v: str, info: ValidationInfo) -> str: