# Phase 167 — API payload diet: gzip, a lazily-drilled KB tree, revalidation, and a measured byte budget

**Source:** `TODO.md` L5 — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data."
**Story:** n/a (owner-directed performance defect)
**Context:** `app/api/docs.py` (`GET /api/docs/tree` L919, the pure builder `build_kb_tree` L781 / `_level_children` L671 / `_folder_counts` L626 / `_subtree_max` L746, `GET /api/docs` L129), `app/schemas.py` (`KbTree` L576, `KbTreeFolder` L501, `KbTreeSource` L539, `KbTreeFile` L447 + the `_drop_image_fields` omission precedent), `app/main.py` (middleware stack: `SessionMiddleware` → routers → `configure_caching` → `SecurityHeadersMiddleware` outermost), `app/core/caching.py` (the `/api/*` pass-through contract), `app/core/logging.py` (§9 log style), `frontend/assets/sources.js` (`loadTree` L1625, `renderLevel` L1888, `treeStats` L1703, `folderExistsIn` L1667, `resetVanishedLocation` L1646), `app/models.py` (`SourcesMeta` — deliberately NOT the ETag source, see A3), phases 94/97/98/99/106/122 (the tree contract), A15 (SSE), A19 (caching), A20 (security headers).

## Objective
`GET /api/docs/tree` stops shipping 2 MB: the first load carries what the RAG view actually renders (sources + one level, with server-computed totals), deeper levels load on drill, every response the client already has revalidates to a 304, and **every** non-SSE response ships gzipped (~8-12× on this JSON). Then the same lens is pointed at the rest of the API with a byte-budget log line and a measured audit, so "any other endpoints that could load a ton of data" becomes a recorded finding with fixes for the offenders instead of a guess.

## Why it is 2 MB (traced 2026-10-08)
`list_kb_tree` returns the FULL recursive tree in one body (the deliberate phase-97 "the RAG view drills CLIENT-side, zero per-level requests" contract): every indexed document as a file node (`path`, `title`, `chunks`, `created_at`, `indexed_at`, + `is_image`/`image_url`/`summary` on image nodes) AND **every stored `folder_summaries` row inlined on its folder/source node**, with no compression anywhere in the stack (the app has no GZip middleware — `app/main.py` adds only Session / caching / security-headers; nothing sets `Content-Encoding`). At the live corpus (≈1 k documents, hundreds of folders with AI descriptions, image docs whose `summary` is a full vision description) that is ~2 MB of uncompressed JSON per Sources-page load, and the browser re-downloads it after every sync and on every shell boot.

## Design (shared by all tasks — the executor reads this, not the chat)

**D1 — selective gzip, SSE never touched (task 01, LOCKED A1).** New `app/core/compression.py`, a **pure-ASGI** middleware (`BaseHTTPMiddleware` is forbidden here — it buffers):
```python
class SelectiveGZipMiddleware:
    def __init__(self, app, *, minimum_size: int, skip_paths: tuple[str, ...] = ("/api/chat",)) -> None:
        self.app = app
        self.gzip = GZipMiddleware(app, minimum_size=minimum_size)  # starlette, reused, not forked
    async def __call__(self, scope, receive, send):
        if scope["type"] != "http" or scope["path"] in self.skip_paths:
            await self.app(scope, receive, send); return
        await self.gzip(scope, receive, send)
```
Exact-path skip (never a prefix — `/api/chat-images` must still compress). Starlette's `GZipMiddleware` already: requires `Accept-Encoding: gzip`, respects `minimum_size`, skips a response that already sets `content-encoding`, appends `Vary: Accept-Encoding`, and streams with `compressobj` + a final flush. `app/main.py`: new `Settings.compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`, `0` = the feature OFF (skip the middleware entirely), negative → fail startup loudly — the `agent_max_rounds` pattern, PLAN §13), wired by `app.add_middleware(...)` **before** the `SecurityHeadersMiddleware` line so SecurityHeaders stays outermost (A20) and gzip sits outside the caching layer (an already-rewritten HTML page compresses; `/api/*` still passes the caching middleware byte-identically). No production nginx change is required; if nginx also gzips, it will not double-encode an already-encoded body.

**D2 — a depth-limited tree + server totals (task 02, LOCKED A2).** `app/api/docs.py`:
- `build_kb_tree(..., depth: int | None = None)` and `_level_children(..., depth)` — `None` = unlimited (the legacy behavior every existing unit test keeps, byte-identical); a positive budget stops the recursion. A folder node whose own children were NOT built carries `truncated=True`, `children=[]`, its recursive `documents` from `counts[sub]` (unchanged) and — the correctness trap — its `updated_at` from a NEW recursive max-date map computed in `_folder_counts` (because `_subtree_max(children)` over an empty child list would be `None`, which would silently break the phase-106 D9 column). Its `summary` still rides the node (the level row shows a subfolder's description).
- `GET /api/docs/tree` gains `depth: int = Query(1, ge=1, le=8)`.
- `KbTree` gains **`totals: KbTreeTotals`** (`documents`, `chunks`, `last_indexed: str | None`) computed in the endpoint from the SAME catalogue rows (no extra query) — `sources.js::treeStats()` cannot sum a truncated tree client-side (it walks children) and the three stat cards must stay honest.
- `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `@model_serializer(mode="wrap")` omission when `False` (the `_drop_image_fields` precedent): an untruncated node's wire shape stays byte-identical, and only the depth-limited nodes add the key.

**D3 — the subtree route + revalidation (task 03, LOCKED A3/A4).**
- `GET /api/docs/tree/node?source=<name>&path=<folder>&depth=1` (admin-gated like `/api/docs/tree`, `path=""` = the source root): one node in the `KbTreeFolder`/`KbTreeSource` shape (its direct children + `truncated` flags), 404 `{"detail": "unknown node"}` for an unregistered+unindexed source or a folder that fails the phase-94 existence rule. The RAG view's drill target; the catalogue query is the same one `GET /api/docs/tree` runs (documented: one query per drill, no N+1).
- Revalidation, both tree routes (and `GET /api/docs`): build → serialize once (`model_dump_json()` / json) → `ETag: W/"<sha256(body)[:32]>"` + `Cache-Control: private, must-revalidate`; `If-None-Match` match (weak/strong and `*` tolerant) → `Response(304)` with the same two headers. Helpers live in a new `app/core/http_cache.py` (`weak_etag(body)`, `etag_matches(header, tag)`) so the pattern is one function, not a copy. **Why not `sources_meta.version`** (LOCKED A3): bumping that generation marks every saved chat `stale` (phase 53) — the wrong tool — and the admin PATCHes (`/api/documents/summary`, `/api/documents/date`, `/api/folders/summary`) change the tree WITHOUT bumping it, so a version-derived tag would serve stale trees; a content hash is exact, needs no migration, and the tree is built anyway. This is a deliberate, recorded extension of A19 (which governs HTML pages + `/assets/*` + the caching pass-through) — `/api/*` gains a validator on two catalogue routes only; the SSE stream and every mutating route are untouched.

**D4 — the client drills lazily (task 04).** `frontend/assets/sources.js`: `loadTree()` → `fetch("/api/docs/tree?depth=1")` (still ONE first load), a `levelCache` (Map keyed `source + "/" + folder`), and a drill that fetches `GET /api/docs/tree/node?…` **only** when the target node reports `truncated === true` and is not cached — merged into the cached node before `renderLevel()`. Never-stale (§7.4): the drill's in-flight state disables the row's link + shows one "Loading…" row (the `.kb-*` empty/loading idiom), a failure shows a `role="alert"` line with a Retry affordance and leaves the current level rendered, and the race token (`loadSeq`) extends to the node fetch (a stale drill response may never paint). `folderExistsIn`/`resetVanishedLocation` stop walking a full tree: the node route's 404 IS the "the folder vanished" signal → reset to the top level. `treeStats()` reads `kbTree.totals` (fallback to the existing walk when `totals` is absent — the defensive no-crash path).

**D5 — the byte budget and the audit (task 05).** New `app/core/response_budget.py`, pure ASGI, one log line per `/api/*` response in the §9 single-line style:
`api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41`
It counts `http.response.body` chunk bytes as they pass (zero buffering, SSE-safe: `duration_ms` for a stream is the stream's life, documented), logs at INFO when `bytes >= settings.log_large_response_bytes` (`BOR_LOG_LARGE_RESPONSE_BYTES`, default 65536, `0` = log every `/api/*` response, negative fails startup) and at DEBUG otherwise. `GET /api/chat` gets a `stream=1` variant of the line. New `scripts/api_payload_audit.py` (`python -m scripts.api_payload_audit --base-url … --user … --password …`): logs in, hits the candidate endpoints (`/api/docs/tree` at depth 1 and unlimited, `/api/docs`, `/api/documents/content` for the largest doc, `/api/chats`, `/api/chats/{id}` for the largest chat, `/api/git-sources`, `/api/suggestions`, `/api/sync/status`, `/api/tokens`, `/api/config`), and prints uncompressed AND gzip bytes + duration per endpoint with a stated budget (256 KB uncompressed = "look at it", 64 KB = "fine"). Findings land in this directory as `AUDIT.md` (endpoint, before/after bytes, the remedy applied or the accepted-why) — the TODO's "any other endpoints" answered with numbers, not vibes. The audit runs against the local 1,000-doc live-replica snapshot (the `restore-test-db` skill) so the numbers are the production order of magnitude, not the fixture KB's.

**D6 — verification (task 06).** New `tests/e2e/test_kb_tree_lazy_load.py`: the Sources page's FIRST `/api/docs/tree` request carries `depth=1` + `totals` and no deep children (route-level byte assertion + `via response.json()` node walk), drilling two levels deep issues the node fetch (exactly once per level, cached the second time), the stat cards match `totals` at every level, a 404 node (folder pruned under us — a `page.route` stub) resets to the top level with no stale breadcrumb, `Accept-Encoding: gzip` yields `Content-Encoding: gzip` + a `Vary` header, and the second visit revalidates to a 304 (`page.route` observes the 304 + the body unchanged from cache). Plus the migrated pins: `tests/unit/test_kb_tree_ui.py` (asserts exactly ONE `fetch("/api/docs/tree")` occurrence), `tests/unit/test_frontend_router.py:591`, `tests/e2e/test_navbar_refresh.py:115` (the `#nav-sources` → `/api/docs/tree` expectation), and the phase-97/99 integration shape tests.

## Dependencies
- `97_kb_tree_catalog` + `99_kb_tree_table_and_back_nav` (complete) — the one-fetch/drill-client-side contract this phase **deliberately supersedes** (owner-confirmed 2026-10-07, TODO L5); `106_document_dates` (D9: a truncated node's `updated_at` must stay the recursive max — D2's trap), `98_sync_summary_visibility` (D3 `summary_pending` on every node, including truncated ones), `122_image_documents` (task 04's image map, which stays a bounded select), `156_table_ux_polish` (the bounded scroll panel + sticky header the drill re-renders into), `82_security_headers` / A19 / A20 (the middleware order this phase must not disturb), A15 (SSE).
- No migration: nothing in the DB changes.

## Tasks
1. `01_gzip_middleware.md` — `app/core/compression.py` + `Settings.compress_min_bytes` + wiring + the compression/SSE/passthrough tests.
2. `02_tree_depth_and_totals.md` — `KbTreeTotals`, the `truncated` omission serializer, `build_kb_tree(depth=…)`, the recursive max-date map, `depth` on `/api/docs/tree`, builder + integration tests.
3. `03_tree_node_endpoint_etag.md` — `app/core/http_cache.py`, `GET /api/docs/tree/node`, ETag/304 on the two tree routes + `/api/docs`, integration matrix.
4. `04_client_lazy_drill.md` — `sources.js` level cache + per-level fetch + §7.4 drill states + `totals` stat cards + the migrated unit pins.
5. `05_response_budget_audit.md` — `app/core/response_budget.py` + `scripts/api_payload_audit.py` + `AUDIT.md` with the measured numbers + the remedies for the offenders (and the accepted-why for the rest) + README/env docs.
6. `06_e2e_payload_diet.md` — `tests/e2e/test_kb_tree_lazy_load.py`, the migrated E2E pins, the full gate set, the commit.

## Testing & Quality
- Unit: the builder depth/truncation/max-date matrix (`tests/unit/test_kb_tree_builder.py`), the schema omission wire-shape pins, `tests/unit/test_caching.py` extended (the `/api/*` pass-through survives gzip coexistence), config validator pins, `tests/unit/test_kb_tree_ui.py` extended for the drill (cache, single fetch per level, race token, loading/alert states, `totals` stat cards).
- Integration: depth + truncated + totals + node-route matrix (200/404/403), the ETag matrix (no header → 200 + `ETag`; match → 304 no body; mismatch → 200; `*`; weak/strong equivalence; a folder-summary PATCH changes the tag), gzip (`Content-Encoding: gzip`, `Vary`, under-`minimum_size` identity, `Accept-Encoding: identity` identity, the SSE stream NOT compressed and arriving frame-by-frame), and the response-budget log line (caplog: INFO over the threshold, DEBUG under, `stream=1` for `/api/chat`).
- E2E: `tests/e2e/test_kb_tree_lazy_load.py` green **in isolation**; the owning regression set green in isolation: `test_kb_tree.py` (migrated), `test_kb_tree_nav.py`, `test_navbar_refresh.py`, `test_document_dates.py`, `test_sync_summary_visibility.py`, `test_table_ux_polish.py`, `test_sources_midstream_bug.py`, `test_ls_tree_drilldown.py` (the agent's `ls` view must stay byte-identical), `test_smoke.py` (the no-CDN integration test stays green).
- Coverage **>90%** on `app/` (the new `compression.py` / `http_cache.py` / `response_budget.py` are small and fully pinned).

## Completion Criteria
- [ ] `GET /api/docs/tree` (default `depth=1`) + `Content-Encoding: gzip` transfers < 40 KB on the audited live-replica corpus (measured by `scripts/api_payload_audit.py`, recorded in `AUDIT.md`) — from ~2 MB — and a repeat load answers **304** with an empty body.
- [ ] Drilling two folders deep renders correct rows (counts, dates, descriptions, thumbnails, pending markers) with exactly one node fetch per uncached level; a vanished folder resets honestly.
- [ ] The stat cards are right at every level (server `totals`), and the tree's per-node `updated_at` is still the recursive max even when the node's children are truncated.
- [ ] `POST /api/chat` streams uncompressed and frame-by-frame (unit + E2E proof); HTML pages still serve `no-cache` with the `?v=` rewrite; every response still carries the A20 headers.
- [ ] `AUDIT.md` lists every candidate endpoint with before/after bytes and its verdict; offenders over the stated budget are fixed or explicitly accepted with a reason.
- [ ] `uv run pytest` green, coverage > 90%, `uv run ruff check . && uv run pyright` clean, the phase E2E green in isolation, one `--no-gpg-sign` commit, phase dir in `complete/`.

## Locked decisions
- **A1 — gzip is selective and pure-ASGI**: `POST /api/chat` (SSE, A15) is skipped by EXACT path; no `BaseHTTPMiddleware`; `BOR_COMPRESS_MIN_BYTES` (`0` = off, negative = fail loudly) — owner-confirmed 2026-10-07 ("gzip + ETag + lazy drill").
- **A2 — `depth` (default 1, max 8) + `totals` on the tree, `truncated` omitted when false**: the depth-limited response keeps every per-node field the view renders (counts, `updated_at`, `summary`, `summary_pending`, image keys) and only omits un-rendered DEPTH; the recursive max-date map is mandatory so phase 106 D9 does not silently regress — owner-confirmed 2026-10-07.
- **A3 — the phase-97 "ONE fetch, drill client-side" contract is superseded** by one initial load + one fetch per drilled level (the deliberate cost of a small first payload; the drill UX, breadcrumb, and scroll-panel behavior stay as phases 97/99/156 built them) — owner-confirmed 2026-10-07.
- **A4 — the ETag is a weak hash of the response body, NOT `sources_meta.version`**: bumping the generation would mark every saved chat `stale` (phase 53), and the admin summary/date/folder PATCHes mutate the tree without bumping it. Recorded as a scoped extension of A19 (validators on two read-only catalogue routes; SSE and mutations untouched) — owner-confirmed 2026-10-07.
- **A5 — the byte budget is a log line + a script, not a hard 413**: no endpoint starts rejecting large payloads this phase (the app's own documents are legitimately large); the audit's 256 KB "look at it" threshold drives follow-up work, recorded in `AUDIT.md`.

## Commit
```bash
git add app/ frontend/ scripts/ tests/ README.md .env.example .agents/phases/ && git commit --no-gpg-sign -m "perf(api): gzip non-SSE responses, a depth-limited KB tree with per-level drill and ETag revalidation, and a measured response byte budget"
```