brain-of-reese/.agents/phases/complete/167_api_payload_diet/00_phase.md

Phase 167 — API payload diet: gzip, a lazily-drilled KB tree, revalidation, and a measured byte budget

Source: TODO.md L5 — "GET https://brain.reeseapps.com/api/docs/tree loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data." Story: n/a (owner-directed performance defect) Context: app/api/docs.py (GET /api/docs/tree L919, the pure builder build_kb_tree L780 / _level_children L671 / _folder_counts L626 / _subtree_max L760, GET /api/docs L129), app/schemas.py (KbTree L576, KbTreeFolder L501, KbTreeSource L539, KbTreeFile L447 + the _drop_image_fields omission precedent), app/main.py (middleware stack: SessionMiddleware → routers → configure_caching → SecurityHeadersMiddleware outermost), app/core/caching.py (the /api/* pass-through contract), app/core/logging.py (§9 log style), frontend/assets/sources.js (loadTree L1625, renderLevel L1888, treeStats L1703, folderExistsIn L1667, resetVanishedLocation L1646), app/models.py (SourcesMeta — deliberately NOT the ETag source, see A3), phases 94/97/98/99/106/122 (the tree contract), A15 (SSE), A19 (caching), A20 (security headers).

Objective

GET /api/docs/tree stops shipping 2 MB: the first load carries what the RAG view actually renders (sources + one level, with server-computed totals), deeper levels load on drill, every response the client already has revalidates to a 304, and every non-SSE response ships gzipped (~8-12× on this JSON). Then the same lens is pointed at the rest of the API with a byte-budget log line and a measured audit, so "any other endpoints that could load a ton of data" becomes a recorded finding with fixes for the offenders instead of a guess.

Why it is 2 MB (traced 2026-10-08)

list_kb_tree returns the FULL recursive tree in one body (the deliberate phase-97 "the RAG view drills CLIENT-side, zero per-level requests" contract): every indexed document as a file node (path, title, chunks, created_at, indexed_at, + is_image/image_url/summary on image nodes) AND every stored folder_summaries row inlined on its folder/source node, with no compression anywhere in the stack (the app has no GZip middleware — app/main.py adds only Session / caching / security-headers; nothing sets Content-Encoding). At the live corpus (≈1 k documents, hundreds of folders with AI descriptions, image docs whose summary is a full vision description) that is ~2 MB of uncompressed JSON per Sources-page load, and the browser re-downloads it after every sync and on every shell boot.

Design (shared by all tasks — the executor reads this, not the chat)

D1 — selective gzip, SSE never touched (task 01, LOCKED A1). New app/core/compression.py, a pure-ASGI middleware (BaseHTTPMiddleware is forbidden here — it buffers):

class SelectiveGZipMiddleware:
    def __init__(self, app, *, minimum_size: int, skip_paths: tuple[str, ...] = ("/api/chat",)) -> None:
        self.app = app
        self.gzip = GZipMiddleware(app, minimum_size=minimum_size)  # starlette, reused, not forked
    async def __call__(self, scope, receive, send):
        if scope["type"] != "http" or scope["path"] in self.skip_paths:
            await self.app(scope, receive, send); return
        await self.gzip(scope, receive, send)

Exact-path skip (never a prefix — /api/chat-images must still compress). Starlette's GZipMiddleware already: requires Accept-Encoding: gzip, respects minimum_size, skips a response that already sets content-encoding, appends Vary: Accept-Encoding, and streams with compressobj + a final flush. app/main.py: new Settings.compress_min_bytes: int = 860 (BOR_COMPRESS_MIN_BYTES, 0 = the feature OFF (skip the middleware entirely), negative → fail startup loudly — the agent_max_rounds pattern, PLAN §13), wired by app.add_middleware(...) before the SecurityHeadersMiddleware line so SecurityHeaders stays outermost (A20) and gzip sits outside the caching layer (an already-rewritten HTML page compresses; /api/* still passes the caching middleware byte-identically). No production nginx change is required; if nginx also gzips, it will not double-encode an already-encoded body.

D2 — a depth-limited tree + server totals (task 02, LOCKED A2). app/api/docs.py:

• build_kb_tree(..., depth: int | None = None) and _level_children(..., depth) — None = unlimited (the legacy behavior every existing unit test keeps, byte-identical); a positive budget stops the recursion. A folder node whose own children were NOT built carries truncated=True, children=[], its recursive documents from counts[sub] (unchanged) and — the correctness trap — its updated_at from a NEW recursive max-date map computed in _folder_counts (because _subtree_max(children) over an empty child list would be None, which would silently break the phase-106 D9 column). Its summary still rides the node (the level row shows a subfolder's description).
• GET /api/docs/tree gains depth: int = Query(1, ge=1, le=8). The default is what breaks the existing pins: the full consumer list is exactly nine files (grep -rl "docs/tree" tests app → tests/integration/test_docs_api.py, tests/integration/test_docs_api_dates.py, tests/unit/test_kb_tree_builder.py, tests/unit/test_kb_tree_ui.py, tests/unit/test_frontend_router.py, tests/e2e/test_kb_tree.py, tests/e2e/test_navbar_refresh.py, tests/e2e/test_document_dates.py, tests/e2e/test_git_source_dates.py), and the TWO DATE SUITES walk straight through the tree with no depth param (test_docs_api_dates.py reaches a → a/b → deeper; test_git_source_dates.py::_find_file recurses by design) — they assert phase-106 D9 dates, not payload shape, so they migrate by requesting ?depth=8 explicitly with every assertion kept (task 02 item 9; the E2E one is named in task 06 item 4).
• KbTree gains totals: KbTreeTotals (documents, chunks, last_indexed: str | None) computed in the endpoint from the SAME catalogue rows (no extra query) — sources.js::treeStats() cannot sum a truncated tree client-side (it walks children) and the three stat cards must stay honest.
• truncated: bool = False on KbTreeFolder/KbTreeSource with a @model_serializer(mode="wrap") omission when False (the _drop_image_fields precedent): an untruncated node's wire shape stays byte-identical, and only the depth-limited nodes add the key.

D3 — the subtree route + revalidation (task 03, LOCKED A3/A4).

• GET /api/docs/tree/node?source=<name>&path=<folder>&depth=1 (admin-gated like /api/docs/tree, path="" = the source root): one node in the KbTreeFolder/KbTreeSource shape (its direct children + truncated flags), 404 {"detail": "unknown node"} for an unregistered+unindexed source or a folder that fails the phase-94 existence rule. The RAG view's drill target; the catalogue query is the same one GET /api/docs/tree runs (documented: one query per drill, no N+1).
• Revalidation, both tree routes (and GET /api/docs): build → serialize once (model_dump_json() / json) → ETag: W/"<sha256(body)[:32]>" + Cache-Control: private, must-revalidate; If-None-Match match (weak/strong and * tolerant) → Response(304) with the same two headers. Helpers live in a new app/core/http_cache.py (weak_etag(body), etag_matches(header, tag)) so the pattern is one function, not a copy. Why not sources_meta.version (LOCKED A3): bumping that generation marks every saved chat stale (phase 53) — the wrong tool — and the admin PATCHes (/api/documents/summary, /api/documents/date, /api/folders/summary) change the tree WITHOUT bumping it, so a version-derived tag would serve stale trees; a content hash is exact, needs no migration, and the tree is built anyway. This is a deliberate, recorded extension of A19 (which governs HTML pages + /assets/* + the caching pass-through) — /api/* gains a validator on two catalogue routes only; the SSE stream and every mutating route are untouched.

D4 — the client drills lazily (task 04). frontend/assets/sources.js: loadTree() → fetch("/api/docs/tree?depth=1") (still ONE first load), a levelCache (Map keyed source + "/" + folder), and a drill that fetches GET /api/docs/tree/node?… only when the target node reports truncated === true and is not cached — merged into the cached node before renderLevel(). Never-stale (§7.4): the drill's in-flight state disables the row's link + shows one "Loading…" row (the .kb-* empty/loading idiom), a failure shows a role="alert" line with a Retry affordance and leaves the current level rendered, and the race token (loadSeq) extends to the node fetch (a stale drill response may never paint). folderExistsIn/resetVanishedLocation stop walking a full tree: the node route's 404 IS the "the folder vanished" signal → reset to the top level. treeStats() reads kbTree.totals (fallback to the existing walk when totals is absent — the defensive no-crash path).

D5 — the byte budget and the audit (task 05). New app/core/response_budget.py, pure ASGI, one log line per /api/* response in the §9 single-line style: api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41 It counts http.response.body chunk bytes as they pass (zero buffering, SSE-safe: duration_ms for a stream is the stream's life, documented), logs at INFO when bytes >= settings.log_large_response_bytes (BOR_LOG_LARGE_RESPONSE_BYTES, default 65536, 0 = log every /api/* response, negative fails startup) and at DEBUG otherwise. GET /api/chat gets a stream=1 variant of the line. New scripts/api_payload_audit.py (python -m scripts.api_payload_audit --base-url … --user … --password …): logs in, hits the candidate endpoints (/api/docs/tree at depth 1 and unlimited, /api/docs, /api/documents/content for the largest doc, /api/chats, /api/chats/{id} for the largest chat, /api/git-sources, /api/suggestions, /api/sync/status, /api/tokens, /api/config), and prints uncompressed AND gzip bytes + duration per endpoint with a stated budget (256 KB uncompressed = "look at it", 64 KB = "fine"). Findings land in this directory as AUDIT.md (endpoint, before/after bytes, the remedy applied or the accepted-why) — the TODO's "any other endpoints" answered with numbers, not vibes. The audit runs against the local 1,000-doc live-replica snapshot (the restore-test-db skill) so the numbers are the production order of magnitude, not the fixture KB's.

D6 — verification (task 06). New tests/e2e/test_kb_tree_lazy_load.py: the Sources page's FIRST /api/docs/tree request carries depth=1 + totals and no deep children (route-level byte assertion + via response.json() node walk), drilling two levels deep issues the node fetch (exactly once per level, cached the second time), the stat cards match totals at every level, a 404 node (folder pruned under us — a page.route stub) resets to the top level with no stale breadcrumb, Accept-Encoding: gzip yields Content-Encoding: gzip + a Vary header, and the second visit revalidates to a 304 (page.route observes the 304 + the body unchanged from cache). Plus the migrated pins: tests/unit/test_kb_tree_ui.py (asserts exactly ONE fetch("/api/docs/tree") occurrence), tests/unit/test_frontend_router.py:591, tests/e2e/test_navbar_refresh.py:115 (the #nav-sources → /api/docs/tree expectation), and the phase-97/99 integration shape tests.

Dependencies

• 97_kb_tree_catalog + 99_kb_tree_table_and_back_nav (complete) — the one-fetch/drill-client-side contract this phase deliberately supersedes (owner-confirmed 2026-10-07, TODO L5); 106_document_dates (D9: a truncated node's updated_at must stay the recursive max — D2's trap), 98_sync_summary_visibility (D3 summary_pending on every node, including truncated ones), 122_image_documents (task 04's image map, which stays a bounded select), 156_table_ux_polish (the bounded scroll panel + sticky header the drill re-renders into), 82_security_headers / A19 / A20 (the middleware order this phase must not disturb), A15 (SSE).
• No migration: nothing in the DB changes.

Tasks

1. 01_gzip_middleware.md — app/core/compression.py + Settings.compress_min_bytes + wiring + the compression/SSE/passthrough tests.
2. 02_tree_depth_and_totals.md — KbTreeTotals, the truncated omission serializer, build_kb_tree(depth=…), the recursive max-date map, depth on /api/docs/tree, builder + integration tests.
3. 03_tree_node_endpoint_etag.md — app/core/http_cache.py, GET /api/docs/tree/node, ETag/304 on the two tree routes + /api/docs, integration matrix.
4. 04_client_lazy_drill.md — sources.js level cache + per-level fetch + §7.4 drill states + totals stat cards + the migrated unit pins.
5. 05_response_budget_audit.md — app/core/response_budget.py + scripts/api_payload_audit.py + AUDIT.md with the measured numbers + the remedies for the offenders (and the accepted-why for the rest) + README/env docs.
6. 06_e2e_payload_diet.md — tests/e2e/test_kb_tree_lazy_load.py, the migrated E2E pins, the full gate set, the commit.

Testing & Quality

• Unit: the builder depth/truncation/max-date matrix (tests/unit/test_kb_tree_builder.py), the schema omission wire-shape pins, tests/unit/test_caching.py extended (the /api/* pass-through survives gzip coexistence), config validator pins, tests/unit/test_kb_tree_ui.py extended for the drill (cache, single fetch per level, race token, loading/alert states, totals stat cards).
• Integration: depth + truncated + totals + node-route matrix (200/404/403), the migrated tests/integration/test_docs_api_dates.py (explicit ?depth=8, every D9 date assertion intact), the ETag matrix (no header → 200 + ETag; match → 304 no body; mismatch → 200; *; weak/strong equivalence; a folder-summary PATCH changes the tag), gzip (Content-Encoding: gzip, Vary, under-minimum_size identity, Accept-Encoding: identity identity, the SSE stream NOT compressed and arriving frame-by-frame), and the response-budget log line (caplog: INFO over the threshold, DEBUG under, stream=1 for /api/chat).
• E2E: tests/e2e/test_kb_tree_lazy_load.py green in isolation; the owning regression set green in isolation: test_kb_tree.py (migrated), test_kb_tree_nav.py, test_navbar_refresh.py, test_document_dates.py, test_git_source_dates.py (its _find_file recurses — it needs the explicit ?depth=8), test_sync_summary_visibility.py, test_table_ux_polish.py, test_sources_midstream_bug.py, test_ls_tree_drilldown.py (the agent's ls view must stay byte-identical), test_smoke.py (the no-CDN integration test stays green).
• Coverage >90% on app/ (the new compression.py / http_cache.py / response_budget.py are small and fully pinned).

Completion Criteria

• GET /api/docs/tree (default depth=1) + Content-Encoding: gzip transfers < 40 KB on the audited live-replica corpus (measured by scripts/api_payload_audit.py, recorded in AUDIT.md) — from ~2 MB — and a repeat load answers 304 with an empty body.
• Drilling two folders deep renders correct rows (counts, dates, descriptions, thumbnails, pending markers) with exactly one node fetch per uncached level; a vanished folder resets honestly.
• The stat cards are right at every level (server totals), and the tree's per-node updated_at is still the recursive max even when the node's children are truncated.
• POST /api/chat streams uncompressed and frame-by-frame (unit + E2E proof); HTML pages still serve no-cache with the ?v= rewrite; every response still carries the A20 headers.
• AUDIT.md lists every candidate endpoint with before/after bytes and its verdict; offenders over the stated budget are fixed or explicitly accepted with a reason.
• uv run pytest green, coverage > 90%, uv run ruff check . && uv run pyright clean, the phase E2E green in isolation, one --no-gpg-sign commit, phase dir in complete/.

Locked decisions

• A1 — gzip is selective and pure-ASGI: POST /api/chat (SSE, A15) is skipped by EXACT path; no BaseHTTPMiddleware; BOR_COMPRESS_MIN_BYTES (0 = off, negative = fail loudly) — owner-confirmed 2026-10-07 ("gzip + ETag + lazy drill").
• A2 — depth (default 1, max 8) + totals on the tree, truncated omitted when false: the depth-limited response keeps every per-node field the view renders (counts, updated_at, summary, summary_pending, image keys) and only omits un-rendered DEPTH; the recursive max-date map is mandatory so phase 106 D9 does not silently regress — owner-confirmed 2026-10-07.
• A3 — the phase-97 "ONE fetch, drill client-side" contract is superseded by one initial load + one fetch per drilled level (the deliberate cost of a small first payload; the drill UX, breadcrumb, and scroll-panel behavior stay as phases 97/99/156 built them) — owner-confirmed 2026-10-07.
• A4 — the ETag is a weak hash of the response body, NOT sources_meta.version: bumping the generation would mark every saved chat stale (phase 53), and the admin summary/date/folder PATCHes mutate the tree without bumping it. Recorded as a scoped extension of A19 (validators on two read-only catalogue routes; SSE and mutations untouched) — owner-confirmed 2026-10-07.
• A5 — the byte budget is a log line + a script, not a hard 413: no endpoint starts rejecting large payloads this phase (the app's own documents are legitimately large); the audit's 256 KB "look at it" threshold drives follow-up work, recorded in AUDIT.md.

Commit

git add app/ frontend/ scripts/ tests/ README.md .env.example .agents/phases/ && git commit --no-gpg-sign -m "perf(api): gzip non-SSE responses, a depth-limited KB tree with per-level drill and ETag revalidation, and a measured response byte budget"