# Phase 167 · Task 05 — Measured API payload audit (the byte budget) Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any other endpoints** that could load a ton of data." This file is the measured answer — every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` / server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a report, never a 413 — nothing was rejected, nothing was capped. ## How it was run | | | |---|---| | Date | 2026-10-08 | | Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) | | App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) | | BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had | | AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) | | Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) | | Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below | The script is **read-only**: it signs in through `POST /api/login` (the session cookie — the single-admin login is password-only, PLAN A10, so there is no `--username`) and GETs. It never mutates, so it is safe against a production target (`--base-url https://…`). ### Corpus caveat (read this before comparing with the 2 MB report) The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents** (verified: `select count(*) from folder_summaries` → `0`, `select count(*) from documents where is_image` → `0`). The production 2 MB payload was 1,000 documents **plus** the hundreds of inlined AI folder descriptions (phase 127/155) and the image documents' full vision descriptions (phase 122/124) — none of which exist in the snapshot. **So the identity bytes below are a lower bound on the production payload**, and what transfers from this measurement is the *structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the ONE level the RAG view renders instead of the whole recursion below it — and every level of a summary-carrying folder is where the production payload's fat lives), gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1 tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict table below names the live re-run as the follow-up. ## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available > Read the BEFORE table's `gzip` column as "no compressor existed": ratio > `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts > are NOT a pre-phase property — the conditional re-requests hit the validators > tasks 01-04 added, which this run left in place because only the transport was > being switched off (the honest way to measure "the same bytes, uncompressed"). > Pre-phase there were no validators at all, so every repeat load cost the > identity column again. ``` api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold) api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json endpoint identity gzip ratio ms verdict ---------------------------------------------------------------------------------------------------------------- tree depth=1 (default) 8,474 8,474 1.000 12 304 tree depth=8 (full) 265,147 265,147 1.000 37 304 tree node homelab/active 20,844 20,844 1.000 11 304 docs catalogue 246,366 246,366 1.000 14 304 document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 892,124 1.000 8 LOOK (> 262144) note: content field 772573 chars saved chats list 12 12 1.000 1 ok saved chat (one) — — — — n/a note: no saved chats git sources 188 188 1.000 1 ok suggestions 74 74 1.000 1 ok sync status (polled) 272 272 1.000 1 ok tokens 13 13 1.000 2 ok config 188 188 1.000 1 ok whoami 37 37 1.000 1 ok api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports. ``` What the pre-phase browser actually paid for the **Sources page first paint** is the sum of the rows it fetches on load, and the pre-phase tree route answered the FULL recursive tree by default (`tree depth=8 (full)` is that shape): `265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app published **no validators at all**, so every repeat load re-paid the same 512 KB. ## AFTER — shipped defaults (depth-1 default + gzip + ETag) api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold) api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json endpoint identity gzip ratio ms verdict ---------------------------------------------------------------------------------------------------------------- tree depth=1 (default) 8,474 1,129 0.133 13 304 tree depth=8 (full) 265,147 23,520 0.089 41 304 tree node homelab/active 20,844 1,064 0.051 11 304 docs catalogue 246,366 45,107 0.183 23 304 document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 60,776 0.068 14 LOOK (> 262144) note: content field 772573 chars saved chats list 12 12 1.000 2 ok saved chat (one) — — — — n/a note: no saved chats git sources 188 188 1.000 1 ok suggestions 74 74 1.000 2 ok sync status (polled) 272 272 1.000 1 ok tokens 13 13 1.000 2 ok config 188 188 1.000 1 ok whoami 37 37 1.000 1 ok api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports. ``` Sources-page first paint, same sum against the AFTER gzip column: `1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity, and every one of those five rows revalidates to **304 / 0 bytes** on the next load). The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×** (31× from the depth limit × 7.5× from gzip), and the second visit is free. ## `curl` proof on the audited corpus ``` $ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree etag: W/"53c86d5d554efd8487ad52ca6cd6f913" cache-control: private, must-revalidate content-length: 8474 vary: Cookie, Accept-Encoding 8474 bytes # wc -c b1.json $ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree etag: W/"53c86d5d554efd8487ad52ca6cd6f913" # the validator survives the encoding content-encoding: gzip 1129 bytes # wc -c b2.gz $ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \ -w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree status=304 size=0 ``` ## The byte-budget log line (the answer to "which endpoint is heavy, right now") Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO); the shipped default is `65536`, so at rest only the big ones reach the log. `grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what went on the wire — the two together are the "how fat, and how fat did the client see it" pair. ``` INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12 INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40 INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10 INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22 INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12 INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0 INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0 ``` `POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded payload to budget — and it is skipped by the gzip layer by exact path (which is why its line reads `enc=identity`). One real turn against this corpus, measured: ``` INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1 ``` 39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture ended because the measuring `curl --max-time 45` hung up — which is itself the other half of the contract: a stream the client walks away from still writes exactly ONE line, with the bytes that actually got out, never one line per frame and never a missing line). Pinned by `tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out` and by the integration pin `tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`. ## Verdict per candidate endpoint | Endpoint | Measured (identity / wire) | Verdict | |---|---|---| | `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. | | `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). | | `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. | | `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). | | `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). | | `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. | | `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. | | `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). | | `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). | | `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. | | `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). | | `GET /api/config` | 188 | **ok** — display strings only. | | `GET /api/whoami` | 37 | **ok** — two fields. | | `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. | ## What the audit changed, and what it proved * **No new remedy was needed beyond tasks 01-04.** The only over-budget row (`/api/documents/content`) is the documented accepted case, and `/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag was **not** applied — a validator on a sub-860-byte response costs a hash and saves nothing (it would not even compress). * **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06) drives the real Sources view in Chromium and pins what this file measured: the boot is ONE `?depth=1` request whose body carries `totals` and no deep children, the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary` (Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the wire answers `304` for a fraction of the first transfer while the table still renders from the cache, and a level the payload only names costs exactly one `/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands `fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP Network events — the DevTools-panel truth, not Playwright's post-cache view. * **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat load free. Had the phase shipped gzip alone, the Sources page would still have paid 23 KB/request forever, and on the summary-inflated production corpus proportionally more. * **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the LOOK bar, already validated + compressed). The byte-budget line at its default `65536` threshold logs it at INFO on every uncached load, so the corpus growth that pushes it over shows up in the log before it shows up for the owner. * **Two limits of this measurement, stated instead of hidden:** (1) the snapshot has no folder summaries and no image docs, so the identity numbers are a lower bound on the production payload (see the caveat above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather than measured — the script is read-only by design (it must stay safe against the deployed app), so it does not invent a chat to size. Both are re-measurable with one command against the live corpus. ## Re-running this audit ```bash # 1. the corpus (destructive to the local DB — see the skill's own backup step) # restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese uv run alembic upgrade head # the snapshot carries an old alembic version # 2. the app uv run uvicorn app.main:app --port 8000 # 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override) uv run python -m scripts.api_payload_audit --base-url http://localhost:8000 # the BEFORE transport, for a before/after pair like the tables above: BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit ```