phase: 167_api_payload_diet
All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
@@ -0,0 +1,236 @@
|
||||
# Phase 167 · Task 05 — Measured API payload audit (the byte budget)
|
||||
|
||||
Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any
|
||||
other endpoints** that could load a ton of data." This file is the measured answer —
|
||||
every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` /
|
||||
server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a
|
||||
report, never a 413 — nothing was rejected, nothing was capped.
|
||||
|
||||
## How it was run
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-10-08 |
|
||||
| Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) |
|
||||
| App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) |
|
||||
| BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had |
|
||||
| AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) |
|
||||
| Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) |
|
||||
| Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below |
|
||||
|
||||
The script is **read-only**: it signs in through `POST /api/login` (the session
|
||||
cookie — the single-admin login is password-only, PLAN A10, so there is no
|
||||
`--username`) and GETs. It never mutates, so it is safe against a production target
|
||||
(`--base-url https://…`).
|
||||
|
||||
### Corpus caveat (read this before comparing with the 2 MB report)
|
||||
|
||||
The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents**
|
||||
(verified: `select count(*) from folder_summaries` → `0`,
|
||||
`select count(*) from documents where is_image` → `0`). The production 2 MB payload
|
||||
was 1,000 documents **plus** the hundreds of inlined AI folder descriptions
|
||||
(phase 127/155) and the image documents' full vision descriptions (phase 122/124) —
|
||||
none of which exist in the snapshot. **So the identity bytes below are a lower bound
|
||||
on the production payload**, and what transfers from this measurement is the
|
||||
*structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the
|
||||
ONE level the RAG view renders instead of the whole recursion below it — and every
|
||||
level of a summary-carrying folder is where the production payload's fat lives),
|
||||
gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1
|
||||
tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict
|
||||
table below names the live re-run as the follow-up.
|
||||
|
||||
## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available
|
||||
|
||||
> Read the BEFORE table's `gzip` column as "no compressor existed": ratio
|
||||
> `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts
|
||||
> are NOT a pre-phase property — the conditional re-requests hit the validators
|
||||
> tasks 01-04 added, which this run left in place because only the transport was
|
||||
> being switched off (the honest way to measure "the same bytes, uncompressed").
|
||||
> Pre-phase there were no validators at all, so every repeat load cost the
|
||||
> identity column again.
|
||||
|
||||
```
|
||||
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||
endpoint identity gzip ratio ms verdict
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
tree depth=1 (default) 8,474 8,474 1.000 12 304
|
||||
tree depth=8 (full) 265,147 265,147 1.000 37 304
|
||||
tree node homelab/active 20,844 20,844 1.000 11 304
|
||||
docs catalogue 246,366 246,366 1.000 14 304
|
||||
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 892,124 1.000 8 LOOK (> 262144)
|
||||
note: content field 772573 chars
|
||||
saved chats list 12 12 1.000 1 ok
|
||||
saved chat (one) — — — — n/a
|
||||
note: no saved chats
|
||||
git sources 188 188 1.000 1 ok
|
||||
suggestions 74 74 1.000 1 ok
|
||||
sync status (polled) 272 272 1.000 1 ok
|
||||
tokens 13 13 1.000 2 ok
|
||||
config 188 188 1.000 1 ok
|
||||
whoami 37 37 1.000 1 ok
|
||||
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||
```
|
||||
|
||||
What the pre-phase browser actually paid for the **Sources page first paint** is the
|
||||
sum of the rows it fetches on load, and the pre-phase tree route answered the FULL
|
||||
recursive tree by default (`tree depth=8 (full)` is that shape):
|
||||
`265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app
|
||||
published **no validators at all**, so every repeat load re-paid the same 512 KB.
|
||||
|
||||
## AFTER — shipped defaults (depth-1 default + gzip + ETag)
|
||||
|
||||
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||
endpoint identity gzip ratio ms verdict
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
tree depth=1 (default) 8,474 1,129 0.133 13 304
|
||||
tree depth=8 (full) 265,147 23,520 0.089 41 304
|
||||
tree node homelab/active 20,844 1,064 0.051 11 304
|
||||
docs catalogue 246,366 45,107 0.183 23 304
|
||||
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 60,776 0.068 14 LOOK (> 262144)
|
||||
note: content field 772573 chars
|
||||
saved chats list 12 12 1.000 2 ok
|
||||
saved chat (one) — — — — n/a
|
||||
note: no saved chats
|
||||
git sources 188 188 1.000 1 ok
|
||||
suggestions 74 74 1.000 2 ok
|
||||
sync status (polled) 272 272 1.000 1 ok
|
||||
tokens 13 13 1.000 2 ok
|
||||
config 188 188 1.000 1 ok
|
||||
whoami 37 37 1.000 1 ok
|
||||
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||
```
|
||||
|
||||
Sources-page first paint, same sum against the AFTER gzip column:
|
||||
`1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity,
|
||||
and every one of those five rows revalidates to **304 / 0 bytes** on the next load).
|
||||
The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×**
|
||||
(31× from the depth limit × 7.5× from gzip), and the second visit is free.
|
||||
|
||||
## `curl` proof on the audited corpus
|
||||
|
||||
```
|
||||
$ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree
|
||||
etag: W/"53c86d5d554efd8487ad52ca6cd6f913"
|
||||
cache-control: private, must-revalidate
|
||||
content-length: 8474
|
||||
vary: Cookie, Accept-Encoding
|
||||
8474 bytes # wc -c b1.json
|
||||
|
||||
$ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree
|
||||
etag: W/"53c86d5d554efd8487ad52ca6cd6f913" # the validator survives the encoding
|
||||
content-encoding: gzip
|
||||
1129 bytes # wc -c b2.gz
|
||||
|
||||
$ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \
|
||||
-w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree
|
||||
status=304 size=0
|
||||
```
|
||||
|
||||
## The byte-budget log line (the answer to "which endpoint is heavy, right now")
|
||||
|
||||
Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO);
|
||||
the shipped default is `65536`, so at rest only the big ones reach the log.
|
||||
`grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what
|
||||
went on the wire — the two together are the "how fat, and how fat did the client see
|
||||
it" pair.
|
||||
|
||||
```
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10
|
||||
INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22
|
||||
INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12
|
||||
INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0
|
||||
INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0
|
||||
```
|
||||
|
||||
`POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded
|
||||
payload to budget — and it is skipped by the gzip layer by exact path (which is why
|
||||
its line reads `enc=identity`). One real turn against this corpus, measured:
|
||||
|
||||
```
|
||||
INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1
|
||||
```
|
||||
|
||||
39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture
|
||||
ended because the measuring `curl --max-time 45` hung up — which is itself the
|
||||
other half of the contract: a stream the client walks away from still writes
|
||||
exactly ONE line, with the bytes that actually got out, never one line per frame
|
||||
and never a missing line). Pinned by
|
||||
`tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out`
|
||||
and by the integration pin
|
||||
`tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`.
|
||||
|
||||
## Verdict per candidate endpoint
|
||||
|
||||
| Endpoint | Measured (identity / wire) | Verdict |
|
||||
|---|---|---|
|
||||
| `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. |
|
||||
| `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). |
|
||||
| `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. |
|
||||
| `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). |
|
||||
| `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). |
|
||||
| `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. |
|
||||
| `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. |
|
||||
| `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). |
|
||||
| `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). |
|
||||
| `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. |
|
||||
| `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). |
|
||||
| `GET /api/config` | 188 | **ok** — display strings only. |
|
||||
| `GET /api/whoami` | 37 | **ok** — two fields. |
|
||||
| `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. |
|
||||
|
||||
## What the audit changed, and what it proved
|
||||
|
||||
* **No new remedy was needed beyond tasks 01-04.** The only over-budget row
|
||||
(`/api/documents/content`) is the documented accepted case, and
|
||||
`/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag
|
||||
was **not** applied — a validator on a sub-860-byte response costs a hash
|
||||
and saves nothing (it would not even compress).
|
||||
* **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06)
|
||||
drives the real Sources view in Chromium and pins what this file measured: the
|
||||
boot is ONE `?depth=1` request whose body carries `totals` and no deep children,
|
||||
the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary`
|
||||
(Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the
|
||||
wire answers `304` for a fraction of the first transfer while the table still
|
||||
renders from the cache, and a level the payload only names costs exactly one
|
||||
`/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands
|
||||
`fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP
|
||||
Network events — the DevTools-panel truth, not Playwright's post-cache view.
|
||||
* **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on
|
||||
the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the
|
||||
second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat
|
||||
load free. Had the phase shipped gzip alone, the Sources page would still
|
||||
have paid 23 KB/request forever, and on the summary-inflated production
|
||||
corpus proportionally more.
|
||||
* **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the
|
||||
LOOK bar, already validated + compressed). The byte-budget line at its
|
||||
default `65536` threshold logs it at INFO on every uncached load, so the
|
||||
corpus growth that pushes it over shows up in the log before it shows up
|
||||
for the owner.
|
||||
* **Two limits of this measurement, stated instead of hidden:** (1) the
|
||||
snapshot has no folder summaries and no image docs, so the identity
|
||||
numbers are a lower bound on the production payload (see the caveat
|
||||
above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather
|
||||
than measured — the script is read-only by design (it must stay safe
|
||||
against the deployed app), so it does not invent a chat to size. Both are
|
||||
re-measurable with one command against the live corpus.
|
||||
|
||||
## Re-running this audit
|
||||
|
||||
```bash
|
||||
# 1. the corpus (destructive to the local DB — see the skill's own backup step)
|
||||
# restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese
|
||||
uv run alembic upgrade head # the snapshot carries an old alembic version
|
||||
# 2. the app
|
||||
uv run uvicorn app.main:app --port 8000
|
||||
# 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override)
|
||||
uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
|
||||
# the BEFORE transport, for a before/after pair like the tables above:
|
||||
BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit
|
||||
```
|
||||
Reference in New Issue
Block a user