phase: 167_api_payload_diet
All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
@@ -0,0 +1,82 @@
|
||||
# Phase 167 — API payload diet: gzip, a lazily-drilled KB tree, revalidation, and a measured byte budget
|
||||
|
||||
**Source:** `TODO.md` L5 — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data."
|
||||
**Story:** n/a (owner-directed performance defect)
|
||||
**Context:** `app/api/docs.py` (`GET /api/docs/tree` L919, the pure builder `build_kb_tree` L780 / `_level_children` L671 / `_folder_counts` L626 / `_subtree_max` L760, `GET /api/docs` L129), `app/schemas.py` (`KbTree` L576, `KbTreeFolder` L501, `KbTreeSource` L539, `KbTreeFile` L447 + the `_drop_image_fields` omission precedent), `app/main.py` (middleware stack: `SessionMiddleware` → routers → `configure_caching` → `SecurityHeadersMiddleware` outermost), `app/core/caching.py` (the `/api/*` pass-through contract), `app/core/logging.py` (§9 log style), `frontend/assets/sources.js` (`loadTree` L1625, `renderLevel` L1888, `treeStats` L1703, `folderExistsIn` L1667, `resetVanishedLocation` L1646), `app/models.py` (`SourcesMeta` — deliberately NOT the ETag source, see A3), phases 94/97/98/99/106/122 (the tree contract), A15 (SSE), A19 (caching), A20 (security headers).
|
||||
|
||||
## Objective
|
||||
`GET /api/docs/tree` stops shipping 2 MB: the first load carries what the RAG view actually renders (sources + one level, with server-computed totals), deeper levels load on drill, every response the client already has revalidates to a 304, and **every** non-SSE response ships gzipped (~8-12× on this JSON). Then the same lens is pointed at the rest of the API with a byte-budget log line and a measured audit, so "any other endpoints that could load a ton of data" becomes a recorded finding with fixes for the offenders instead of a guess.
|
||||
|
||||
## Why it is 2 MB (traced 2026-10-08)
|
||||
`list_kb_tree` returns the FULL recursive tree in one body (the deliberate phase-97 "the RAG view drills CLIENT-side, zero per-level requests" contract): every indexed document as a file node (`path`, `title`, `chunks`, `created_at`, `indexed_at`, + `is_image`/`image_url`/`summary` on image nodes) AND **every stored `folder_summaries` row inlined on its folder/source node**, with no compression anywhere in the stack (the app has no GZip middleware — `app/main.py` adds only Session / caching / security-headers; nothing sets `Content-Encoding`). At the live corpus (≈1 k documents, hundreds of folders with AI descriptions, image docs whose `summary` is a full vision description) that is ~2 MB of uncompressed JSON per Sources-page load, and the browser re-downloads it after every sync and on every shell boot.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
**D1 — selective gzip, SSE never touched (task 01, LOCKED A1).** New `app/core/compression.py`, a **pure-ASGI** middleware (`BaseHTTPMiddleware` is forbidden here — it buffers):
|
||||
```python
|
||||
class SelectiveGZipMiddleware:
|
||||
def __init__(self, app, *, minimum_size: int, skip_paths: tuple[str, ...] = ("/api/chat",)) -> None:
|
||||
self.app = app
|
||||
self.gzip = GZipMiddleware(app, minimum_size=minimum_size) # starlette, reused, not forked
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] != "http" or scope["path"] in self.skip_paths:
|
||||
await self.app(scope, receive, send); return
|
||||
await self.gzip(scope, receive, send)
|
||||
```
|
||||
Exact-path skip (never a prefix — `/api/chat-images` must still compress). Starlette's `GZipMiddleware` already: requires `Accept-Encoding: gzip`, respects `minimum_size`, skips a response that already sets `content-encoding`, appends `Vary: Accept-Encoding`, and streams with `compressobj` + a final flush. `app/main.py`: new `Settings.compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`, `0` = the feature OFF (skip the middleware entirely), negative → fail startup loudly — the `agent_max_rounds` pattern, PLAN §13), wired by `app.add_middleware(...)` **before** the `SecurityHeadersMiddleware` line so SecurityHeaders stays outermost (A20) and gzip sits outside the caching layer (an already-rewritten HTML page compresses; `/api/*` still passes the caching middleware byte-identically). No production nginx change is required; if nginx also gzips, it will not double-encode an already-encoded body.
|
||||
|
||||
**D2 — a depth-limited tree + server totals (task 02, LOCKED A2).** `app/api/docs.py`:
|
||||
- `build_kb_tree(..., depth: int | None = None)` and `_level_children(..., depth)` — `None` = unlimited (the legacy behavior every existing unit test keeps, byte-identical); a positive budget stops the recursion. A folder node whose own children were NOT built carries `truncated=True`, `children=[]`, its recursive `documents` from `counts[sub]` (unchanged) and — the correctness trap — its `updated_at` from a NEW recursive max-date map computed in `_folder_counts` (because `_subtree_max(children)` over an empty child list would be `None`, which would silently break the phase-106 D9 column). Its `summary` still rides the node (the level row shows a subfolder's description).
|
||||
- `GET /api/docs/tree` gains `depth: int = Query(1, ge=1, le=8)`. **The default is what breaks the existing pins:** the full consumer list is exactly nine files (`grep -rl "docs/tree" tests app` → `tests/integration/test_docs_api.py`, `tests/integration/test_docs_api_dates.py`, `tests/unit/test_kb_tree_builder.py`, `tests/unit/test_kb_tree_ui.py`, `tests/unit/test_frontend_router.py`, `tests/e2e/test_kb_tree.py`, `tests/e2e/test_navbar_refresh.py`, `tests/e2e/test_document_dates.py`, `tests/e2e/test_git_source_dates.py`), and the TWO DATE SUITES walk straight through the tree with no depth param (`test_docs_api_dates.py` reaches `a` → `a/b` → deeper; `test_git_source_dates.py::_find_file` recurses by design) — they assert phase-106 D9 dates, not payload shape, so they migrate by requesting `?depth=8` explicitly with every assertion kept (task 02 item 9; the E2E one is named in task 06 item 4).
|
||||
- `KbTree` gains **`totals: KbTreeTotals`** (`documents`, `chunks`, `last_indexed: str | None`) computed in the endpoint from the SAME catalogue rows (no extra query) — `sources.js::treeStats()` cannot sum a truncated tree client-side (it walks children) and the three stat cards must stay honest.
|
||||
- `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `@model_serializer(mode="wrap")` omission when `False` (the `_drop_image_fields` precedent): an untruncated node's wire shape stays byte-identical, and only the depth-limited nodes add the key.
|
||||
|
||||
**D3 — the subtree route + revalidation (task 03, LOCKED A3/A4).**
|
||||
- `GET /api/docs/tree/node?source=<name>&path=<folder>&depth=1` (admin-gated like `/api/docs/tree`, `path=""` = the source root): one node in the `KbTreeFolder`/`KbTreeSource` shape (its direct children + `truncated` flags), 404 `{"detail": "unknown node"}` for an unregistered+unindexed source or a folder that fails the phase-94 existence rule. The RAG view's drill target; the catalogue query is the same one `GET /api/docs/tree` runs (documented: one query per drill, no N+1).
|
||||
- Revalidation, both tree routes (and `GET /api/docs`): build → serialize once (`model_dump_json()` / json) → `ETag: W/"<sha256(body)[:32]>"` + `Cache-Control: private, must-revalidate`; `If-None-Match` match (weak/strong and `*` tolerant) → `Response(304)` with the same two headers. Helpers live in a new `app/core/http_cache.py` (`weak_etag(body)`, `etag_matches(header, tag)`) so the pattern is one function, not a copy. **Why not `sources_meta.version`** (LOCKED A3): bumping that generation marks every saved chat `stale` (phase 53) — the wrong tool — and the admin PATCHes (`/api/documents/summary`, `/api/documents/date`, `/api/folders/summary`) change the tree WITHOUT bumping it, so a version-derived tag would serve stale trees; a content hash is exact, needs no migration, and the tree is built anyway. This is a deliberate, recorded extension of A19 (which governs HTML pages + `/assets/*` + the caching pass-through) — `/api/*` gains a validator on two catalogue routes only; the SSE stream and every mutating route are untouched.
|
||||
|
||||
**D4 — the client drills lazily (task 04).** `frontend/assets/sources.js`: `loadTree()` → `fetch("/api/docs/tree?depth=1")` (still ONE first load), a `levelCache` (Map keyed `source + "/" + folder`), and a drill that fetches `GET /api/docs/tree/node?…` **only** when the target node reports `truncated === true` and is not cached — merged into the cached node before `renderLevel()`. Never-stale (§7.4): the drill's in-flight state disables the row's link + shows one "Loading…" row (the `.kb-*` empty/loading idiom), a failure shows a `role="alert"` line with a Retry affordance and leaves the current level rendered, and the race token (`loadSeq`) extends to the node fetch (a stale drill response may never paint). `folderExistsIn`/`resetVanishedLocation` stop walking a full tree: the node route's 404 IS the "the folder vanished" signal → reset to the top level. `treeStats()` reads `kbTree.totals` (fallback to the existing walk when `totals` is absent — the defensive no-crash path).
|
||||
|
||||
**D5 — the byte budget and the audit (task 05).** New `app/core/response_budget.py`, pure ASGI, one log line per `/api/*` response in the §9 single-line style:
|
||||
`api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41`
|
||||
It counts `http.response.body` chunk bytes as they pass (zero buffering, SSE-safe: `duration_ms` for a stream is the stream's life, documented), logs at INFO when `bytes >= settings.log_large_response_bytes` (`BOR_LOG_LARGE_RESPONSE_BYTES`, default 65536, `0` = log every `/api/*` response, negative fails startup) and at DEBUG otherwise. `GET /api/chat` gets a `stream=1` variant of the line. New `scripts/api_payload_audit.py` (`python -m scripts.api_payload_audit --base-url … --user … --password …`): logs in, hits the candidate endpoints (`/api/docs/tree` at depth 1 and unlimited, `/api/docs`, `/api/documents/content` for the largest doc, `/api/chats`, `/api/chats/{id}` for the largest chat, `/api/git-sources`, `/api/suggestions`, `/api/sync/status`, `/api/tokens`, `/api/config`), and prints uncompressed AND gzip bytes + duration per endpoint with a stated budget (256 KB uncompressed = "look at it", 64 KB = "fine"). Findings land in this directory as `AUDIT.md` (endpoint, before/after bytes, the remedy applied or the accepted-why) — the TODO's "any other endpoints" answered with numbers, not vibes. The audit runs against the local 1,000-doc live-replica snapshot (the `restore-test-db` skill) so the numbers are the production order of magnitude, not the fixture KB's.
|
||||
|
||||
**D6 — verification (task 06).** New `tests/e2e/test_kb_tree_lazy_load.py`: the Sources page's FIRST `/api/docs/tree` request carries `depth=1` + `totals` and no deep children (route-level byte assertion + `via response.json()` node walk), drilling two levels deep issues the node fetch (exactly once per level, cached the second time), the stat cards match `totals` at every level, a 404 node (folder pruned under us — a `page.route` stub) resets to the top level with no stale breadcrumb, `Accept-Encoding: gzip` yields `Content-Encoding: gzip` + a `Vary` header, and the second visit revalidates to a 304 (`page.route` observes the 304 + the body unchanged from cache). Plus the migrated pins: `tests/unit/test_kb_tree_ui.py` (asserts exactly ONE `fetch("/api/docs/tree")` occurrence), `tests/unit/test_frontend_router.py:591`, `tests/e2e/test_navbar_refresh.py:115` (the `#nav-sources` → `/api/docs/tree` expectation), and the phase-97/99 integration shape tests.
|
||||
|
||||
## Dependencies
|
||||
- `97_kb_tree_catalog` + `99_kb_tree_table_and_back_nav` (complete) — the one-fetch/drill-client-side contract this phase **deliberately supersedes** (owner-confirmed 2026-10-07, TODO L5); `106_document_dates` (D9: a truncated node's `updated_at` must stay the recursive max — D2's trap), `98_sync_summary_visibility` (D3 `summary_pending` on every node, including truncated ones), `122_image_documents` (task 04's image map, which stays a bounded select), `156_table_ux_polish` (the bounded scroll panel + sticky header the drill re-renders into), `82_security_headers` / A19 / A20 (the middleware order this phase must not disturb), A15 (SSE).
|
||||
- No migration: nothing in the DB changes.
|
||||
|
||||
## Tasks
|
||||
1. `01_gzip_middleware.md` — `app/core/compression.py` + `Settings.compress_min_bytes` + wiring + the compression/SSE/passthrough tests.
|
||||
2. `02_tree_depth_and_totals.md` — `KbTreeTotals`, the `truncated` omission serializer, `build_kb_tree(depth=…)`, the recursive max-date map, `depth` on `/api/docs/tree`, builder + integration tests.
|
||||
3. `03_tree_node_endpoint_etag.md` — `app/core/http_cache.py`, `GET /api/docs/tree/node`, ETag/304 on the two tree routes + `/api/docs`, integration matrix.
|
||||
4. `04_client_lazy_drill.md` — `sources.js` level cache + per-level fetch + §7.4 drill states + `totals` stat cards + the migrated unit pins.
|
||||
5. `05_response_budget_audit.md` — `app/core/response_budget.py` + `scripts/api_payload_audit.py` + `AUDIT.md` with the measured numbers + the remedies for the offenders (and the accepted-why for the rest) + README/env docs.
|
||||
6. `06_e2e_payload_diet.md` — `tests/e2e/test_kb_tree_lazy_load.py`, the migrated E2E pins, the full gate set, the commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the builder depth/truncation/max-date matrix (`tests/unit/test_kb_tree_builder.py`), the schema omission wire-shape pins, `tests/unit/test_caching.py` extended (the `/api/*` pass-through survives gzip coexistence), config validator pins, `tests/unit/test_kb_tree_ui.py` extended for the drill (cache, single fetch per level, race token, loading/alert states, `totals` stat cards).
|
||||
- Integration: depth + truncated + totals + node-route matrix (200/404/403), the migrated `tests/integration/test_docs_api_dates.py` (explicit `?depth=8`, every D9 date assertion intact), the ETag matrix (no header → 200 + `ETag`; match → 304 no body; mismatch → 200; `*`; weak/strong equivalence; a folder-summary PATCH changes the tag), gzip (`Content-Encoding: gzip`, `Vary`, under-`minimum_size` identity, `Accept-Encoding: identity` identity, the SSE stream NOT compressed and arriving frame-by-frame), and the response-budget log line (caplog: INFO over the threshold, DEBUG under, `stream=1` for `/api/chat`).
|
||||
- E2E: `tests/e2e/test_kb_tree_lazy_load.py` green **in isolation**; the owning regression set green in isolation: `test_kb_tree.py` (migrated), `test_kb_tree_nav.py`, `test_navbar_refresh.py`, `test_document_dates.py`, `test_git_source_dates.py` (its `_find_file` recurses — it needs the explicit `?depth=8`), `test_sync_summary_visibility.py`, `test_table_ux_polish.py`, `test_sources_midstream_bug.py`, `test_ls_tree_drilldown.py` (the agent's `ls` view must stay byte-identical), `test_smoke.py` (the no-CDN integration test stays green).
|
||||
- Coverage **>90%** on `app/` (the new `compression.py` / `http_cache.py` / `response_budget.py` are small and fully pinned).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/docs/tree` (default `depth=1`) + `Content-Encoding: gzip` transfers < 40 KB on the audited live-replica corpus (measured by `scripts/api_payload_audit.py`, recorded in `AUDIT.md`) — from ~2 MB — and a repeat load answers **304** with an empty body.
|
||||
- [ ] Drilling two folders deep renders correct rows (counts, dates, descriptions, thumbnails, pending markers) with exactly one node fetch per uncached level; a vanished folder resets honestly.
|
||||
- [ ] The stat cards are right at every level (server `totals`), and the tree's per-node `updated_at` is still the recursive max even when the node's children are truncated.
|
||||
- [ ] `POST /api/chat` streams uncompressed and frame-by-frame (unit + E2E proof); HTML pages still serve `no-cache` with the `?v=` rewrite; every response still carries the A20 headers.
|
||||
- [ ] `AUDIT.md` lists every candidate endpoint with before/after bytes and its verdict; offenders over the stated budget are fixed or explicitly accepted with a reason.
|
||||
- [ ] `uv run pytest` green, coverage > 90%, `uv run ruff check . && uv run pyright` clean, the phase E2E green in isolation, one `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — gzip is selective and pure-ASGI**: `POST /api/chat` (SSE, A15) is skipped by EXACT path; no `BaseHTTPMiddleware`; `BOR_COMPRESS_MIN_BYTES` (`0` = off, negative = fail loudly) — owner-confirmed 2026-10-07 ("gzip + ETag + lazy drill").
|
||||
- **A2 — `depth` (default 1, max 8) + `totals` on the tree, `truncated` omitted when false**: the depth-limited response keeps every per-node field the view renders (counts, `updated_at`, `summary`, `summary_pending`, image keys) and only omits un-rendered DEPTH; the recursive max-date map is mandatory so phase 106 D9 does not silently regress — owner-confirmed 2026-10-07.
|
||||
- **A3 — the phase-97 "ONE fetch, drill client-side" contract is superseded** by one initial load + one fetch per drilled level (the deliberate cost of a small first payload; the drill UX, breadcrumb, and scroll-panel behavior stay as phases 97/99/156 built them) — owner-confirmed 2026-10-07.
|
||||
- **A4 — the ETag is a weak hash of the response body, NOT `sources_meta.version`**: bumping the generation would mark every saved chat `stale` (phase 53), and the admin summary/date/folder PATCHes mutate the tree without bumping it. Recorded as a scoped extension of A19 (validators on two read-only catalogue routes; SSE and mutations untouched) — owner-confirmed 2026-10-07.
|
||||
- **A5 — the byte budget is a log line + a script, not a hard 413**: no endpoint starts rejecting large payloads this phase (the app's own documents are legitimately large); the audit's 256 KB "look at it" threshold drives follow-up work, recorded in `AUDIT.md`.
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ frontend/ scripts/ tests/ README.md .env.example .agents/phases/ && git commit --no-gpg-sign -m "perf(api): gzip non-SSE responses, a depth-limited KB tree with per-level drill and ETag revalidation, and a measured response byte budget"
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 01 — Selective gzip: `app/core/compression.py`, `BOR_COMPRESS_MIN_BYTES`, SSE untouched (phase 167 D1, LOCKED A1)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Every HTTP response except the `POST /api/chat` SSE stream compresses with gzip, with a config kill switch, the existing middleware order and contracts (A19 caching pass-through, A20 security headers outermost) untouched, and a test that proves the chat stream still arrives frame-by-frame and uncompressed.
|
||||
|
||||
## Work
|
||||
1. New `app/core/compression.py` — the middleware of phase 167 D1, **pure ASGI** (never `BaseHTTPMiddleware`, which buffers and would break streaming responses):
|
||||
- `class SelectiveGZipMiddleware` holding `self.app` and a composed `starlette.middleware.gzip.GZipMiddleware(self.app, minimum_size=minimum_size)`; `__call__` delegates to the gzip path unless `scope["type"] != "http"` or `scope["path"] in skip_paths` (EXACT match — `/api/chat-images` must still compress), in which case it calls `self.app` directly;
|
||||
- `DEFAULT_SKIP_PATHS = ("/api/chat",)` module constant, docstring names A15 (SSE) and the buffering reason; do NOT re-implement compression math — Starlette's `GZipMiddleware` owns `Accept-Encoding` negotiation, `minimum_size`, the already-`content-encoding` skip, the `Vary: Accept-Encoding` append, and the `compressobj` streaming + final flush;
|
||||
- module docstring: the TODO's 2 MB defect, the ~8-12× JSON expectation, the SSE exclusion, and "nginx may also gzip; an already-encoded body is never double-encoded".
|
||||
2. `app/config.py` — `compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`) beside the other ints, with the `agent_max_rounds` validator pattern (`_compress_min_bytes_non_negative`: negative → `ValueError("compress_min_bytes must be >= 0 (0 = compression disabled)")`) and the docstring stating `0` = the feature is OFF (no middleware), `> 0` = the Starlette minimum-size floor (860 = the classic nginx default, below which compression can inflate).
|
||||
3. `app/main.py` — wire it, added **before** the `app.add_middleware(SecurityHeadersMiddleware)` line so SecurityHeaders stays outermost (A20, the existing comment says "last add_middleware"): construct `SelectiveGZipMiddleware` only when `settings.compress_min_bytes > 0` (the `0` = OFF path adds nothing at all), with a comment naming TODO L5, the SSE skip, and the position (outside `configure_caching`, inside SecurityHeaders → the rewritten HTML pages compress too, `/api/*` still passes the caching layer byte-identically before compression).
|
||||
4. `.env.example` — `BOR_COMPRESS_MIN_BYTES=860` with the three-value comment (0 off, negative refuses startup, the 860 rationale).
|
||||
5. New `tests/unit/test_compression.py`:
|
||||
- a tiny app with a ~50 KB JSON route + a ~50 byte route, wrapped in the middleware: `Accept-Encoding: gzip` → `Content-Encoding: gzip` + a body that `gzip.decompress`es to the original + `Vary` mentions `accept-encoding`; `Accept-Encoding: identity` → no `Content-Encoding`; the small route → identity (under the floor); a response that already set `content-encoding` → untouched;
|
||||
- `skip_paths`: a `/api/chat`-named route in the test app is NOT compressed and streams incrementally (assert the received chunks arrive before the response completes — the `list(stream)` idiom the existing SSE tests use), while a `/api/chat-images/x` route IS compressed (the exact-match pin);
|
||||
- `scope["type"] == "lifespan"`/`"websocket"` pass-through;
|
||||
- config: `compress_min_bytes = 0` → `create_app` adds no gzip layer (assert via `app.user_middleware`'s classes), `-1` → the validator raises naming the field.
|
||||
6. Integration pin `tests/integration/test_docs_api.py` (or the middleware-owning integration file — locate via `SecurityHeadersMiddleware` in `tests/`): the real app's `GET /api/health`-class small response stays identity; a large `/api/docs/tree` (seed ≥ 50 docs) returns `Content-Encoding: gzip` and decompresses to the documented JSON; the A20 headers (`CSP`, `X-Content-Type-Options`, `X-Frame-Options`/`frame-ancestors`) are STILL present on a gzipped response (the outer-layer invariant); the HTML page contract (A19: `no-cache`, `?v=` rewrite, no validators) survives decompression.
|
||||
7. ASSUMPTION: `Settings.compress_min_bytes` (name + default 860) — the TODO named no knob; the house kill-switch convention (`0` off, negative loud) is what makes it owner-controllable without a code change.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit + integration as above; `uv run pytest tests/integration/test_chat_api.py -v` (or the suite that owns the SSE contract) stays green — the stream is not compressed and not buffered.
|
||||
- Coverage: **>90%** on `app/core/compression.py` (every branch listed above is pinned).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `curl -H 'Accept-Encoding: gzip' …/api/docs/tree | wc -c` beats the identity size by an order of magnitude on any corpus above the floor; `Vary: Accept-Encoding` is present.
|
||||
- [ ] `POST /api/chat` responses carry NO `Content-Encoding` and stream frame-by-frame (test-proven), and `/api/chat-images` still compresses.
|
||||
- [ ] `BOR_COMPRESS_MIN_BYTES=0` removes the middleware entirely; `-1` refuses startup naming the field.
|
||||
- [ ] A20 headers, A19 page/asset caching, and the `/api/*` pass-through are unchanged; `uv run pytest` green; ruff + pyright clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 02 — Depth-limited tree + server totals: `truncated`, the recursive max-date map, `KbTreeTotals` (phase 167 D2, LOCKED A2)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET …/api/docs/tree loads 2 MB in production which causes long loading times."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The pure builder can stop at a depth budget: a node whose children were not built says so (`truncated: true`) while keeping every field the Sources view renders (recursive count, recursive `updated_at`, its own `summary`, `summary_pending`, the image affordance), the response gains the `totals` the stat cards need, and `GET /api/docs/tree` defaults to `depth=1`.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — `KbTreeTotals(BaseModel)`: `documents: int = Field(ge=0)`, `chunks: int = Field(ge=0)`, `last_indexed: str | None = None` (docstring: the catalogue-wide totals the RAG view's three stat cards read, because a depth-limited tree cannot be summed client-side — the phase-97 `treeStats()` walk needed the whole tree).
|
||||
2. `app/schemas.py` — `KbTree` gains `totals: KbTreeTotals = KbTreeTotals()` (docstring: additive top-level key, TODO L5) and `KbTreeFolder` / `KbTreeSource` gain `truncated: bool = False` with an OMIT-WHEN-FALSE serializer. Follow the file's existing omission idiom (`_drop_image_fields` + `@model_serializer(mode="wrap")`): add the `truncated` key only when `True`, so an untruncated node's wire shape is byte-identical to today (every existing wire pin stays green). `KbTreeFile` never truncates (a file has no children).
|
||||
3. `app/api/docs.py::_folder_counts` — extend the return to `(folders, counts, max_dates)` where `max_dates[folder]` is the folder's RECURSIVE MAX document `created_at` (the same ISO-8601 strings, so the lexicographic max is the chronological one — the phase-106 D9 rule), computed from the rows in the same pass. Rationale in the docstring: a truncated node has NO built children, so `_subtree_max(children)` would read `None` and silently lose the D9 column; the aggregate map is the truncation-proof source. Keep `_subtree_max` for the expanded path UNCHANGED and add a unit test asserting the two agree for every expanded folder (one concept, two derivations, cross-pinned).
|
||||
4. `app/api/docs.py::_level_children` — add `depth: int | None`; per subfolder: when `depth is None` or `depth > 1`, build the node as today with the recursion at `depth - 1`; when the budget is exhausted, build `KbTreeFolder(path=sub, documents=counts[sub], updated_at=max_dates[sub], summary=summaries.get((source, sub)), summary_pending=<unchanged rule>, children=[], truncated=True)`. Update the docstring with the depth rule + the truncated-node field set (the level row renders a subfolder's name, count, date and description WITHOUT its contents).
|
||||
5. `app/api/docs.py::build_kb_tree` — thread `depth` (`None` = unlimited: every existing unit call and every existing expectation is byte-identical); `_source_node` passes the budget down and never marks the SOURCE node itself truncated at `depth ≥ 1` (the source's direct children are always built — the top level of the RAG view is source → level 1).
|
||||
6. `app/api/docs.py::list_kb_tree` — `depth: int = Query(1, ge=1, le=8)` (docstring: 1 = sources + one level, the view's first paint; 8 = the practical KB ceiling; the endpoint computes `totals` from the catalogue rows it already fetched: `documents = len(rows)`, `chunks = sum(chunks)`, `last_indexed = max(indexed_at)` → `None` for an empty KB) and passes it to the builder. `GET /api/docs` and the agent's `ls` path are untouched (pin it: `app/rag/agent.py` imports are unchanged).
|
||||
7. `tests/unit/test_kb_tree_builder.py` (extend): depth `None` vs `1` vs `2` vs deep on the multi-level fixture — the truncated node's exact field set (count recursive, `updated_at` = the deep max, summary present, `summary_pending` present, `children == []`, `truncated is True`), the totals math incl. the empty KB (`last_indexed is None`), the `_subtree_max`/`max_dates` cross-check on every expanded folder, and the phase-94/98 invariants (existence rule, `missing_folder_summaries` cross-check) at each depth.
|
||||
8. `tests/integration/test_docs_api.py` (extend): `GET /api/docs/tree` default → depth 1 (deep children absent, `truncated` present on a subfolder that has them, `totals` correct vs `GET /api/docs`), `?depth=3` → deeper, `?depth=0` / `?depth=9` / `?depth=abc` → 422, anonymous → 403 unchanged, and the wire-shape pin: an UNtruncated node's JSON keys are exactly the pre-phase set (no `truncated` key).
|
||||
9. Migrate the TWO date suites that consume the tree with no depth param (they break on the new default — the phase changes the default, so it owns the repair; every assertion stays, only the URL gains `?depth=8`):
|
||||
- `tests/integration/test_docs_api_dates.py` (~L157): `admin_client.get("/api/docs/tree")` → `"/api/docs/tree?depth=8"`, keeping the whole `a` → `a/b` → deeper D9 walk (deep date beats shallow at every level) and the 0-document-source `updated_at is None` case; add a one-line comment naming phase 167 (the default is the view's first paint, `depth=8` is the whole tree).
|
||||
- `tests/e2e/test_git_source_dates.py` (~L419): the `page.request.get(f"{app_url}/api/docs/tree")` call gains `?depth=8` so `_find_file` (which recurses by design) still reaches `old/…` and `recent/…`; the folder `updated_at` date assertions stay verbatim.
|
||||
- Add the truncation-proof counterpart so the guarantee is tested BOTH ways: at the DEFAULT depth, a truncated folder node's `updated_at` still equals the deep max (this is exactly what the task-02 `max_dates` map buys — the date contract survives the payload diet), pinned in `test_docs_api_dates.py` and in `tests/unit/test_kb_tree_builder.py`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit + integration above; `uv run pytest tests/unit/test_kb_tree_builder.py tests/integration/test_docs_api.py -v` green.
|
||||
- Coverage **>90%** on the modified `app/api/docs.py` / `app/schemas.py` lines.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Default `/api/docs/tree` returns sources + ONE level with correct `totals`, and every node that has unloaded children carries `truncated: true`.
|
||||
- [ ] A truncated node still carries its recursive count, its recursive `updated_at`, its `summary`, and `summary_pending` — phase 98/106 semantics intact at every depth (unit + integration).
|
||||
- [ ] `depth=None` (builder default) reproduces today's tree byte-for-byte; untruncated nodes' wire shape is unchanged.
|
||||
- [ ] `uv run pytest` green; coverage > 90%; ruff + pyright clean.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 03 — The subtree route and revalidation: `GET /api/docs/tree/node`, `app/core/http_cache.py`, ETag/304 (phase 167 D3, LOCKED A3/A4)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "Fix that any any other endpoints that could load a ton of data."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
One small admin-gated route serves a single tree node's direct children (the drill's fetch target), and the catalogue read routes (`/api/docs/tree`, `/api/docs/tree/node`, `/api/docs`) carry a content-exact weak `ETag` + `Cache-Control: private, must-revalidate` so a repeat load costs a 304 with no body — without touching `sources_meta` (which would mark saved chats stale) or the SSE/mutating surface.
|
||||
|
||||
## Work
|
||||
1. New `app/core/http_cache.py` (small, total, no I/O):
|
||||
- `weak_etag(body: str | bytes) -> str` → `W/"<sha256 hexdigest[:32]>"` (docstring: content-exact by construction; 32 hex chars is the file's truncation convention and collisions are not a correctness risk for a revalidated cache — a false match only ever costs one stale repaint on a 2 MB namespace, and the tag is over the EXACT served bytes);
|
||||
- `etag_matches(header: str | None, tag: str) -> bool` → tolerant of `*`, a comma-separated list, and weak/strong equivalence (`W/"x"` matches `"x"` both directions), whitespace-tolerant, `None`/empty → `False`;
|
||||
- `revalidate_headers(etag: str) -> dict[str, str]` → `{"ETag": etag, "Cache-Control": "private, must-revalidate"}` (ONE place for the pair; docstring names A19's scope — HTML pages stay `no-cache` with no validators, `/assets/*` stay immutable, this is the read-only API arm).
|
||||
2. `app/api/docs.py` — `GET /api/docs/tree/node`:
|
||||
- signature: `source: str = Query(...)`, `path: str = Query("")` (source-relative; `""` = the source root node), `depth: int = Query(1, ge=1, le=8)`, `db=Depends(get_db)`, `_admin=Depends(require_admin)` (`403 {"detail": "admin only"}` — the same gate as `/api/docs/tree`), `request: Request`;
|
||||
- composition: the SAME catalogue query as `list_kb_tree` (one query, no N+1 — the file's docstring paragraph says so), the same `folder_summaries` + image-docs bounded selects, then build the single node through the pure helpers: resolve the source (registry name or indexed source), `_folder_counts(rows)` for that source, `_level_children(source, path, folders, counts, rows, summaries, images, depth=depth)`; the node is a `KbTreeSource` when `path == ""` else a `KbTreeFolder` with `documents=counts[path]`, `updated_at=max_dates[path]`, `summary`, `summary_pending` (the phase-98 D3 rule), `truncated` per the depth budget;
|
||||
- 404 `{"detail": "unknown node"}` when: the source is neither registered nor indexed, OR the folder fails the phase-94 existence rule (`path` not in `folders` and `path != ""`). Never a filesystem path built from `path` (docstring: the row-only rule, the phase-10 traversal argument reused);
|
||||
- `response_model=None` on the decorator with the union return annotated, and the body returned as a raw `Response` (below) so the serializer is not run twice.
|
||||
3. `app/api/docs.py` — ETag wiring on the three read routes (`/api/docs/tree`, `/api/docs/tree/node`, `/api/docs`): build the model → `body = model.model_dump_json()` (for `/api/docs`: `DocList.model_dump_json()`) → `etag = weak_etag(body)` → `if etag_matches(request.headers.get("if-none-match"), etag): return Response(status_code=304, headers=revalidate_headers(etag))` → else `Response(content=body, media_type="application/json", headers=revalidate_headers(etag))`. Docstring per route: the client is `fetch` (a 304 is served from its cache transparently), and `must-revalidate` guarantees a sync or an admin edit is seen on the next load — the tag changes because the body changes. Keep `response_model=KbTree`/`DocList` on the decorators for the OpenAPI schema (a returned `Response` short-circuits serialization — note that in the docstring).
|
||||
- ASSUMPTION: `GET /api/documents/content` is NOT given a validator this phase (one document's body, on demand, already compressed; the audit in task 05 decides whether it needs more) — recorded so the scope is explicit.
|
||||
4. `tests/integration/test_docs_api.py` (extend):
|
||||
- the node route: 200 for a source root (`path=""`, the `KbTreeSource` shape), 200 for a nested folder (correct `documents`/`updated_at`/`summary`/`summary_pending`/`children` at `depth=1` with `truncated` on the deeper subfolders), `?depth=2` deeper, 404 `unknown node` for an unknown source / a nonexistent folder / a `path` that is a FILE's path, 403 anonymous, and a cross-check that the node's children equal the same level of the unlimited `/api/docs/tree?depth=8` response (ONE concept, two routes, pinned equal);
|
||||
- ETag matrix on all three routes: first GET → 200 + `ETag` + `Cache-Control: private, must-revalidate`; re-send with `If-None-Match` → 304 + empty body + both headers; a wrong tag → 200; a `W/`-vs-strong mismatch pair → 304; `*` → 304; after `PATCH /api/folders/summary` (and after `PATCH /api/documents/date`) the tag CHANGES and the next GET is a 200 with the new tree (the A4 proof — a version-derived tag would have failed exactly here);
|
||||
- gzip coexistence: a gzipped 200 still carries the `ETag` (the header survives the compression layer) and a conditional request returns a body-less 304.
|
||||
5. `tests/unit/test_http_cache.py` (new): `weak_etag` stability/length/prefix (`W/"…"`), bytes-vs-str equality, `etag_matches` for `None`, empty, `*`, single, comma-list, weak↔strong both directions, whitespace, and a tag that merely shares a prefix → `False`.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration + unit above; `uv run pytest tests/integration/test_docs_api.py tests/unit/test_http_cache.py -v` green; `tests/integration/test_caching.py` stays green (the `/api/*` pass-through and the HTML rewrite are untouched by a route-level header).
|
||||
- Coverage **>90%** on `app/core/http_cache.py` and the new route lines.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/docs/tree/node?source=X&path=Y` returns exactly that folder's direct children with the same per-node fields the full tree produces (equality pinned against `depth=8`).
|
||||
- [ ] A repeat request with `If-None-Match` answers **304, no body**; a folder-summary or document-date edit changes the tag and forces the next 200.
|
||||
- [ ] Anonymous gets `403 {"detail":"admin only"}`; an unknown source/folder gets `404 {"detail":"unknown node"}`; nothing builds a filesystem path from `path`.
|
||||
- [ ] `uv run pytest` green; coverage > 90%; ruff + pyright clean.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 04 — The client drills lazily: level cache, per-level fetch, §7.4 drill states, `totals` stat cards (phase 167 D4, LOCKED A3)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET …/api/docs/tree loads 2 MB in production which causes long loading times."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The RAG view loads one small tree, fetches a folder's contents the first time it is opened, caches it, keeps the breadcrumb/scroll-panel/editing behavior identical, and never shows a stale or half-updated level — the UX cost of A3 paid in full so the user cannot tell the tree is no longer one payload.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/sources.js::loadTree()` (~L1625) — `fetch("/api/docs/tree?depth=1")`; on success: store the tree, CLEAR `levelCache` (a refresh invalidates every drilled level — the sync/refresh call sites at ~L1258, ~L1386, ~L2192, ~L2213 keep working unchanged), then `resetVanishedLocation()` + `renderLevel()` as today. Keep the `loadSeq` race token and the `renderEmpty()` failure path.
|
||||
2. `frontend/assets/sources.js` — new module state + the drill:
|
||||
- `const levelCache = new Map();` keyed `source + "/" + folder` (a node's `path`), populated ONLY by a node fetch;
|
||||
- `function needsLevelFetch(node) { return node.truncated === true && !levelCache.has(key); }`;
|
||||
- `enterFolder(source, folder)` (the folder-row click path, today a pure `current = …; renderLevel()`): when the target needs a fetch → set the in-flight state (`drillPending = {source, folder}`, the row link gets `aria-busy="true"` + `aria-disabled="true"` so a double click cannot start a second fetch), `await fetch("/api/docs/tree/node?source=…&path=…")`, merge the result (`Object.assign(targetNode, node)` + `levelCache.set(key, true)`), clear the flags, then `current = …; renderLevel()`. A stale response (token changed) never paints (the existing `my === loadSeq` convention extended to the drill).
|
||||
- In-flight rendering: render the level the user is LEAVING unchanged and show one loading row in the folders panel using the file's existing status/empty idiom (a `role="status"` `.kb-level-note`-style line "Loading <folder>…" — reuse an existing class if one fits, otherwise add ONE `.kb-level-loading` rule to `styles.css` with tokens only, no animation beyond the existing house spinner if one exists);
|
||||
- failure (non-OK or a network throw): `role="alert"` line "Couldn't load <folder> — Retry." with a real `<button type="button">Retry</button>` that re-runs the same fetch, and the CURRENT level stays rendered (never blank, never a stale breadcrumb — PLAN §7.4 "every failure path re-enables its controls").
|
||||
3. `frontend/assets/sources.js` — the existence rules stop walking a full tree:
|
||||
- `folderExistsIn(src, folderPath)` (~L1667) is replaced by `folderKnownIn(src, folderPath)`: true when the path's whole ancestor chain is present in what the client holds (the loaded tree + cached levels), so `resetVanishedLocation()` can still answer after a re-fetch WITHOUT assuming a full tree; when a drill fetch answers `404` the folder is gone → `current = { source: null, folder: null }` + `renderLevel()` (the honest reset, the phase-97 "never stale" contract carried over), and the failing `levelCache` entry is not cached;
|
||||
- `currentLevelNode()` (~L1680) keeps walking, but through cached nodes (unchanged semantics for loaded levels).
|
||||
4. `frontend/assets/sources.js::treeStats()` (~L1703) — read `kbTree.totals` (`documents`, `chunks`, `last_indexed`) when present, and keep the existing recursive walk ONLY as the defensive fallback when `totals` is absent (a tree served by a pre-phase process); the three stat cards (`statDocs`, `statChunks`, `statLast`) render identically at every level (pin the exact formatting call — `fmtDate(st.last)` — unchanged).
|
||||
5. `frontend/assets/sources.js` docstring (the module header, ~L55-103) — rewrite the "ONE fetch of GET /api/docs/tree returns the whole tree" paragraph for the depth-1 + per-level model, citing TODO L5 and phase 167, and keep the phase-79 race-token, phase-90 upload and phase-98 pending narratives accurate.
|
||||
6. `tests/unit/test_kb_tree_ui.py` (MIGRATE — it currently asserts `js.count('fetch("/api/docs/tree")') == 1` at ~L250 and the "ONE fetch" contract at ~L9/L27): the fetch assertion becomes `fetch("/api/docs/tree?depth=1")` exactly once + `fetch("/api/docs/tree/node?` exactly once, and the "drills client-side" docstring assertions become the level-cache assertions: `levelCache` declared once, cleared inside `loadTree`, consulted by `needsLevelFetch`, written only after a 200; the 404 → reset branch exists; `drillPending` guards a second click; the Retry button is a real `<button>`; `treeStats` prefers `totals`. Preserve every OTHER phase-97/99 pin (breadcrumb, sticky header, scroll panel, editor reset, empty state).
|
||||
7. `tests/unit/test_frontend_router.py:591` — the shell-route pin (`'fetch("/api/docs/tree")' in body`) moves to the `?depth=1` URL, keeping its original intent (the Sources view loads the catalogue from the tree endpoint, not the flat one).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the migrated + new pins above; run the owning set for collateral: `uv run pytest tests/unit -k "kb_tree or frontend_router or table_ux or source" -v`.
|
||||
- No `app/` change → coverage gate unmoved (> 90% on `app/` stays green).
|
||||
- Frontend house rules hold: `createElement` + `textContent` only (no HTML strings), `aria-busy`/`aria-disabled` on the in-flight link, `role="status"`/`role="alert"` for the two states, `:focus-visible` on the new Retry button via the global rule, tokens-only CSS if a new class is added.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] First paint = ONE `depth=1` request; the second visit to a folder is free (cache hit, zero network — unit-pinned).
|
||||
- [ ] A drilled level shows correct rows (counts, dates, descriptions, pending markers, image thumbnails) inside the phase-156 bounded panel with the sticky header intact.
|
||||
- [ ] In-flight = `aria-busy` + one status line; failure = `role="alert"` + working Retry with the current level still rendered; a 404 resets to the top level with no stale breadcrumb; a stale response never paints.
|
||||
- [ ] The three stat cards match the server `totals` at every level, and the pre-phase fallback path exists.
|
||||
- [ ] Full `uv run pytest` green; ruff + pyright clean.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 05 — The byte budget: `/api/*` response-size log line + `scripts/api_payload_audit.py` + `AUDIT.md` (phase 167 D5, LOCKED A5)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "Fix that **any any other endpoints that could load a ton of data**."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The app can answer "which endpoint is heavy, right now, in production?" from its own logs, and the owner's "any other endpoints" gets a measured, written answer: every candidate endpoint's uncompressed and gzipped size against a stated budget, with each offender either fixed or explicitly accepted.
|
||||
|
||||
## Work
|
||||
1. New `app/core/response_budget.py` — **pure ASGI** (no buffering, SSE-safe):
|
||||
- wraps `send`, counting the bytes of every `http.response.body` message's `body` chunk, and `receive`-independent timing (`time.monotonic()` at entry → at `more_body=False` completion);
|
||||
- logs ONE §9-style line per `/api/*` response on the `http.response.body` completion (for a stream that is when the stream ends — say so in the docstring):
|
||||
`api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41`
|
||||
(`enc` = the response's `content-encoding` or `identity`; add ` stream=1` for `media_type == "text/event-stream"` responses, where `bytes` is the streamed total and `duration_ms` is the stream's life);
|
||||
- level: INFO when `bytes >= settings.log_large_response_bytes`, DEBUG otherwise; non-`/api/` paths and non-HTTP scopes pass through silently; an exception in the wrapped app still logs (finally-style, `bytes` as counted) and re-raises unchanged;
|
||||
- `304`s log `bytes=0` (the win this phase bought is visible in the log, not hidden).
|
||||
2. `app/config.py` — `log_large_response_bytes: int = 65536` (`BOR_LOG_LARGE_RESPONSE_BYTES`) with the `agent_max_rounds` validator (negative → `ValueError` naming the field; `0` = log EVERY `/api/*` response at INFO). `.env.example` entry with the three-value comment.
|
||||
3. `app/main.py` — `app.add_middleware(ResponseBudgetMiddleware)` **before** the gzip `add_middleware` call, so the stack is SecurityHeaders (outermost) → gzip → response-budget → caching → session → routes. Consequence to state in the comment: `bytes=` is the bytes the ROUTE produced (uncompressed) while `enc=` reports what went on the wire — the two together answer "how fat is it, and how fat did the client see it".
|
||||
4. New `scripts/api_payload_audit.py` (`python -m scripts.api_payload_audit`), stdlib + `httpx` only, `app.core.debugging` conditional import preserved if it touches boot (AGENTS.md rule 7 — it does not import the app factory, so just follow the other scripts' shape, see `scripts/eval_retrieval.py`):
|
||||
- args: `--base-url` (default `http://localhost:8000`), `--username`/`--password` (or `BOR_ADMIN_*` env reads, never hardcoded), `--budget` (default `262144`), `--path-tree` etc. only if needed;
|
||||
- logs in (the real `/api/login` session cookie), then for each candidate: `GET /api/docs/tree?depth=1`, `/api/docs/tree?depth=8`, `/api/docs/tree/node?source=<first>&path=<first folder>`, `GET /api/docs`, `GET /api/documents/content` for the LARGEST document (ask `/api/docs` then pick by chunks) — its `content` length is the size, `GET /api/chats`, `GET /api/chats/{id}` for the largest chat, `/api/git-sources`, `/api/suggestions`, `/api/sync/status`, `/api/tokens`, `/api/config`, `/api/whoami` — measuring BOTH `identity` and `gzip` request variants (explicit `Accept-Encoding`) via `response.content` length + wall time, plus one conditional re-request for the ETag routes (expect 304/0 bytes);
|
||||
- prints a table (endpoint, identity bytes, gzip bytes, ratio, ms, verdict) where the verdict is `ok` / `LOOK (> budget)` / `304`, exits 0 (it is a report, not a gate) — and prints the corpus it measured (document count from `/api/docs`) so numbers are interpretable.
|
||||
5. `.agents/phases/todo/167_api_payload_diet/AUDIT.md` (new, committed with the phase) — the measured record: how it was run (the local DB restored to the **1,000-document live-replica snapshot** via the `restore-test-db` skill, so the order of magnitude matches production), the full table before (uncompressed, no middleware) and after (depth-1 + gzip + ETag), and one verdict line per endpoint: **fixed** (tree: 2 MB → X KB, second load 304) / **accepted with why** (`/api/documents/content` — one document on demand, compressed; `/api/chats/{id}` — the owner opening one saved chat; `/api/sync/status` — tiny but polled, so the log line is what matters) / **follow-up** (anything still over `--budget`, with the proposed remedy named). Do not hand-write numbers: paste the script's output.
|
||||
6. Remedies in scope if the audit shows an offender (apply, don't just note): add the task-03 ETag pair to any other read route that is large and re-fetched (e.g. `/api/git-sources` if it grows past the budget); raise nothing, reject nothing (LOCKED A5).
|
||||
7. `README.md` — an "Operational notes" subsection: the log line format + how to read it (`grep ' :: api ' `), `BOR_LOG_LARGE_RESPONSE_BYTES`, running the audit script against a restored replica, and the deploy note that the app gzips itself (so nginx `gzip on` is optional, and `proxy_set_header Accept-Encoding` passthrough is fine).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_response_budget.py`, new): the line's exact shape for a large 200 (INFO), a small 200 (DEBUG), a 304 (`bytes=0`), a non-`/api/` path (nothing logged), a streaming response (`stream=1`, bytes = sum of chunks, logs on completion), the `enc=gzip` vs `enc=identity` reading, an app exception (logged + re-raised), and the config validator (`0` → INFO for everything, negative → refuses startup).
|
||||
- Script smoke: `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` against a running dev app (document count + at least the tree rows printed; the script must fail loudly with a clear message when the app is unreachable or the login is rejected).
|
||||
- Coverage **>90%** on `app/core/response_budget.py`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] One parseable `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per `/api/*` response at the configured level, with the SSE stream logged once at stream end and never buffered.
|
||||
- [ ] `AUDIT.md` exists in this phase directory with real numbers (before/after), the corpus it measured, and a verdict per candidate endpoint.
|
||||
- [ ] Every endpoint over the stated budget is either fixed in this phase or carries a named follow-up with its reason.
|
||||
- [ ] README documents the log line, the env var, the audit script, and the self-gzip deploy note; `uv run pytest` green; ruff + pyright clean.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 06 — `tests/e2e/test_kb_tree_lazy_load.py`, the migrated pins, the full gates (phase 167 D6)
|
||||
|
||||
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
An isolated Playwright suite proves the owner-visible outcome — a small first Sources load, correct content when drilling, no refetch, an honest reset when a folder disappears, gzip on the wire, a 304 on the second visit — and every pin the superseded phase-97 contract owned (D3/A3) is migrated rather than deleted. Then the phase gates close it out.
|
||||
|
||||
## Work
|
||||
1. New `tests/e2e/test_kb_tree_lazy_load.py` (module-scoped `app_server`/`app_url`; seed a multi-level KB directly through the DB the way `tests/e2e/test_kb_tree.py` does — ≥ 3 sources, a 3-deep folder chain, one folder with a stored `folder_summaries` row, one pending folder, and one image doc if `BOR_IMAGES=true` is cheap in this suite's env; read `tests/e2e/test_kb_tree.py` and `tests/e2e/test_kb_tree_nav.py` for the seeding + drill helpers first):
|
||||
- **the first load is small**: collect the network (`page.on("response")`) for the Sources view boot → exactly one `/api/docs/tree?depth=1` request; its JSON has `totals` with the expected `documents`/`chunks`, its top-level source nodes carry `documents` counts that include the deep files (recursive), and NO node deeper than level 1 exists while level-1 subfolders report `truncated === true` (assert on the parsed body, not on bytes);
|
||||
- **the stat cards** match `totals` at the top level AND after drilling two levels (the numbers never change — they are catalogue-wide);
|
||||
- **the drill**: click into a level-1 folder → exactly one `/api/docs/tree/node?source=…&path=…` request; rows render (subfolders sorted, then files in catalog order) with the stored description in the level block, the pending marker on the pending folder, and the phase-156 sticky header/scroll panel intact; click into a deeper folder → a second node fetch; go BACK to the first folder → **zero** new requests (cache);
|
||||
- **the revalidation**: reload the view → the browser re-sends `If-None-Match` (observe the request header) and the response is `304` with an empty body while the table still renders (the `page.route`/`expect_response` idiom);
|
||||
- **gzip**: assert the tree response's `Content-Encoding` is `gzip` and `Vary` includes `accept-encoding` (Chromium sends `Accept-Encoding: gzip, deflate, br` by default — pin what actually arrives, and if the harness negotiates identity, assert the server side in the integration suite instead and say so in the test docstring);
|
||||
- **the vanished folder** (§7.4): a `page.route` stub that answers the node fetch with `404 {"detail":"unknown node"}` → the alert line + Retry button appear, the level the user was on stays rendered; click a parent crumb → the level is fine; then let the real route answer and Retry → the level loads;
|
||||
- **in-flight**: a `page.route` handler that delays the node response → the clicked link is `aria-busy`/`aria-disabled` and the status line shows, and a second click does NOT issue a second request.
|
||||
2. `tests/e2e/test_kb_tree.py` (MIGRATE, do not delete): the suite's "ONE fetch of `/api/docs/tree`, drill is client-side" assertions become the depth-1 + per-level assertions (the file's outline comment block first, then each assertion); every OTHER claim it owns (breadcrumb text, back navigation, table columns, dates, summary editor round-trip, pending markers, empty state, the anonymous gate) stays, re-aimed at the level the user is on. Each edit is a re-point, never a removal — state that in the commit body.
|
||||
3. `tests/e2e/test_navbar_refresh.py:115` — the `("#nav-sources", "/sources.html", "/api/docs/tree")` expectation matches the new URL (keep the substring `/api/docs/tree` if the assertion is a substring test — verify, then pin precisely); re-run that suite.
|
||||
4. `tests/e2e/test_kb_tree_nav.py`, `test_document_dates.py`, `test_git_source_dates.py`, `test_sync_summary_visibility.py`, `test_table_ux_polish.py`, `test_sources_midstream_bug.py`, `test_ls_tree_drilldown.py` (the agent's `ls` must be untouched) — run in isolation, repair ONLY where a condition assumed the full tree loaded (e.g. a test that expects a deep folder's rows before drilling — add the drill step; a test that walks the tree JSON — give it `?depth=8`, the repair task 02 item 9 defines), never by weakening an assertion.
|
||||
5. Final gates + docs, in one pass:
|
||||
- `uv run pytest` (unit + integration) green;
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → `app/` > 90% (report the number in the commit body);
|
||||
- `uv run ruff check . && uv run pyright` clean;
|
||||
- `uv run pytest tests/e2e/test_kb_tree_lazy_load.py -v --no-cov` green in isolation + the six regression suites in isolation + `tests/e2e/test_smoke.py`;
|
||||
- `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` unchanged (retrieval is untouched — record it);
|
||||
- `AUDIT.md` (task 05) reflects the FINAL state after these fixes; `README.md` + `AGENTS.md` reviewed in full once and every inconsistency from this phase fixed (the tree endpoint table, the env var list, the deploy note about self-gzip);
|
||||
- UI Structure Check (AGENTS.md rule 5) for the drill: the level stays inside the 72rem container, the bounded scroll panel + sticky header survive a lazy render, the loading/alert lines are `role="status"`/`role="alert"`, the Retry button meets 44px + `:focus-visible`, contrast pairs are the existing AA tokens.
|
||||
6. Move `.agents/phases/todo/167_api_payload_diet/` → `.agents/phases/complete/` (with `AUDIT.md`) and commit with the `00_phase.md` message, `--no-gpg-sign`, one atomic commit (AGENTS.md rule 8).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the new suite green in isolation + the six migrated/regression suites + `test_smoke.py` in isolation.
|
||||
- Coverage **>90%** on `app/` including `compression.py`, `http_cache.py`, `response_budget.py` and the new route/builder lines.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Boot = one `depth=1` request whose body has `totals` and no deep children; drill = one node fetch per uncached level, zero on a cache hit; reload = 304.
|
||||
- [ ] Every level renders the same data the pre-phase full tree rendered (counts, dates, descriptions, pending markers, thumbnails) — proven by the migrated `test_kb_tree.py` claims, none dropped.
|
||||
- [ ] Failure paths are honest and recoverable (alert + Retry, level preserved, vanished folder resets the breadcrumb), and the agent's `ls` output is unchanged.
|
||||
- [ ] The measured numbers are in `AUDIT.md`, the coverage number is in the commit body, `uv run pytest` green, ruff + pyright clean, one atomic `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||
Reference in New Issue
Block a user