phase: 167_api_payload_diet

All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
2026-10-08 21:23:49 -04:00
parent ab9391d019
commit 2ac37455f0
73 changed files with 7313 additions and 228 deletions
@@ -0,0 +1,236 @@
 # Phase 167 · Task 05 — Measured API payload audit (the byte budget)
 Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any
 other endpoints** that could load a ton of data." This file is the measured answer —
 every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` /
 server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a
 report, never a 413 — nothing was rejected, nothing was capped.
 ## How it was run
 | | |
 |---|---|
 | Date | 2026-10-08 |
 | Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) |
 | App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) |
 | BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had |
 | AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) |
 | Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) |
 | Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below |
 The script is **read-only**: it signs in through `POST /api/login` (the session
 cookie — the single-admin login is password-only, PLAN A10, so there is no
 `--username`) and GETs. It never mutates, so it is safe against a production target
 (`--base-url https://…`).
 ### Corpus caveat (read this before comparing with the 2 MB report)
 The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents**
 (verified: `select count(*) from folder_summaries` → `0`,
 `select count(*) from documents where is_image` → `0`). The production 2 MB payload
 was 1,000 documents **plus** the hundreds of inlined AI folder descriptions
 (phase 127/155) and the image documents' full vision descriptions (phase 122/124) —
 none of which exist in the snapshot. **So the identity bytes below are a lower bound
 on the production payload**, and what transfers from this measurement is the
 *structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the
 ONE level the RAG view renders instead of the whole recursion below it — and every
 level of a summary-carrying folder is where the production payload's fat lives),
 gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1
 tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict
 table below names the live re-run as the follow-up.
 ## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available
 > Read the BEFORE table's `gzip` column as "no compressor existed": ratio
 > `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts
 > are NOT a pre-phase property — the conditional re-requests hit the validators
 > tasks 01-04 added, which this run left in place because only the transport was
 > being switched off (the honest way to measure "the same bytes, uncompressed").
 > Pre-phase there were no validators at all, so every repeat load cost the
 > identity column again.
 ```
 api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
 api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
  endpoint                                                                identity        gzip  ratio      ms  verdict
  ----------------------------------------------------------------------------------------------------------------
  tree depth=1 (default)                                                     8,474       8,474  1.000      12  304
  tree depth=8 (full)                                                      265,147     265,147  1.000      37  304
  tree node homelab/active                                                  20,844      20,844  1.000      11  304
  docs catalogue                                                           246,366     246,366  1.000      14  304
  document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json)     892,124     892,124  1.000       8  LOOK (> 262144)
      note: content field 772573 chars
  saved chats list                                                              12          12  1.000       1  ok
  saved chat (one)                                                               —           —      —       —  n/a
      note: no saved chats
  git sources                                                                  188         188  1.000       1  ok
  suggestions                                                                   74          74  1.000       1  ok
  sync status (polled)                                                         272         272  1.000       1  ok
  tokens                                                                        13          13  1.000       2  ok
  config                                                                       188         188  1.000       1  ok
  whoami                                                                        37          37  1.000       1  ok
 api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
 ```
 What the pre-phase browser actually paid for the **Sources page first paint** is the
 sum of the rows it fetches on load, and the pre-phase tree route answered the FULL
 recursive tree by default (`tree depth=8 (full)` is that shape):
 `265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app
 published **no validators at all**, so every repeat load re-paid the same 512 KB.
 ## AFTER — shipped defaults (depth-1 default + gzip + ETag)
 api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
 api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
  endpoint                                                                identity        gzip  ratio      ms  verdict
  ----------------------------------------------------------------------------------------------------------------
  tree depth=1 (default)                                                     8,474       1,129  0.133      13  304
  tree depth=8 (full)                                                      265,147      23,520  0.089      41  304
  tree node homelab/active                                                  20,844       1,064  0.051      11  304
  docs catalogue                                                           246,366      45,107  0.183      23  304
  document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json)     892,124      60,776  0.068      14  LOOK (> 262144)
      note: content field 772573 chars
  saved chats list                                                              12          12  1.000       2  ok
  saved chat (one)                                                               —           —      —       —  n/a
      note: no saved chats
  git sources                                                                  188         188  1.000       1  ok
  suggestions                                                                   74          74  1.000       2  ok
  sync status (polled)                                                         272         272  1.000       1  ok
  tokens                                                                        13          13  1.000       2  ok
  config                                                                       188         188  1.000       1  ok
  whoami                                                                        37          37  1.000       1  ok
 api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
 ```
 Sources-page first paint, same sum against the AFTER gzip column:
 `1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity,
 and every one of those five rows revalidates to **304 / 0 bytes** on the next load).
 The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×**
 (31× from the depth limit × 7.5× from gzip), and the second visit is free.
 ## `curl` proof on the audited corpus
 ```
 $ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree
  etag: W/"53c86d5d554efd8487ad52ca6cd6f913"
  cache-control: private, must-revalidate
  content-length: 8474
  vary: Cookie, Accept-Encoding
  8474 bytes                                     # wc -c b1.json
 $ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree
  etag: W/"53c86d5d554efd8487ad52ca6cd6f913"     # the validator survives the encoding
  content-encoding: gzip
  1129 bytes                                     # wc -c b2.gz
 $ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \
       -w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree
 status=304 size=0
 ```
 ## The byte-budget log line (the answer to "which endpoint is heavy, right now")
 Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO);
 the shipped default is `65536`, so at rest only the big ones reach the log.
 `grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what
 went on the wire — the two together are the "how fat, and how fat did the client see
 it" pair.
 ```
 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14
 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
 INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11
 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34
 INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40
 INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10
 INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22
 INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12
 INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0
 INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0
 ```
 `POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded
 payload to budget — and it is skipped by the gzip layer by exact path (which is why
 its line reads `enc=identity`). One real turn against this corpus, measured:
 ```
 INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1
 ```
 39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture
 ended because the measuring `curl --max-time 45` hung up — which is itself the
 other half of the contract: a stream the client walks away from still writes
 exactly ONE line, with the bytes that actually got out, never one line per frame
 and never a missing line). Pinned by
 `tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out`
 and by the integration pin
 `tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`.
 ## Verdict per candidate endpoint
 | Endpoint | Measured (identity / wire) | Verdict |
 |---|---|---|
 | `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. |
 | `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). |
 | `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. |
 | `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). |
 | `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). |
 | `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. |
 | `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. |
 | `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). |
 | `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). |
 | `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. |
 | `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). |
 | `GET /api/config` | 188 | **ok** — display strings only. |
 | `GET /api/whoami` | 37 | **ok** — two fields. |
 | `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. |
 ## What the audit changed, and what it proved
 * **No new remedy was needed beyond tasks 01-04.** The only over-budget row
  (`/api/documents/content`) is the documented accepted case, and
  `/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag
  was **not** applied — a validator on a sub-860-byte response costs a hash
  and saves nothing (it would not even compress).
 * **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06)
  drives the real Sources view in Chromium and pins what this file measured: the
  boot is ONE `?depth=1` request whose body carries `totals` and no deep children,
  the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary`
  (Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the
  wire answers `304` for a fraction of the first transfer while the table still
  renders from the cache, and a level the payload only names costs exactly one
  `/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands
  `fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP
  Network events — the DevTools-panel truth, not Playwright's post-cache view.
 * **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on
  the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the
  second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat
  load free. Had the phase shipped gzip alone, the Sources page would still
  have paid 23 KB/request forever, and on the summary-inflated production
  corpus proportionally more.
 * **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the
  LOOK bar, already validated + compressed). The byte-budget line at its
  default `65536` threshold logs it at INFO on every uncached load, so the
  corpus growth that pushes it over shows up in the log before it shows up
  for the owner.
 * **Two limits of this measurement, stated instead of hidden:** (1) the
  snapshot has no folder summaries and no image docs, so the identity
  numbers are a lower bound on the production payload (see the caveat
  above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather
  than measured — the script is read-only by design (it must stay safe
  against the deployed app), so it does not invent a chat to size. Both are
  re-measurable with one command against the live corpus.
 ## Re-running this audit
 ```bash
 # 1. the corpus (destructive to the local DB — see the skill's own backup step)
 #    restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese
 uv run alembic upgrade head                      # the snapshot carries an old alembic version
 # 2. the app
 uv run uvicorn app.main:app --port 8000
 # 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override)
 uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
 # the BEFORE transport, for a before/after pair like the tables above:
 BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit
 ```
@@ -0,0 +1,22 @@
 All gates green. Final report:
 **Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
 - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
 - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.
 **Gates (all 100% green)**
 - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
 - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
 - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).
 **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
 - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
 - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
 - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.
 **Notable decisions / deviations**
 - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
 - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
 - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.
 **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,130 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 10%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 19%]
 ........................................................................ [ 21%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 28%]
 ........................................................................ [ 30%]
 ........................................................................ [ 32%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 37%]
 ........................................................................ [ 39%]
 ........................................................................ [ 41%]
 ........................................................................ [ 43%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 48%]
 ........................................................................ [ 50%]
 ........................................................................ [ 52%]
 ........................................................................ [ 54%]
 ........................................................................ [ 56%]
 ........................................................................ [ 57%]
 ........................................................................ [ 59%]
 ........................................................................ [ 61%]
 ........................................................................ [ 63%]
 ........................................................................ [ 65%]
 ........................................................................ [ 66%]
 ........................................................................ [ 68%]
 ........................................................................ [ 70%]
 ........................................................................ [ 72%]
 ........................................................................ [ 74%]
 ........................................................................ [ 75%]
 ........................................................................ [ 77%]
 ........................................................................ [ 79%]
 ........................................................................ [ 81%]
 ........................................................................ [ 83%]
 ........................................................................ [ 85%]
 ........................................................................ [ 86%]
 ........................................................................ [ 88%]
 ........................................................................ [ 90%]
 ........................................................................ [ 92%]
 ........................................................................ [ 94%]
 ........................................................................ [ 95%]
 ........................................................................ [ 97%]
 ........................................................................ [ 99%]
 ....................                                                     [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  234      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    280      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/http_cache.py            22      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/response_budget.py       63      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       73      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           6102     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,19 @@
 **Task 01 complete — all green.**
 **Implemented**
 - `app/core/compression.py`: `SelectiveGZipMiddleware` (pure ASGI, composes Starlette `GZipMiddleware`, `DEFAULT_SKIP_PATHS=("/api/chat",)` EXACT match — `/api/chat-images` still compresses) + `_RebufferSmallResponses` framing seam
 - `app/config.py`: `compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`) with `agent_max_rounds`-pattern validator (`0` = off, `-1` refuses startup naming the field)
 - `app/main.py`: wired between `configure_caching` and `SecurityHeadersMiddleware` (A20 stays outermost); `0` adds nothing at all; `.env.example` entry with the three-value comment
 - `tests/unit/test_compression.py` (19 tests: gzip/identity/floor/pre-encoded skip, SSE frame-by-frame raw-ASGI interleaving proof + `list(stream)` idiom, exact-match pin, lifespan/websocket pass-through, wiring order, kill switch); `tests/integration/test_docs_api.py` + 4 pins (tree gzip→decompress-equal, A20 headers on gzipped body, small `/api/health` identity, A19 page contract through gzip)
 **Gates**
 - `uv run pytest` → **3903 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **99%** overall, `app/core/compression.py` **100%**
 - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
 - E2E isolated (exit 0): `test_smoke`, `test_chat_rag`, `test_cache_busting`, `test_asset_cache_revalidation`, `test_kb_tree`
 - Measured: 50-doc tree 9,708 → 571 bytes (**17×**), `Vary: … Accept-Encoding` present
 **Docs**: README.md — gzip bullet in "Caching & deploys" + `BOR_COMPRESS_MIN_BYTES` config table row; AGENTS.md: no doc impact (no env/middleware content there).
 **Decisions**: (1) The `_RebufferSmallResponses` seam was required: starlette 1.x's `CachingMiddleware` (BaseHTTPMiddleware, below gzip per LOCKED order) re-frames every pass-through body (`more_body=True`), voiding Starlette's minimum-size exemption — `/api/health` came back gzip-inflated (33→71 B). The seam only re-delivers the buffered message shape Starlette's own exemption checks; no compression math forked. (2) Frame-by-frame proof is a deterministic raw-ASGI interleaving test (TestClient buffers bodies and cannot prove incremental arrival).
 **Next pending task**: `167_api_payload_diet/02_tree_depth_and_totals.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,127 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 11%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 20%]
 ........................................................................ [ 22%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 29%]
 ........................................................................ [ 31%]
 ........................................................................ [ 33%]
 ........................................................................ [ 35%]
 ........................................................................ [ 36%]
 ........................................................................ [ 38%]
 ........................................................................ [ 40%]
 ........................................................................ [ 42%]
 ........................................................................ [ 44%]
 ........................................................................ [ 46%]
 ........................................................................ [ 47%]
 ........................................................................ [ 49%]
 ........................................................................ [ 51%]
 ........................................................................ [ 53%]
 ........................................................................ [ 55%]
 ........................................................................ [ 57%]
 ........................................................................ [ 59%]
 ........................................................................ [ 60%]
 ........................................................................ [ 62%]
 ........................................................................ [ 64%]
 ........................................................................ [ 66%]
 ........................................................................ [ 68%]
 ........................................................................ [ 70%]
 ........................................................................ [ 71%]
 ........................................................................ [ 73%]
 ........................................................................ [ 75%]
 ........................................................................ [ 77%]
 ........................................................................ [ 79%]
 ........................................................................ [ 81%]
 ........................................................................ [ 83%]
 ........................................................................ [ 84%]
 ........................................................................ [ 86%]
 ........................................................................ [ 88%]
 ........................................................................ [ 90%]
 ........................................................................ [ 92%]
 ........................................................................ [ 94%]
 ........................................................................ [ 95%]
 ........................................................................ [ 97%]
 ........................................................................ [ 99%]
 ...............                                                          [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  196      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    273      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       71      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   487      0   100%
 --------------------------------------------------
 TOTAL                           5952     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,15 @@
 **Task 02 complete — depth-limited KB tree + server totals (phase 167 D2, LOCKED A2).**
 Implemented:
 - `app/schemas.py`: new `KbTreeTotals` (documents/chunks/last_indexed, ge=0-bounded); `KbTree.totals` (additive top-level key); `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `_drop_truncated_when_false` wrap serializer (the `_drop_image_fields` idiom) — untruncated nodes stay wire byte-identical; `KbTreeFile` untouched.
 - `app/api/docs.py`: `_folder_counts` → `(folders, counts, max_dates)` (recursive max `created_at` map, same pass); `_level_children(..., depth)` — budget-exhausted subfolders ship `truncated=true`, `children=[]`, recursive count + `max_dates`-derived `updated_at` (D9-proof), `summary`, `summary_pending`; `build_kb_tree(..., depth=None)` (None = byte-identical unlimited); source never truncated at depth≥1; endpoint gains `depth: Query(1, ge=1, le=8)` + `_kb_totals()` computed from the same rows (no extra query). `GET /api/docs` / agent `ls` untouched.
 - Tests: 11 new unit pins (depth matrix, byte-identity at None, truncated field set, `_subtree_max`↔`max_dates` cross-check at every depth, phase-94/98 invariants per depth, totals math/empty/validation, image affordance at depth 1); 4 new integration tests (default depth=1 + totals vs `/api/docs`, depth=3, 422 matrix, pre-phase wire-shape pin); migrated deep-walking pins to explicit `?depth=8` (`test_docs_api.py` ×6, `test_docs_api_dates.py` + new truncated-date test, `test_git_source_dates.py`, `test_document_dates.py`).
 - Docs: README RAG section documents depth/truncated/totals; AGENTS.md: no command/env changes → untouched.
 Gates: `uv run pytest --cov=app` → **3919 passed, coverage 99%** (`app/api/docs.py` 99% — only the pre-existing defensive `continue` uncovered; `app/schemas.py` 100%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
 Notable decisions:
 - **Deviation (plan gap):** `tests/e2e/test_document_dates.py` also fetched the tree with no depth (its `_find_file` recurses) — same repair class as item 9's two suites; migrated with `?depth=8` too.
 - **Expected interim E2E state:** client `sources.js` (task 04's scope) still walks the tree for stat cards/drill, so `test_kb_tree.py` stat/drill assertions now fail against the intentional depth=1 default (verified the failure is exactly `#stat-docs` under-counting, not a server defect). This is the owner-confirmed A3 supersede; task 04 (client lazy drill + `totals` cards) and task 06 (E2E gate) repair it. Unit/integration suites (the project's `testpaths`) are 100% green; sources.js pins stay green untouched.
 Next pending task: `167_api_payload_diet/03_tree_node_endpoint_etag.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,127 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 11%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 20%]
 ........................................................................ [ 22%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 29%]
 ........................................................................ [ 31%]
 ........................................................................ [ 33%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 38%]
 ........................................................................ [ 40%]
 ........................................................................ [ 42%]
 ........................................................................ [ 44%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 49%]
 ........................................................................ [ 51%]
 ........................................................................ [ 53%]
 ........................................................................ [ 55%]
 ........................................................................ [ 56%]
 ........................................................................ [ 58%]
 ........................................................................ [ 60%]
 ........................................................................ [ 62%]
 ........................................................................ [ 64%]
 ........................................................................ [ 66%]
 ........................................................................ [ 67%]
 ........................................................................ [ 69%]
 ........................................................................ [ 71%]
 ........................................................................ [ 73%]
 ........................................................................ [ 75%]
 ........................................................................ [ 77%]
 ........................................................................ [ 78%]
 ........................................................................ [ 80%]
 ........................................................................ [ 82%]
 ........................................................................ [ 84%]
 ........................................................................ [ 86%]
 ........................................................................ [ 88%]
 ........................................................................ [ 90%]
 ........................................................................ [ 91%]
 ........................................................................ [ 93%]
 ........................................................................ [ 95%]
 ........................................................................ [ 97%]
 ........................................................................ [ 99%]
 ...............................                                          [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  207      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    273      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       71      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           5981     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,17 @@
 ## Report — phase 167, task 03 (`167_api_payload_diet/03_tree_node_endpoint_etag`)
 **Implemented**
 - `app/core/http_cache.py` (new, 100% covered): `weak_etag` (`W/"sha256[:32]"`, str≡bytes), tolerant `etag_matches` (`*`, comma-lists, weak↔strong both directions, whitespace, `None`/empty→False), `revalidate_headers` (one place for `ETag` + `Cache-Control: private, must-revalidate`; docstrings name the A4/A19 scope).
 - `app/api/docs.py`: extracted `_kb_catalogue` (one catalogue query + the two bounded selects, shared by both tree routes — no N+1) and `_revalidated_json` (serialize once → tag → 304/200); `GET /api/docs/tree/node?source&path&depth` (admin gate, `path=""`→`KbTreeSource` via `_source_node`, else `KbTreeFolder` with `counts`/`max_dates`/summary/`summary_pending`; 404 `{"detail":"unknown node"}` incl. file-paths and traversal — row-only, no filesystem surface); ETag/304 wired on `/api/docs`, `/api/docs/tree`, `/api/docs/tree/node` (`response_model` kept for OpenAPI; node uses `response_model=None` + union annotation).
 - Tests: `tests/unit/test_http_cache.py` (12 pins) + 10 new integration tests in `tests/integration/test_docs_api.py` (node shape/depth/404-matrix/403/422s, node≡`depth=8` tree level equality, full ETag matrix on all 3 routes, A4 bust after folder-summary + document-date PATCHes with `sources_meta.version` provably unmoved, gzip coexistence).
 **Gates** — `uv run pytest` → **3941 passed**; `uv run pytest tests/integration/test_docs_api.py tests/unit/test_http_cache.py -v` → **71 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/core/http_cache.py` 100%, `app/api/docs.py` 99% (only the pre-existing defensive `continue`); `uv run ruff check .` clean; `uv run pyright` 0 errors; caching suites stay green (49 passed).
 **Docs** — README updated (Sources section: node route + revalidation; Caching section: the read-only-API validator arm of A19/A4). AGENTS.md: no doc impact. No new env vars.
 **Notable decisions**
 - The node-route node itself is never `truncated` (its direct children are always built — same rule as `KbTreeSource` at depth ≥1); deeper levels ride the children's flags.
 - `tests/e2e/test_kb_tree.py` has 2 failures (stat cards, 2nd-level drill) — **not introduced here**: they stem from task 02's `depth=1` default and are explicitly owned by task 04 (`sources.js` lazy drill + `totals`) and task 06 (E2E pin migration) per the phase plan; unit+integration (the project suite per `testpaths`) is 100% green.
 - ASSUMPTION kept per task text: `/api/documents/content` gets no validator this phase (task 05's audit decides).
 **Next pending task:** `167_api_payload_diet/04_client_lazy_drill.md`
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,128 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 10%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 20%]
 ........................................................................ [ 21%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 29%]
 ........................................................................ [ 31%]
 ........................................................................ [ 32%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 38%]
 ........................................................................ [ 40%]
 ........................................................................ [ 42%]
 ........................................................................ [ 43%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 49%]
 ........................................................................ [ 51%]
 ........................................................................ [ 52%]
 ........................................................................ [ 54%]
 ........................................................................ [ 56%]
 ........................................................................ [ 58%]
 ........................................................................ [ 60%]
 ........................................................................ [ 62%]
 ........................................................................ [ 63%]
 ........................................................................ [ 65%]
 ........................................................................ [ 67%]
 ........................................................................ [ 69%]
 ........................................................................ [ 71%]
 ........................................................................ [ 73%]
 ........................................................................ [ 74%]
 ........................................................................ [ 76%]
 ........................................................................ [ 78%]
 ........................................................................ [ 80%]
 ........................................................................ [ 82%]
 ........................................................................ [ 84%]
 ........................................................................ [ 85%]
 ........................................................................ [ 87%]
 ........................................................................ [ 89%]
 ........................................................................ [ 91%]
 ........................................................................ [ 93%]
 ........................................................................ [ 95%]
 ........................................................................ [ 96%]
 ........................................................................ [ 98%]
 .....................................................                    [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  234      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    273      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/http_cache.py            22      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       71      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           6030     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,15 @@
 **Task 04 complete — client lazy drill (phase 167 D4/A3).**
 - `frontend/assets/sources.js`: load is now ONE `fetch("/api/docs/tree?depth=1")`; `levelCache` (Map, `source + "/" + folder`) cleared on every fresh tree; `needsLevelFetch` + `enterFolder`/`fetchLevel` fetch `GET /api/docs/tree/node?…` only for uncached `truncated` nodes, `Object.assign`-merge + cache only after a 200; `drillPending` double-click guard with `aria-busy`/`aria-disabled` on the row link; `role="status"` "Loading <folder>…" row (departing level stays rendered) and `role="alert"` "Couldn't load … — Retry" with a real `<button type="button">`; `loadSeq` extended to the drill (+ `navSeq` so navigation invalidates it, never a stale paint); node 404 → honest top-level reset; `folderExistsIn` → `folderKnownIn` (ancestor-chain over held levels); `treeStats()` prefers server `totals` (walk kept as fallback); popstate adopts rehydrate via `ensureCurrentLevel` (fixed the forward-nav E2E regression); module docstring rewritten for the depth-1 + per-level model.
 - `frontend/assets/styles.css`: one tokens-only `.kb-level-loading` rule (Retry reuses the existing `.retry-btn` pill).
 - Tests: migrated `tests/unit/test_kb_tree_ui.py` (depth=1/node fetch counts, levelCache lifecycle, race token, aria states, status/alert rows, 404 reset, totals-first stats, folderKnownIn, docstring pins — all phase-97/99 pins preserved) and `tests/unit/test_frontend_router.py:591`; migrated `tests/e2e/test_navbar_refresh.py` request-pin to `?depth=1`.
 Results:
 - `uv run pytest` → **3948 passed**; `--cov=app` → **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
 - E2E in isolation, all green: `test_kb_tree.py` (8), `test_kb_tree_nav.py` (4, incl. the back/forward walk), `test_navbar_refresh.py`, `test_document_dates.py`, `test_git_source_dates.py`, `test_sync_summary_visibility.py`, `test_table_ux_polish.py` (26), `test_sources_midstream_bug.py` + `test_ls_tree_drilldown.py` + `test_smoke.py` (12).
 Docs: README RAG-view paragraph extended (per-level cache, Loading/Retry states, 404 reset); no AGENTS.md impact.
 Notable: added `navSeq` + `ensureCurrentLevel` beyond the task sketch — popstate/forward adopts of a not-yet-refetched level would otherwise render a half-empty level (`test_kb_tree_nav.py::test_back_button_walks_the_breadcrumb` caught it; now green).
 Next pending task: `167_api_payload_diet/05_response_budget_audit.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,128 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 10%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 20%]
 ........................................................................ [ 21%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 29%]
 ........................................................................ [ 31%]
 ........................................................................ [ 32%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 38%]
 ........................................................................ [ 40%]
 ........................................................................ [ 41%]
 ........................................................................ [ 43%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 49%]
 ........................................................................ [ 51%]
 ........................................................................ [ 52%]
 ........................................................................ [ 54%]
 ........................................................................ [ 56%]
 ........................................................................ [ 58%]
 ........................................................................ [ 60%]
 ........................................................................ [ 62%]
 ........................................................................ [ 63%]
 ........................................................................ [ 65%]
 ........................................................................ [ 67%]
 ........................................................................ [ 69%]
 ........................................................................ [ 71%]
 ........................................................................ [ 72%]
 ........................................................................ [ 74%]
 ........................................................................ [ 76%]
 ........................................................................ [ 78%]
 ........................................................................ [ 80%]
 ........................................................................ [ 82%]
 ........................................................................ [ 83%]
 ........................................................................ [ 85%]
 ........................................................................ [ 87%]
 ........................................................................ [ 89%]
 ........................................................................ [ 91%]
 ........................................................................ [ 93%]
 ........................................................................ [ 94%]
 ........................................................................ [ 96%]
 ........................................................................ [ 98%]
 ............................................................             [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  234      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    273      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/http_cache.py            22      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       71      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           6030     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,17 @@
 All green. Final state verified.
 **Implemented (task 05)**
 - `app/core/response_budget.py` — pure-ASGI byte-budget middleware: one `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per `/api/*` response (`stream=1` for `text/event-stream`), INFO at/over budget, DEBUG below, `304`→`bytes=0`, non-`/api/`+non-HTTP silent, exceptions logged and re-raised unchanged; `bytes=` = route bytes, `enc=` = wire encoding (reads the shared start-message the gzip layer decorates in place — documented, integration-pinned).
 - `Settings.log_large_response_bytes=65536` + `agent_max_rounds`-style validator (`0`=all INFO, negative refuses startup); `.env.example` entry; `app/main.py` wires it **before** the gzip call → SecurityHeaders → gzip → response-budget → caching → session.
 - `scripts/api_payload_audit.py` (stdlib+httpx, read-only, exit 0 report / 2 unreachable-or-bad-login) + `AUDIT.md` with verbatim BEFORE (`BOR_COMPRESS_MIN_BYTES=0`) and AFTER tables, curl proof, real log lines incl. `POST /api/chat … stream=1`, and a verdict per endpoint.
 - Tests: new `tests/unit/test_response_budget.py` (24), new `tests/integration/test_response_budget_api.py` (7), one SSE `stream=1` pin in `tests/integration/test_chat_api.py`; migrated the two `test_compression.py` middleware-stack pins.
 **Live-infra used (local dev only)**: restored the 1,000-doc live replica via the `restore-test-db` skill (safety backup `/tmp/pre_restore_20261008_073453.dump`; fingerprint `docs=1000 chunks=8866`; alembic 0011→0027), ran the dev server for the audit, then stopped it. Key measured numbers: tree depth=1 **8,474 B → 1,129 B gzip → 304/0** (full depth-8 tree 265,147 B); `/api/docs` 246,366→45,107; `/api/documents/content` 892,124→60,776 (LOOK, accepted with why). AUDIT.md states the snapshot has **0 folder summaries / 0 image docs**, so identity bytes are a lower bound on the 2 MB production report.
 **Docs**: README "Operational notes: the response byte budget" (log format, `grep ' :: api '`, `BOR_LOG_LARGE_RESPONSE_BYTES`, audit run recipe, self-gzip/nginx deploy note) + config-reference row; AGENTS.md rule 10 + quick-reference line.
 **Gates (exact)**: `uv run pytest --cov=app --cov-report=term-missing` → **3980 passed**, TOTAL 99%, `app/core/response_budget.py` **100%**; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors; `uv run pytest tests/e2e/test_smoke.py -q --no-cov` → green.
 **Deviations**: no `--username` flag (the single-admin `/api/login` is password-only, A10 — documented in the script); no new ETag remedies applied — `/api/git-sources` measures 188 B and `/api/documents/content` is the task's documented accepted case with a named follow-up.
 **Next pending task**: `167_api_payload_diet/06_e2e_payload_diet.md`.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,130 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 10%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 19%]
 ........................................................................ [ 21%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 28%]
 ........................................................................ [ 30%]
 ........................................................................ [ 32%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 37%]
 ........................................................................ [ 39%]
 ........................................................................ [ 41%]
 ........................................................................ [ 43%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 48%]
 ........................................................................ [ 50%]
 ........................................................................ [ 52%]
 ........................................................................ [ 54%]
 ........................................................................ [ 56%]
 ........................................................................ [ 57%]
 ........................................................................ [ 59%]
 ........................................................................ [ 61%]
 ........................................................................ [ 63%]
 ........................................................................ [ 65%]
 ........................................................................ [ 66%]
 ........................................................................ [ 68%]
 ........................................................................ [ 70%]
 ........................................................................ [ 72%]
 ........................................................................ [ 74%]
 ........................................................................ [ 75%]
 ........................................................................ [ 77%]
 ........................................................................ [ 79%]
 ........................................................................ [ 81%]
 ........................................................................ [ 83%]
 ........................................................................ [ 85%]
 ........................................................................ [ 86%]
 ........................................................................ [ 88%]
 ........................................................................ [ 90%]
 ........................................................................ [ 92%]
 ........................................................................ [ 94%]
 ........................................................................ [ 95%]
 ........................................................................ [ 97%]
 ........................................................................ [ 99%]
 ....................                                                     [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  234      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    280      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/http_cache.py            22      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/response_budget.py       63      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       73      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           6102     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -0,0 +1,14 @@
 Final verification pass complete. Everything is green.
 **Phase 167 task 06 — report**
 1. **New suite** `tests/e2e/test_kb_tree_lazy_load.py` — **11 passed in isolation**: one `?depth=1` boot request (zero node fetches, parsed body: `totals` 8 docs/12 chunks, recursive source counts, no node deeper than level 1, every level-1 folder `truncated: true`, D9 recursive date on the truncated `alpha`); gzip + `Vary: Accept-Encoding` + `W/"…"` ETag on the real Chromium negotiation; one node fetch per uncached level (source click = zero); two levels deep then back = zero new fetches, cache never re-fetched; D4 pending marker + image thumbnail + phase-156 sticky header/scroll panel/44 px Retry + 72rem container in a lazily rendered level; `role="status"`/"Loading alpha…" in-flight state where a second click issues no second request; 500 → alert + Retry with the level preserved, Retry recovers; 404 → honest reset to the top level.
 2. **Revalidation**: Chromium hands `fetch` the cached 200, so reload and History→Sources revalidation are pinned on the CDP Network events (DevTools-panel truth): `If-None-Match` sent, wire `304`, transfer < ½ the first load, table still renders; plus the never-stale half — a folder-description PATCH busts the tag → `200` + the new text renders.
 3. **Migrated, never deleted**: the superseded "ONE fetch / drill is client-side" claims re-pointed in `test_kb_tree.py` (module contract + `_wait_top_level` + `_drill` + the breadcrumb-back pin) and in 8 sibling suites' drill helpers (`test_kb_tree_nav`, `test_admin_auth`, `test_folder_summary_voice`, `test_sync_summary_visibility`, `test_import_documents`, `test_extensionless_import`, `test_import_extensions_env`, `test_quadlet_jinja_import`). All re-ran green in isolation.
 4. **Regression set, isolation**: kb_tree 8, kb_tree_nav 4, navbar_refresh 7, document_dates 6, git_source_dates 4, sync_summary_visibility 3, table_ux_polish 6, sources_midstream_bug 6, ls_tree_drilldown 3 (agent `ls` unchanged), smoke 3 — all pass.
 5. **Full gates**: `uv run pytest` **3980 passed**; `--cov=app` **TOTAL 99 %** (`compression.py`, `http_cache.py`, `response_budget.py` 100 %, `api/docs.py` 99 %); `ruff check .` and `pyright` clean.
 6. **Retrieval battery**: no `app/rag/**` file is touched by this phase (`git diff --name-only`), so retrieval is unchanged by construction; the battery itself cannot measure here — the shared E2E Postgres holds no corpus (the suites truncate it), so every question deflects at `best_cosine=0.0000`.
 7. **Defect found and fixed** (pre-existing, not phase 167): `test_sources_midstream_bug.py::_reset_db` truncated only `chunks, documents, query_log`, so a `kb_overview` row left by another suite's sync leaked into the prompt and the mock echoed `(kb: …)` into the answer, breaking its exact-text pin. It now truncates the same house list as `test_kb_tree._truncate_all`; verified failing before, 6/6 after, including directly after `test_kb_tree`. 27 other suites still truncate narrowly — same latent order coupling, left alone as out of scope.
 8. **Deviation flagged**: task 06's vanished-folder bullet says a `404` shows alert + Retry, but `00_phase.md` D4/D6 and the shipped client make the node route's 404 the vanished signal (reset to the top level). Both paths are pinned honestly — 404 → honest reset, 500 → alert + Retry with the level preserved — no assertion weakened.
 9. **Docs**: `AUDIT.md` gained the browser-verification paragraph (gzip, 304, one fetch per level); README's tree/node endpoint table, gzip + ETag deploy note, env var list and `AGENTS.md` rule 10 / quick-reference were reviewed and are already phase-167 consistent; `.env.example` carries both new vars.
 10. No git index/history commands run, no `todo/` phase file moved, renamed or edited.
@@ -0,0 +1 @@
 ok
@@ -0,0 +1,130 @@
 ........................................................................ [  1%]
 ........................................................................ [  3%]
 ........................................................................ [  5%]
 ........................................................................ [  7%]
 ........................................................................ [  9%]
 ........................................................................ [ 10%]
 ........................................................................ [ 12%]
 ........................................................................ [ 14%]
 ........................................................................ [ 16%]
 ........................................................................ [ 18%]
 ........................................................................ [ 19%]
 ........................................................................ [ 21%]
 ........................................................................ [ 23%]
 ........................................................................ [ 25%]
 ........................................................................ [ 27%]
 ........................................................................ [ 28%]
 ........................................................................ [ 30%]
 ........................................................................ [ 32%]
 ........................................................................ [ 34%]
 ........................................................................ [ 36%]
 ........................................................................ [ 37%]
 ........................................................................ [ 39%]
 ........................................................................ [ 41%]
 ........................................................................ [ 43%]
 ........................................................................ [ 45%]
 ........................................................................ [ 47%]
 ........................................................................ [ 48%]
 ........................................................................ [ 50%]
 ........................................................................ [ 52%]
 ........................................................................ [ 54%]
 ........................................................................ [ 56%]
 ........................................................................ [ 57%]
 ........................................................................ [ 59%]
 ........................................................................ [ 61%]
 ........................................................................ [ 63%]
 ........................................................................ [ 65%]
 ........................................................................ [ 66%]
 ........................................................................ [ 68%]
 ........................................................................ [ 70%]
 ........................................................................ [ 72%]
 ........................................................................ [ 74%]
 ........................................................................ [ 75%]
 ........................................................................ [ 77%]
 ........................................................................ [ 79%]
 ........................................................................ [ 81%]
 ........................................................................ [ 83%]
 ........................................................................ [ 85%]
 ........................................................................ [ 86%]
 ........................................................................ [ 88%]
 ........................................................................ [ 90%]
 ........................................................................ [ 92%]
 ........................................................................ [ 94%]
 ........................................................................ [ 95%]
 ........................................................................ [ 97%]
 ........................................................................ [ 99%]
 ....................                                                     [100%]
 =============================== warnings summary ===============================
 .venv/lib/python3.13/site-packages/fastapi/testclient.py:1
  /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
    from starlette.testclient import TestClient as TestClient  # noqa
 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
 ================================ tests coverage ================================
 _______________ coverage: platform linux, python 3.13.13-final-0 _______________
 Name                           Stmts   Miss  Cover
 --------------------------------------------------
 app/__init__.py                    1      0   100%
 app/api/__init__.py                0      0   100%
 app/api/auth.py                   52      0   100%
 app/api/chat.py                  277      1    99%
 app/api/chat_images.py            50      0   100%
 app/api/chats.py                 110      0   100%
 app/api/config.py                 13      0   100%
 app/api/doc_drafts.py             99      0   100%
 app/api/docs.py                  234      1    99%
 app/api/git_sources.py           259      0   100%
 app/api/health.py                 10      0   100%
 app/api/steering.py               42      0   100%
 app/api/suggestions.py            33      0   100%
 app/api/sync.py                  204      0   100%
 app/api/tokens.py                 40      0   100%
 app/api/ui_settings.py            55      0   100%
 app/config.py                    280      0   100%
 app/core/__init__.py               0      0   100%
 app/core/auth.py                  45      0   100%
 app/core/caching.py              124      0   100%
 app/core/compression.py           59      0   100%
 app/core/debugging.py             29      2    93%
 app/core/docs_push.py             39      0   100%
 app/core/errors.py                 5      0   100%
 app/core/http_cache.py            22      0   100%
 app/core/logging.py               13      0   100%
 app/core/rate_limit.py            44      0   100%
 app/core/response_budget.py       63      0   100%
 app/core/security_headers.py      20      0   100%
 app/core/theming.py               38      0   100%
 app/core/tokens.py                44      0   100%
 app/db.py                         22      0   100%
 app/main.py                       73      0   100%
 app/models.py                    136      0   100%
 app/rag/__init__.py                0      0   100%
 app/rag/agent.py                 454      1    99%
 app/rag/archive_upload.py        134      0   100%
 app/rag/chunker.py               220      4    98%
 app/rag/doc_dates.py              18      0   100%
 app/rag/folder_summaries.py      134      0   100%
 app/rag/git_sources.py            77      0   100%
 app/rag/importer.py              841      4    99%
 app/rag/llm.py                   246      1    99%
 app/rag/md_images.py              63      0   100%
 app/rag/overview.py               78      0   100%
 app/rag/pdf.py                   192      0   100%
 app/rag/prompts.py               156      0   100%
 app/rag/retriever.py             252      3    99%
 app/rag/scaffolding.py            55      0   100%
 app/rag/source_removal.py         41      0   100%
 app/rag/sources_meta.py           16      0   100%
 app/rag/suggestions.py            27      0   100%
 app/rag/summarizer.py             58      0   100%
 app/schemas.py                   505      0   100%
 --------------------------------------------------
 TOTAL                           6102     17    99%
 coverage gate: app/ 99% (>90%) OK
 All checks passed!
 0 errors, 0 warnings, 0 informations
 WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
 Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
 validation OK
@@ -12,6 +12,24 @@ BOR_ENVIRONMENT=development
 #  a local .env is ignored.)
 # BOR_LOG_LEVEL=INFO
 # BOR_STATIC_DIR=frontend          # dev default; container sets /app/static
 # BOR_COMPRESS_MIN_BYTES=860       # gzip responses of >= this many bytes when the
                                   # client accepts it (phase 167, TODO L5 — the 2 MB
                                   # /api/docs/tree defect; ~8-12x on this JSON).
                                   # 0 = feature OFF (no middleware at all);
                                   # negative refuses startup naming the field;
                                   # 860 = the classic nginx default, below which
                                   # compression can inflate a body instead of
                                   # shrinking it. POST /api/chat (SSE, A15) is
                                   # NEVER compressed.
 # BOR_LOG_LARGE_RESPONSE_BYTES=65536  # INFO/DEBUG cut for the per-response
                                   # "api method=… bytes=… enc=…" line the
                                   # byte budget logs for every /api/* response
                                   # (phase 167, TODO L5 — read it with
                                   # grep ' :: api '):
                                   # >= 65536 (64 KB) -> INFO, below -> DEBUG
                                   # (invisible at BOR_LOG_LEVEL=INFO);
                                   # 0 = log EVERY /api/* response at INFO;
                                   # negative refuses startup naming the field.
 # --- Database (matches `podman compose` db service) ---
 BOR_DATABASE_URL=postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese
@@ -28,7 +28,12 @@
   the primary class shows the strip in place of the text, the
   additive class never hides the text: Text/Pages toggle group
   (`aria-pressed`, Text default) with the strip behind it
-   (phase 163)) and
+   (phase 163); a lazily-drilled KB-tree level keeps the level the
   user is ON rendered while its node fetch runs, and its ONE
   in-flight row (`role="status"`) / alert + Retry row
   (`role="alert"`, a real `<button>`) sits inside that same bounded
   scroll panel — and the stat cards read the server's `totals`,
   never a walk of a depth-limited tree (phase 167)) and
   meets WCAG 2.1 AA basics (semantic
   landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the
   stream).
@@ -51,8 +56,13 @@
   Playwright E2E file passes **in isolation**
   (`uv run pytest tests/e2e/test_<story>.py -v --no-cov`).
 10. **Amply log, visibly feed back:** server-side, log the required
-    per-turn line (PLAN §9); UI-side, honor the "never stale" feedback
+    per-turn line (PLAN §9) **and** the per-response byte-budget line
-    contract (PLAN §7.4).
+    (`app/core/response_budget.py` — one
    `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per
    `/api/*` response, INFO at/over `BOR_LOG_LARGE_RESPONSE_BYTES`, read it
    with `grep ' :: api '`, audit it with
    `uv run python -m scripts.api_payload_audit`, phase 167); UI-side,
    honor the "never stale" feedback contract (PLAN §7.4).
 ## Quick reference
 ```bash
@@ -66,4 +76,5 @@ uv run playwright install chromium           # once
 uv run pytest tests/e2e/test_smoke.py -v --no-cov
 uv run ruff check . && uv run pyright        # lint + types
 uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt   # retrieval regression battery (phases 119/162)
 uv run python -m scripts.api_payload_audit                      # endpoint byte audit vs the 256 KB budget (phase 167; app must be running)
 ```
@@ -159,6 +159,30 @@ document's date from the document viewer. The date also feeds the small
 recency boost in retrieval ranking (`BOR_RECENCY_BOOST`). Admin-only —
 anonymous visitors see a sign-in gate instead.
 The tree payload is bounded (phase 167): `GET /api/docs/tree` builds only
 `depth` levels (default **1** — the sources plus one level, the page's first
 paint; `depth=2..8` builds deeper, anything else is a 422) instead of the old
 ~2 MB full recursive dump. A node whose children were not built carries
 `"truncated": true` and ships with its **recursive** document count, updated
 date, description and pending marker (`children: []`); untruncated nodes are
 byte-for-byte what they were before. The three stat cards read the
 server-computed top-level `totals` (`documents`, `chunks`, `last_indexed`) so
 they stay honest whatever depth was requested. Responses also ship gzipped
 (see *Caching & deploys*). Deeper levels load on demand through
 `GET /api/docs/tree/node?source=X&path=Y&depth=N` (admin-only, phase 167):
 one node — the source root for `path=""` — answered in exactly the shape the
 full tree produces for it (unknown source or folder → 404 `unknown node`),
 built from the same single catalogue query. The RAG view fetches each drilled
 level once and caches it (re-opening a folder costs no network), showing one
 *Loading…* row while a level arrives and a *Couldn't load … — Retry* line on
 failure with the current level staying rendered; a folder that vanished since
 it was listed (the node 404) resets the view to the top level. The
 breadcrumb, the bounded scroll panel, and the description editors behave
 exactly as before. All three catalogue reads
 (`/api/docs`, `/api/docs/tree`, `/api/docs/tree/node`) also carry a weak
 `ETag` + `Cache-Control: private, must-revalidate`, so reloading a page you
 just looked at costs a bodiless 304 (see *Caching & deploys*).
 ### Git sources (`/git-sources.html`)
 The admin-managed source registry: add or remove git repositories, upload
@@ -942,8 +966,13 @@ uploaded archives):
 ## Caching & deploys
 A deploy is a commit — and the browser must see it **without a hard refresh**.
-One Starlette middleware (`app/core/caching.py`) applies the rule at the
+The first three bullets below are that A19 rule, applied at the transport layer
-transport layer:
+by `app/core/caching.py`; the last two are phase 167 — gzip from
 `app/core/compression.py` and per-route validators from `app/core/http_cache.py`.
 Outermost to innermost, the stack is security-headers → gzip (only when
 `BOR_COMPRESS_MIN_BYTES > 0`) → the byte-budget logger → caching → routes, so
 security headers land on every response and `bytes=` in the budget line is the
 route's own, uncompressed payload (see *Operational notes*):
 - **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
  `last-modified`** — each visit always gets a fresh 200 body.
@@ -954,6 +983,26 @@ transport layer:
  `HEAD` — so every commit/deploy flips the token and the versioned asset
  URLs change with it. A checkout without `.git` falls back to a stable
  content hash of the `frontend/` tree.
 - **Responses are gzipped (phase 167).** Every HTTP response except the
  `POST /api/chat` SSE stream goes through selective gzip
  (`app/core/compression.py`): when the client accepts gzip and the body
  is at least `BOR_COMPRESS_MIN_BYTES` (default `860`; `0` = the feature
  is OFF), KB JSON ships ~8-12x smaller — the fix for the ~2 MB
  `/api/docs/tree` load. The SSE stream is skipped by EXACT path (A15:
  frames keep arriving one by one, uncompressed), small responses stay
  identity, and an already-encoded body is never double-encoded, so a
  fronting nginx may gzip too.
 - **Catalogue reads revalidate (phase 167).** `GET /api/docs`,
  `GET /api/docs/tree` and `GET /api/docs/tree/node` ship a content-exact
  weak `ETag` (SHA-256 of the served body, `app/core/http_cache.py`) with
  `Cache-Control: private, must-revalidate` — a repeat load costs a 304
  with no body, and a sync or an admin edit (folder summary, document
  date) changes the body, hence the tag, so it is seen on the next load.
  The tag is deliberately NOT `sources_meta.version` (bumping that marks
  saved chats stale, and those edits never bump it). This is a scoped
  extension of the rule above: validators live on these three read-only
  routes only — HTML pages keep publishing none, and the SSE stream and
  every mutating route are untouched.
 No CDN, no new services, no build-step change: the middleware rewrites the
 asset references of the known pages in flight.
@@ -962,6 +1011,61 @@ asset references of the known pages in flight.
 > page visit, so the browser revalidates the HTML once and starts requesting
 > the versioned assets; every commit after that is picked up automatically.
 ### Operational notes: the response byte budget (phase 167)
 Every `/api/*` response writes ONE parseable line, so "which endpoint is
 heavy, right now?" is answerable from the app's own log:
 ```
 2026-10-08 07:40:57 INFO     app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
 ```
 ```bash
 grep ' :: api ' server.log                    # every measured response
 grep ' :: api ' server.log | grep 'bytes=[0-9]\{6,\}'   # the six-figure ones
 ```
 - **`bytes=`** is what the ROUTE produced (uncompressed), **`enc=`** is what
  went on the wire (`gzip` or `identity`) — "how fat is it, and how fat did
  the client see it". A revalidated `304` logs `bytes=0` on purpose: that is
  the win you want to see.
 - **Level:** `INFO` at or over `BOR_LOG_LARGE_RESPONSE_BYTES` (default
  `65536`), `DEBUG` below it, so the default log stays readable.
  `BOR_LOG_LARGE_RESPONSE_BYTES=0` logs EVERY `/api/*` response at INFO —
  the mode for a one-off investigation. A negative value refuses startup.
 - **`stream=1`** marks the SSE turn (`POST /api/chat`, A15): `bytes` is the
  streamed total and `duration_ms` the stream's life. One line per stream,
  written when it ends — including a stream the client cancels (with the
  bytes that got out). The middleware is pure ASGI and never buffers.
 **Measuring the whole API** (the audit behind the fix):
 ```bash
 # the corpus that matches production's order of magnitude
 #   (destructive to the local DB — see the restore-test-db skill)
 uv run uvicorn app.main:app --port 8000
 uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
 ```
 It signs in with `BOR_ADMIN_PASSWORD` (or `--password`; the single-admin
 login has no username), measures every candidate read route in BOTH
 `identity` and `gzip` forms plus one conditional revalidation, prints the
 corpus it measured, and marks anything over `--budget` (default `262144`)
 with `LOOK`. It is **read-only** (safe against the deployed app) and always
 exits 0 when it got its numbers — it is a report, not a gate (LOCKED A5:
 nothing in this phase rejects a large payload). Exit `2` means the app was
 unreachable or the login was rejected. The measured before/after tables and
 the verdict per endpoint live in the phase record — `167_api_payload_diet/AUDIT.md`
 under `.agents/phases/`.
 > **Deploy note (compression):** the app gzips its own responses, so nginx
 > `gzip on` is **optional** — leaving it off ships the same bytes, and
 > leaving it on is fine too: an already-encoded body is never double-encoded
 > by either side. Passing the client's header through untouched
 > (`proxy_set_header Accept-Encoding $http_accept_encoding;`, or simply not
 > overriding it) is correct; unsetting it would hand every client
 > uncompressed JSON.
 ---
 ## Configuration reference
@@ -982,6 +1086,8 @@ to `.env` and adjust.
 | `BOR_ENVIRONMENT` | `development` | `development` or `production` (the prod compose profile sets `production`) |
 | `BOR_LOG_LEVEL` | `INFO` | App log level |
 | `BOR_STATIC_DIR` | `frontend` | Where static files are served from (the container image sets `/app/static`) |
 | `BOR_COMPRESS_MIN_BYTES` | `860` | Selective-gzip floor (phase 167): responses ≥ this size compress when the client accepts gzip; `0` = OFF (no middleware), negative refuses startup. `POST /api/chat` (SSE) is never compressed |
 | `BOR_LOG_LARGE_RESPONSE_BYTES` | `65536` | INFO/DEBUG cut for the per-response `api method=… bytes=… enc=…` byte-budget line (phase 167, `grep ' :: api '`); `0` = log every `/api/*` response at INFO, negative refuses startup. See *Operational notes* |
 ### Database & concurrency
@@ -18,10 +18,34 @@ date folds to today, D3) + flag it manual, or clear the manual flag
 write: NO LLM/embedding call — a date is never embedded (the
 deliberate contrast with the phase-57 ``is_summary`` re-embed).
-GET /api/docs/tree — the admin's full recursive KB tree in one fetch
+GET /api/docs/tree — the admin's KB tree (phase 97, task 02; depth-
-(phase 97, task 02): the same drill-down tree the agent's ``ls``
+limited since phase 167, task 02): the same drill-down tree the agent's
-walks, with the file metadata the RAG view's rows and stat cards need
+``ls`` walks, with the file metadata the RAG view's rows and stat cards
-(the view drills client-side; ``GET /api/docs`` is untouched).
+need. The default ``depth=1`` ships the sources plus ONE level with
 server-computed ``totals`` — a node whose children were not built
 carries ``truncated: true`` (the TODO L5 2 MB payload fix; ``GET
 /api/docs`` and the agent's ``ls`` path are untouched).
 GET /api/docs/tree/node — ONE tree node's direct children (phase 167,
 task 03, D3): the RAG view's per-level drill fetch — the same catalogue
 composition as ``/api/docs/tree`` (ONE catalogue query per drill, no
 N+1) resolved to a single ``KbTreeSource`` (``path=""``, the source
 root) or ``KbTreeFolder``. Admin-gated like the tree; 404
 ``{"detail": "unknown node"}`` for a source that is neither registered
 nor indexed, or a folder failing the phase-94 existence rule. Row-only
 like every other route here: no request value ever reaches a
 filesystem path — ``../``-style values simply are not folders (the
 phase-10 traversal argument, reused).
 Revalidation (phase 167, task 03, LOCKED A4 — a scoped extension of
 A19): the three catalogue READ routes (``/api/docs``,
 ``/api/docs/tree``, ``/api/docs/tree/node``) serialize once and carry a
 content-exact weak ``ETag`` + ``Cache-Control: private,
 must-revalidate`` (the :mod:`app.core.http_cache` helpers), so a repeat
 load costs a 304 with no body. The tag is a hash of the served bytes —
 NOT ``sources_meta.version`` (bumping that would mark saved chats
 stale, and the admin PATCHes do not bump it). ``sources_meta``, the
 SSE stream and the mutating surface are untouched.
 GET /api/documents/{doc_id}/image — the phase-122 (task 04) image
 BYTES route: the persistent copy behind an ``is_image`` document's
@@ -70,14 +94,16 @@ from collections.abc import Mapping, Sequence
 from datetime import UTC, datetime
 from pathlib import Path
-from fastapi import APIRouter, Depends, HTTPException
+from fastapi import APIRouter, Depends, HTTPException, Query, Request
-from fastapi.responses import FileResponse
+from fastapi.responses import FileResponse, Response
 from pydantic import BaseModel
 from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 from app.api.sync import _sanitize_error
 from app.config import get_settings
 from app.core.auth import require_admin, require_user
 from app.core.http_cache import etag_matches, revalidate_headers, weak_etag
 from app.db import get_db
 from app.models import Chunk, Document, FolderSummary
 from app.rag.agent import list_source_names
@@ -99,6 +125,7 @@ from app.schemas import (
    KbTreeFile,
    KbTreeFolder,
    KbTreeSource,
    KbTreeTotals,
    SummaryResult,
    SummaryUpdate,
 )
@@ -128,9 +155,10 @@ def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
@router.get("/docs", response_model=DocList)
 def list_indexed_documents(
    request: Request,
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
-) -> DocList:
+) -> Response:
    """All indexed documents with per-document chunk counts.
    Admin-only (phase 16 — the catalog is what the admin sign-in gates;
@@ -139,6 +167,16 @@ def list_indexed_documents(
    ``admin only`` and the Sources page renders its sign-in gate instead.
    An empty list means the knowledge base has not been imported yet —
    the Sources page renders its designed empty state in that case.
    Phase 167 (task 03, D3/A4): the response carries the content-exact
    weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair
    (see :mod:`app.core.http_cache`) — the client is ``fetch``, which
    serves a matched conditional request from its cache transparently,
    and ``must-revalidate`` guarantees a sync is seen on the next load
    (the tag changes because the body changes). The decorator keeps
    ``response_model=DocList`` for the OpenAPI schema only — a returned
    :class:`~fastapi.responses.Response` short-circuits FastAPI's
    serialization, so the model is dumped exactly ONCE here.
    """
    rows = db.execute(
        select(
@@ -161,7 +199,7 @@ def list_indexed_documents(
        )
        .order_by(Document.source, Document.path)
    ).all()
-    return DocList(
+    model = DocList(
        documents=[
            DocSummary(
                id=str(row.id),
@@ -175,6 +213,7 @@ def list_indexed_documents(
            for row in rows
        ]
    )
    return _revalidated_json(request, model)
@router.get("/documents/content", response_model=DocContent)
@@ -625,8 +664,9 @@ TreeFileRow = tuple[str, str, int, str, str]
 def _folder_counts(
    rows: Sequence[TreeFileRow],
-) -> tuple[set[str], dict[str, int]]:
+) -> tuple[set[str], dict[str, int], dict[str, str]]:
-    """One source's folders (existence rule) + recursive counts (pure).
+    """One source's folders (existence rule) + recursive counts + the
    recursive max-date map (all pure).
    The phase-94 rules, reused verbatim from
    :func:`app.rag.agent.group_folder_listing` (ONE concept end to end —
@@ -640,9 +680,22 @@ def _folder_counts(
    ``folder + "/"`` — exactly the set the sync-time folder summary
    describes.
-    Returns ``(folders, counts)`` — the folder-name set and the
+    Returns ``(folders, counts, max_dates)`` — the folder-name set, the
    per-folder count map (every folder counts ≥ 1 by construction: its
-    own descendants, or the file wearing its name, exist).
+    own descendants, or the file wearing its name, exist), and the
    per-folder RECURSIVE MAX document ``created_at`` map — the same
    subtree the count walks, over the same ISO-8601 strings (one
    ``isoformat()`` shape, so the lexicographic max is the
    chronological one — the phase-106 D9 rule).
    The max map is the phase-167 (task 02, D2) truncation-proof date
    source: a depth-limited node has NO built children, so
    :func:`_subtree_max` over its empty child list would read ``None``
    and silently lose the D9 column; the aggregate — computed from the
    rows in the same pass — keeps the date honest at every depth. The
    expanded path keeps deriving ``updated_at`` via
    :func:`_subtree_max` (one concept, two derivations — cross-pinned
    equal by a unit test).
    """
    folders: set[str] = set()
    for path, _title, _chunks, _indexed_at, _created_at in rows:
@@ -651,14 +704,19 @@ def _folder_counts(
            folders.add(folder)
            folder = folder_of(folder)
    counts: dict[str, int] = {folder: 0 for folder in folders}
-    for path, _title, _chunks, _indexed_at, _created_at in rows:
+    max_dates: dict[str, str] = {}
    for path, _title, _chunks, _indexed_at, created_at in rows:
        if path in folders:
            counts[path] += 1
            if created_at > max_dates.get(path, ""):
                max_dates[path] = created_at
        folder = folder_of(path)
        while folder:
            counts[folder] += 1
            if created_at > max_dates.get(folder, ""):
                max_dates[folder] = created_at
            folder = folder_of(folder)
-    return folders, counts
+    return folders, counts, max_dates
 #: One image-docs map value (phase 122, task 04):
@@ -673,9 +731,11 @@ def _level_children(
    folder: str,
    folders: set[str],
    counts: dict[str, int],
    max_dates: dict[str, str],
    rows: Sequence[TreeFileRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
 ) -> list[KbTreeFolder | KbTreeFile]:
    """One level's children (pure): subfolders in path order, then the
    direct files in input (catalog) order.
@@ -691,6 +751,20 @@ def _level_children(
    file list is NOT capped (the ``ls`` 50-line cap is a model-context
    budget — the UI is for humans). Recurses one level per call.
    *depth* (phase 167, task 02, D2/A2) is the REMAINING build budget
    for this level's subfolders: ``None`` = unlimited (the pre-phase
    recursion, byte-identical); a positive budget is spent here and
    ``depth - 1`` rides the recursion. When the budget is exhausted
    (``depth == 1`` reaching its subfolders) the subfolder nodes are
    TRUNCATED instead of recursed: ``children=[]``, ``truncated=True``,
    and every field the level row still renders — the recursive
    ``documents`` count, the recursive ``updated_at`` from *max_dates*
    (NOT ``_subtree_max`` over the empty child list — the D9 trap), the
    node's own ``summary`` (the row shows a SUBFOLDER's description),
    and ``summary_pending`` under the unchanged rule. The level's
    DIRECT files are always built — the budget limits DEPTH, never the
    row set of the levels it does send.
    Each folder node also carries the phase-98 D3 ``summary_pending``
    flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
    stored ``folder_summaries`` row for ``(source, sub)`` — the same
@@ -709,9 +783,12 @@ def _level_children(
    byte-identical).
    """
    children: list[KbTreeFolder | KbTreeFile] = []
    expand = depth is None or depth > 1
    next_depth = None if depth is None else depth - 1
    for sub in sorted(g for g in folders if folder_of(g) == folder):
        if expand:
            sub_children = _level_children(
-            source, sub, folders, counts, rows, summaries, images
+                source, sub, folders, counts, max_dates, rows, summaries, images, next_depth
            )
            children.append(
                KbTreeFolder(
@@ -724,6 +801,22 @@ def _level_children(
                    children=sub_children,
                )
            )
        else:
            # Phase 167 (D2): budget exhausted — the node rides with its
            # recursive aggregate fields and truncated=True; the client
            # drills via the node route (task 03).
            children.append(
                KbTreeFolder(
                    path=sub,
                    documents=counts[sub],
                    updated_at=max_dates[sub],
                    summary=summaries.get((source, sub)),
                    summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
                    and (source, sub) not in summaries,
                    children=[],
                    truncated=True,
                )
            )
    for path, title, chunks, indexed_at, created_at in rows:
        if folder_of(path) == folder:
            image_info = images.get((source, path)) if images else None
@@ -761,7 +854,10 @@ def _subtree_max(children: Sequence[KbTreeFolder | KbTreeFile]) -> str | None:
    """A node's ``updated_at`` (phase 106, D9): the subtree's MAX
    document ``created_at``, computed from the node's direct children —
    files contribute their ``created_at``, subfolder nodes contribute
-    their (already recursive) ``updated_at``.
+    their (already recursive) ``updated_at``. UNCHANGED for the
    expanded path since phase 167 (the truncated path uses the
    :func:`_folder_counts` max map instead — one concept, two
    derivations, cross-pinned equal by a unit test).
    All values share the same UTC ``isoformat()`` shape (the endpoint
    converts every ``created_at`` before the builder runs), so the
@@ -782,6 +878,7 @@ def build_kb_tree(
    doc_rows: Sequence[TreeDocRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
 ) -> list[KbTreeSource]:
    """The pure tree builder behind ``GET /api/docs/tree`` (phase 97,
    task 02) — module-level and DB-free so unit tests drive it
@@ -847,6 +944,17 @@ def build_kb_tree(
      pending, single-file folders included — and a registered
      0-document source never is.
    *depth* (phase 167, task 02, D2/A2) bounds how deep the tree is
    BUILT: ``None`` (the default) = unlimited — every pre-phase call
    and expectation is byte-identical; a positive budget stops the
    recursion (see :func:`_level_children`) so the response carries the
    sources plus *depth* levels, every node whose children were not
    built flagged ``truncated=True`` with its recursive count, its
    recursive ``updated_at`` (the max-date map — D9 survives
    truncation), its ``summary`` and ``summary_pending``. A SOURCE node
    is never truncated at ``depth ≥ 1`` — the view's top level is
    source → level 1, so the source's direct children are always built.
    ONE concept end to end: the builder reuses
    :func:`app.rag.folder_summaries.folder_of` and the phase-94
    existence / count rules, so — for a single-source dataset — its
@@ -872,10 +980,14 @@ def build_kb_tree(
        if name in listed:  # defensive: list_source_names dedupes
            continue
        listed.add(name)
-        tree.append(_source_node(name, by_source.get(name, ()), summaries, images))
+        tree.append(
            _source_node(name, by_source.get(name, ()), summaries, images, depth)
        )
    for source in sorted(by_source):
        if source not in listed:
-            tree.append(_source_node(source, by_source[source], summaries, images))
+            tree.append(
                _source_node(source, by_source[source], summaries, images, depth)
            )
    return tree
@@ -884,6 +996,7 @@ def _source_node(
    rows: Sequence[TreeFileRow],
    summaries: Mapping[tuple[str, str], str],
    images: Mapping[tuple[str, str], ImageDocInfo] | None = None,
    depth: int | None = None,
 ) -> KbTreeSource:
    """One source node (pure): whole-source count + the source-root
    summary + the root level's children (direct subfolders + direct
@@ -902,10 +1015,17 @@ def _source_node(
    ``updated_at`` (phase 106, D9): the source's subtree max —
    :func:`_subtree_max` over the root level's children; ``None`` for a
-    0-document source (no children, no dates).
+    0-document source (no children, no dates). Correct at every depth:
    a TRUNCATED child still carries its recursive ``updated_at`` (the
    max-date map), so the source's max never silently drops the levels
    below the budget (phase 167, D2). The SOURCE node itself is never
    marked ``truncated`` at ``depth ≥ 1`` — its direct children are
    always built (the view's top level).
    """
-    folders, counts = _folder_counts(rows)
+    folders, counts, max_dates = _folder_counts(rows)
-    children = _level_children(source, "", folders, counts, rows, summaries, images)
+    children = _level_children(
        source, "", folders, counts, max_dates, rows, summaries, images, depth
    )
    return KbTreeSource(
        name=source,
        documents=len(rows),
@@ -916,32 +1036,56 @@ def _source_node(
    )
-@router.get("/docs/tree", response_model=KbTree)
+def _kb_totals(doc_rows: Sequence[TreeDocRow]) -> KbTreeTotals:
-def list_kb_tree(
+    """The catalogue totals beside the tree (phase 167, task 02, D2) —
-    db: Session = Depends(get_db),  # noqa: B008
+    computed from the SAME rows the builder consumes (no extra query):
-    _admin: None = Depends(require_admin),  # noqa: B008
+    ``documents = len(rows)``, ``chunks`` = the sum of the per-document
-) -> KbTree:
+    chunk counts, ``last_indexed`` = the MAX ``indexed_at`` stamp
-    """The full recursive KB tree in ONE fetch (phase 97, task 02).
+    (lexicographic max = chronological max over one ISO-8601 shape),
    ``None`` for an empty KB. The RAG view's stat cards read these
    because a depth-limited tree cannot be summed client-side (the
    phase-97 ``treeStats()`` walk only sees the sent levels).
    """
    return KbTreeTotals(
        documents=len(doc_rows),
        chunks=sum(chunks for _s, _p, _t, chunks, _i, _c in doc_rows),
        last_indexed=max((indexed for _s, _p, _t, _ch, indexed, _c in doc_rows), default=None),
    )
    Admin-only, like ``GET /api/docs`` — anonymous callers get 403
    ``admin only`` (the RAG view's anonymous gate never fetches the
    tree). The RAG view drills CLIENT-side: this is the view's single
    fetch, zero per-level requests (the ``00_phase.md`` "The tree
    endpoint" contract).
-    Composition: the registry source names (``list_source_names`` —
+def _kb_catalogue(
-    the superset rule's registry half, imported from ``app.rag.agent``
+    db: Session,
-    exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped
+) -> tuple[
-    catalogue query ``GET /api/docs`` runs (the document ``id``
+    list[str],
-    excluded — the tree has no document ids) + ALL stored
+    list[TreeDocRow],
-    ``folder_summaries`` rows (a bounded select — one row per
+    dict[tuple[str, str], str],
-    existing folder at the ≥ 1-doc rule; rows for sources the tree
+    dict[tuple[str, str], ImageDocInfo],
-    does not list are never referenced by the builder) + the phase-122
+]:
-    (task 04) image-docs map (a second bounded select over the
+    """The catalogue composition the tree routes share (phase 167,
-    ``is_image`` rows with a servable ``image_path`` — empty for every
+    task 03) — ONE outerjoin/grouped catalogue query plus the two
-    pre-phase KB, so the response stays byte-identical to pre-phase)
+    bounded side selects, no N+1 anywhere (the drill pays exactly what
-    — through the pure :func:`build_kb_tree`. ``GET /api/docs`` itself
+    the first paint pays). Returns ``(names, doc_rows, summaries,
-    is untouched.
+    images)``:
    * *names* — the registry source names (``list_source_names`` — the
      superset rule's registry half, imported from ``app.rag.agent``
      exactly as ``app/api/chat.py`` does);
    * *doc_rows* — the SAME outerjoin/grouped catalogue query
      ``GET /api/docs`` runs (the document ``id`` excluded — the tree
      has no document ids), both stamps converted to ISO-8601 HERE so
      the pure builder stays DB-free;
    * *summaries* — ALL stored ``folder_summaries`` rows (a bounded
      select — one row per existing folder at the ≥ 1-doc rule; rows
      for sources the tree does not list are never referenced by the
      builder);
    * *images* — the phase-122 (task 04) image-docs map: a second
      bounded select over the ``is_image`` rows with a servable
      ``image_path`` only (a handful of rows at KB scale, never the
      whole catalog; the catalogue query above stays byte-identical).
      EMPTY for every pre-phase KB (no image rows), so the response
      stays byte-identical to pre-phase — the fields are row-driven,
      not toggle-driven (a surviving image doc keeps its thumbnail
      through a toggle-off sync, the prune guard's UX side).
    """
    names = list_source_names(db)
    rows = db.execute(
@@ -974,14 +1118,7 @@ def list_kb_tree(
            select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
        ).all()
    }
-    # Phase 122 (task 04): the image-docs affordance map — a bounded
+    images: dict[tuple[str, str], ImageDocInfo] = {
    # select over the ``is_image`` rows only (a handful of rows at KB
    # scale, never the whole catalog; the catalogue query above stays
    # byte-identical). EMPTY for every pre-phase KB (no image rows), so
    # the response stays byte-identical to pre-phase — the fields are
    # row-driven, not toggle-driven (a surviving image doc keeps its
    # thumbnail through a toggle-off sync, the prune guard's UX side).
    images: dict[tuple[str, str], tuple[str, str]] = {
        (source, path): (str(doc_id), summary)
        for source, path, doc_id, summary in db.execute(
            select(
@@ -992,4 +1129,163 @@ def list_kb_tree(
            ).where(Document.is_image.is_(True), Document.image_path.is_not(None))
        ).all()
    }
-    return KbTree(sources=build_kb_tree(names, doc_rows, summaries, images))
+    return names, doc_rows, summaries, images
 def _revalidated_json(request: Request, model: BaseModel) -> Response:
    """Serialize once, hash the EXACT served bytes, answer 304 when the
    client already holds them (phase 167, task 03, D3/A4 — the
    :mod:`app.core.http_cache` pattern in ONE place for the three
    catalogue reads).
    The client is ``fetch``: a matched conditional request is served
    from its cache transparently (fetch never surfaces the 304), and
    ``must-revalidate`` guarantees a sync or an admin edit is seen on
    the next load — the tag changes because the body changes (the A4
    proof: a ``sources_meta.version``-derived tag would keep serving
    stale trees after the folder-summary / document-date PATCHes, which
    never bump it). ``model_dump_json`` runs exactly ONCE — the same
    string feeds the tag and the body, and the returned ``Response``
    short-circuits FastAPI's own serialization.
    """
    body = model.model_dump_json()
    etag = weak_etag(body)
    if etag_matches(request.headers.get("if-none-match"), etag):
        return Response(status_code=304, headers=revalidate_headers(etag))
    return Response(
        content=body, media_type="application/json", headers=revalidate_headers(etag)
    )
@router.get("/docs/tree", response_model=KbTree)
 def list_kb_tree(
    request: Request,
    depth: int = Query(1, ge=1, le=8),
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
 ) -> Response:
    """The KB tree, DEPTH-LIMITED by default (phase 97, task 02;
    phase 167, task 02, D2/A2 — the TODO L5 2 MB payload fix).
    ``depth`` bounds the levels built under each source:
    ``1`` (default) = sources + ONE level — the RAG view's first paint;
    ``8`` = the practical KB ceiling (the whole tree, the pre-phase
    shape plus the additive ``totals`` key). Every node whose children
    were NOT built carries ``truncated: true`` (with its recursive
    count, recursive ``updated_at``, ``summary`` and
    ``summary_pending`` intact) so the view can drill level by level;
    ``totals`` (documents / chunks / ``last_indexed``) is computed here
    from the full catalogue rows, so the stat cards stay honest at any
    depth.
    Admin-only, like ``GET /api/docs`` — anonymous callers get 403
    ``admin only`` (the RAG view's anonymous gate never fetches the
    tree). The phase-97 "one fetch, drill client-side" contract is
    deliberately superseded (LOCKED A3): one initial load plus one
    fetch per drilled level (task 03's node route).
    Composition: :func:`_kb_catalogue` (the registry source names, the
    SAME outerjoin/grouped catalogue query ``GET /api/docs`` runs, ALL
    stored ``folder_summaries`` rows, and the phase-122 image-docs map)
    through the pure :func:`build_kb_tree` + :func:`_kb_totals`.
    ``GET /api/docs`` itself is untouched.
    Phase 167 (task 03, D3/A4): the response carries the content-exact
    weak ``ETag`` + ``Cache-Control: private, must-revalidate`` pair via
    :func:`_revalidated_json` — a repeat load costs a 304 with no body
    (the client is ``fetch``, which serves a matched conditional request
    from its cache transparently). The decorator keeps
    ``response_model=KbTree`` for the OpenAPI schema only — a returned
    :class:`~fastapi.responses.Response` short-circuits FastAPI's
    serialization, so the model is dumped exactly ONCE here.
    """
    names, doc_rows, summaries, images = _kb_catalogue(db)
    model = KbTree(
        sources=build_kb_tree(names, doc_rows, summaries, images, depth),
        totals=_kb_totals(doc_rows),
    )
    return _revalidated_json(request, model)
@router.get("/docs/tree/node", response_model=None)
 def get_kb_tree_node(
    request: Request,
    source: str = Query(...),
    path: str = Query(""),
    depth: int = Query(1, ge=1, le=8),
    db: Session = Depends(get_db),  # noqa: B008
    _admin: None = Depends(require_admin),  # noqa: B008
 ) -> KbTreeSource | KbTreeFolder | Response:
    """ONE tree node's direct children — the RAG view's per-level drill
    fetch (phase 167, task 03, D3/A3: the deliberate cost of the small
    first payload is one request per drilled level).
    ``source`` is the registry name OR an indexed source (the superset
    rule); ``path`` is SOURCE-RELATIVE, ``""`` (the default) = the
    source root. The node answers in the shape the full tree produces
    for the same node — a :class:`~app.schemas.KbTreeSource` for
    ``path == ""``, else a :class:`~app.schemas.KbTreeFolder` carrying
    the recursive ``documents`` count, the recursive ``updated_at``
    (the phase-106 D9 max — the truncation-proof max-date map), its
    ``summary`` and ``summary_pending`` (the unchanged phase-98 D3
    rule). The node's DIRECT children are always built — the *depth*
    budget only decides whether those children's subfolders recurse
    (the same rule that never truncates a SOURCE node at ``depth ≥ 1``,
    see :func:`_level_children`) — so the node itself is never
    ``truncated``: what arrived is what the level renders, and deeper
    levels ride the children's own flags.
    Admin-gated — the same :func:`~app.core.auth.require_admin` gate as
    ``/api/docs/tree`` (anonymous → 403 ``admin only``). 404
    ``{"detail": "unknown node"}`` when the source is NEITHER
    registered NOR indexed, or when a non-empty ``path`` fails the
    phase-94 existence rule (a folder exists ⟺ some indexed path
    starts with ``path + "/"`` — a document's OWN path is not a folder,
    unless a deeper path wears it).
    Row-only, like every route in this file: NOTHING here builds a
    filesystem path from ``path`` — the node is derived from catalogue
    ROWS alone, so a ``../``-style value is simply not a folder and 404s
    with no filesystem surface (the phase-10 traversal argument, reused
    by :func:`build_kb_tree`).
    Composition: the SAME catalogue query ``list_kb_tree`` runs (via
    :func:`_kb_catalogue` — ONE query per drill, no N+1; the tree's
    counts and recursive dates are only honest over the whole
    catalogue), then the pure helpers scoped to the one source. Carries
    the same ``ETag`` + ``private, must-revalidate`` pair as the other
    two catalogue reads (:func:`_revalidated_json`).
    ``response_model=None`` on the decorator: the return is a UNION
    annotated for the reader while the body is returned as a raw
    ``Response`` — the serializer must not run twice (once for the
    tag, once inside FastAPI).
    """
    names, doc_rows, summaries, images = _kb_catalogue(db)
    rows: list[TreeFileRow] = [
        (row_path, title, chunks, indexed_at, created_at)
        for row_source, row_path, title, chunks, indexed_at, created_at in doc_rows
        if row_source == source
    ]
    if source not in names and not rows:
        raise HTTPException(status_code=404, detail="unknown node")
    folders, counts, max_dates = _folder_counts(rows)
    if path and path not in folders:
        raise HTTPException(status_code=404, detail="unknown node")
    node: KbTreeSource | KbTreeFolder
    if path:
        children = _level_children(
            source, path, folders, counts, max_dates, rows, summaries, images, depth
        )
        node = KbTreeFolder(
            path=path,
            documents=counts[path],
            updated_at=max_dates[path],
            summary=summaries.get((source, path)),
            summary_pending=counts[path] >= MIN_DOCS_PER_FOLDER
            and (source, path) not in summaries,
            children=children,
        )
    else:
        # The source root — the very node the full tree builds.
        node = _source_node(source, rows, summaries, images, depth)
    return _revalidated_json(request, node)
@@ -46,6 +46,35 @@ class Settings(BaseSettings):
    log_level: str = "INFO"
    static_dir: str = "frontend"
    # --- HTTP response compression (phase 167, TODO L5) ---
    #: Byte floor for the selective gzip layer
    #: (``app/core/compression.py``, ``BOR_COMPRESS_MIN_BYTES``): responses
    #: of at least this many bytes are gzipped when the client accepts it
    #: (Starlette's ``GZipMiddleware`` owns the negotiation, the
    #: already-encoded skip and the ``Vary`` append). ``0`` = the feature
    #: is OFF — the middleware is not added at all (the kill switch);
    #: ``> 0`` = the Starlette minimum-size floor, default 860 — the
    #: classic nginx default, below which compression can INFLATE a body
    #: instead of shrinking it. A negative value is a typo — the validator
    #: fails startup loudly (the ``agent_max_rounds`` pattern). The SSE
    #: stream (``POST /api/chat``, A15) is never compressed at any value.
    compress_min_bytes: int = 860
    # --- Response byte budget (phase 167, TODO L5) ---
    #: INFO/DEBUG cut for the per-response ``api method=… bytes=…`` line
    #: (``app/core/response_budget.py``, ``BOR_LOG_LARGE_RESPONSE_BYTES``).
    #: ``/api/*`` responses of at least this many bytes (the bytes the ROUTE
    #: produced, before compression) log at INFO; below it the line is
    #: DEBUG, i.e. invisible at the default ``BOR_LOG_LEVEL=INFO`` so the
    #: log stays readable while "which endpoint is heavy, right now" is
    #: still one ``grep ' :: api '`` away. Default 65536 (64 KB) — the
    #: "fine" bar of the phase-167 audit; the audit's 256 KB budget is what
    #: marks an offender (LOCKED A5: a log line and a script, never a 413).
    #: ``0`` = log EVERY ``/api/*`` response at INFO (the loud mode, for a
    #: one-off investigation); a negative value is a typo — the validator
    #: fails startup loudly (the ``agent_max_rounds`` pattern).
    log_large_response_bytes: int = 65536
    # --- UI customization (phase 62, TODO L3) ---
    # Defaults are the phase-61 neutral copy — UNSET => byte-identical UI.
    # (Phase 91, task 03: the retired CSS-file theme env var is gone —
@@ -768,6 +797,29 @@ class Settings(BaseSettings):
            raise ValueError("chat_max_concurrent must be >= 1")
        return v
    @field_validator("compress_min_bytes")
    @classmethod
    def _compress_min_bytes_non_negative(cls, v: int) -> int:
        """``0`` is the compression kill switch (the middleware is not
        added at all) — a negative value is a typo (the
        ``agent_max_rounds`` pattern, phase 167)."""
        if v < 0:
            raise ValueError("compress_min_bytes must be >= 0 (0 = compression disabled)")
        return v
    @field_validator("log_large_response_bytes")
    @classmethod
    def _log_large_response_bytes_non_negative(cls, v: int) -> int:
        """``0`` is the loud mode (every ``/api/*`` response logs its byte
        line at INFO) — a negative value is a typo that would make the
        comparison constant, so it fails startup naming the field (the
        ``agent_max_rounds`` pattern, phase 167)."""
        if v < 0:
            raise ValueError(
                "log_large_response_bytes must be >= 0 (0 = log every /api/* response)"
            )
        return v
    @field_validator("docs_branch", "docs_base_branch")
    @classmethod
    def _docs_branch_tokens(cls, v: str, info: ValidationInfo) -> str:
@@ -0,0 +1,179 @@
 """Selective gzip for every HTTP response but the SSE chat stream (phase 167, D1, TODO L5).
 The defect (TODO L5): ``GET /api/docs/tree`` loads ~2 MB in production and
 the Sources page pays that on every sync and every shell boot. Nothing in
 this stack compressed — ``app/main.py`` added only Session / caching /
 security-headers, and no response ever carried ``Content-Encoding``. KB
 JSON is extremely repetitive (key names, paths, ISO timestamps), so
 gzipping it lands at roughly **8-12× smaller** on the wire: the same bytes
 the client already knows how to read, ~one order of magnitude faster.
 The SSE stream is EXCLUDED (A15, phase 167 LOCKED A1): ``POST /api/chat``
 is a ``text/event-stream`` whose whole value is that frames arrive *while
 the model is still talking*. Any middleware that buffers the response —
 ``BaseHTTPMiddleware`` is the classic offender, and the reason this module
 is **pure ASGI** — would deliver the entire turn at the end, silently
 breaking the typing animation, the tool lines, and the cancel surface. So
 the skip is by EXACT path (never a prefix: ``/api/chat-images/…`` keeps
 its compression) and the skipped scope is handed to the app untouched.
 The compression math is deliberately NOT re-implemented here. Starlette's
 :class:`~starlette.middleware.gzip.GZipMiddleware` owns ``Accept-Encoding``
 negotiation (a client that does not speak gzip gets identity bytes), the
 ``minimum_size`` floor, the skip when a response ALREADY sets
 ``content-encoding``, the ``Vary: Accept-Encoding`` append, and the
 streaming ``compressobj`` + final flush. It is also what makes the stack
 cooperate with a fronting proxy: **nginx may also gzip, but an
 already-encoded body is never double-encoded** — Starlette preserves the
 upstream ``Content-Encoding`` verbatim, and so does nginx.
 One seam IS patched here, and it is framing, not math: the
 ``CachingMiddleware`` below us is a ``BaseHTTPMiddleware``, and starlette
 1.x re-frames every response it emits upstream (``_StreamingResponse``
 ships each body chunk with ``more_body=True`` plus an empty terminator),
 so Starlette's buffered small-response exemption can never fire for
 anything that travelled through it — a 33-byte ``/api/health`` would
 stream into the compressor and come back INFLATED, exactly what the
 ``minimum_size`` floor exists to prevent. :class:`_RebufferSmallResponses`
 sits under the responder and hands it back the single buffered body
 message the exemption was written for, bounded to the floor.
 """
 from __future__ import annotations
 from starlette.datastructures import Headers
 from starlette.middleware.gzip import GZipMiddleware
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 #: Paths that are NEVER compressed, matched EXACTLY (never by prefix).
 #: ``POST /api/chat`` is the SSE stream (A15): frame-by-frame delivery is
 #: the entire point of the endpoint, and buffering the response to feed it
 #: through a compressor would deliver the whole turn at completion (and a
 #: never-ending stream never completes on a cancel). The exact match is
 #: load-bearing — ``/api/chat-images/…`` shares the prefix and must keep
 #: routing through the compression layer.
 DEFAULT_SKIP_PATHS: tuple[str, ...] = ("/api/chat",)
 class _RebufferSmallResponses:
    """Hold a declared-small body so Starlette's floor exemption fires.
    When ``http.response.start`` carries ``content-length`` below
    ``minimum_size`` — byte-for-byte the condition Starlette's buffered
    path checks — the messages are held (bounded by the floor) and
    re-emitted as ONE ``http.response.body`` with ``more_body=False``,
    undoing the ``BaseHTTPMiddleware`` re-framing so the responder's
    small-response exemption sees what it was written to see and the
    bytes stay identity (no ``Content-Encoding``, no ``Vary`` append).
    A lying header can only cost the buffer, never correctness: if the
    body outgrows the floor despite the header, everything held is
    forwarded as it came and the rest streams through the normal
    (compressing) path. Non-body messages while holding (e.g.
    ``http.response.pathsend``) also flush the hold untouched.
    """
    def __init__(self, app: ASGIApp, *, minimum_size: int) -> None:
        self.app = app
        self.minimum_size = minimum_size
    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        holding = False
        armed = True  # the start decision happens at most once per response
        held: list[Message] = []
        total = 0
        async def send_filter(message: Message) -> None:
            nonlocal holding, armed, total
            message_type = message["type"]
            if message_type == "http.response.start" and armed:
                armed = False
                length = Headers(scope=message).get("content-length")
                if (
                    length is not None
                    and length.isdigit()
                    and int(length) < self.minimum_size
                ):
                    held.append(message)
                    holding = True
                    return
                await send(message)
                return
            if holding and message_type == "http.response.body":
                held.append(message)
                total += len(message.get("body", b""))
                if not message.get("more_body", False):
                    # Whole body in hand, still under the floor: re-emit
                    # it as the single buffered message the exemption wants.
                    start = held[0]
                    merged = b"".join(m.get("body", b"") for m in held[1:])
                    holding = False
                    held.clear()
                    await send(start)
                    await send({"type": "http.response.body", "body": merged, "more_body": False})
                elif total >= self.minimum_size:
                    # The content-length lied — stop holding, stream on.
                    holding = False
                    for held_message in held:
                        await send(held_message)
                    held.clear()
                return
            if holding:
                # pathsend or anything unexpected mid-hold: flush as-is.
                holding = False
                for held_message in held:
                    await send(held_message)
                held.clear()
            await send(message)
        await self.app(scope, receive, send_filter)
 class SelectiveGZipMiddleware:
    """Pure-ASGI selective gzip (phase 167, D1 — no ``BaseHTTPMiddleware``).
    Holds the wrapped ``app`` and a composed
    :class:`~starlette.middleware.gzip.GZipMiddleware` built over the SAME
    app (via the :class:`_RebufferSmallResponses` seam, so the floor still
    works under the caching layer's re-framing); ``__call__`` delegates to
    the gzip path unless the scope is not HTTP (websocket / lifespan pass
    straight through) or the request path is one of ``skip_paths`` — in
    which case the app is called DIRECTLY, with the original
    ``receive``/``send``, so a streaming response reaches the client frame
    by frame, uncompressed.
    ``minimum_size`` is the Starlette floor (bytes): responses smaller than
    it are forwarded without compression because gzip can make them
    bigger. It is supplied by the caller (``Settings.compress_min_bytes``;
    ``0`` removes this middleware entirely at wiring time — see
    ``app/main.py``). The SSE stream (``POST /api/chat``, A15) is never
    compressed at any value.
    """
    def __init__(
        self,
        app: ASGIApp,
        *,
        minimum_size: int,
        skip_paths: tuple[str, ...] = DEFAULT_SKIP_PATHS,
    ) -> None:
        self.app = app
        self.minimum_size = minimum_size
        self.skip_paths = tuple(skip_paths)
        # The compression itself stays Starlette's: negotiation, floor,
        # already-encoded skip, Vary append, streaming compressobj + flush.
        self.gzip = GZipMiddleware(
            _RebufferSmallResponses(app, minimum_size=minimum_size),
            minimum_size=minimum_size,
        )
    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        if scope["type"] != "http" or scope["path"] in self.skip_paths:
            # Non-HTTP scopes have no response to compress; the SSE stream
            # must never see this layer (A15 — see DEFAULT_SKIP_PATHS).
            await self.app(scope, receive, send)
            return
        await self.gzip(scope, receive, send)
@@ -0,0 +1,112 @@
 """HTTP revalidation validators for the read-only catalogue reads
 (phase 167, task 03, D3 — LOCKED A4).
 Three pure functions, no I/O, ONE place for the pattern (the routes
 compose them; the alternative was a copy per route):
 * :func:`weak_etag` — the validator: a weak ``ETag`` over the EXACT
  bytes being served,
 * :func:`etag_matches` — the tolerant ``If-None-Match`` evaluator,
 * :func:`revalidate_headers` — the header pair a validated response
  must carry.
 **Why a content hash, not ``sources_meta.version``** (LOCKED A4):
 bumping that generation marks every saved chat ``stale`` (phase 53) —
 the wrong tool for a payload revalidation — and the admin PATCHes
 (``/api/documents/summary``, ``/api/documents/date``,
 ``/api/folders/summary``) mutate the tree WITHOUT bumping it, so a
 version-derived tag would serve stale trees. A hash of the serialized
 body is content-exact by construction: the tag changes ⟺ the body
 changes, it needs no migration, and the tree is built anyway (the hash
 costs microseconds next to the query). The tag is over the EXACT served
 bytes — the endpoint serializes once and hands those bytes to both the
 hash and the ``Response``.
 **Why a truncation to 32 hex chars is safe**: 32 hex chars of SHA-256
 is this file's truncation convention. Collisions are not a correctness
 risk for a revalidated cache: a false match only ever costs one stale
 repaint on a ~2 MB namespace (the next unconditional load corrects it),
 while 128 bits of digest make even that astronomically unlikely.
 A19 SCOPE (:func:`revalidate_headers`, LOCKED A4 as a recorded, scoped
 extension of A19): HTML pages stay ``no-cache`` with NO validators (the
 caching middleware strips inbound conditionals and outbound validators —
 the phase-54 contract), ``/assets/*`` stay ``immutable`` for a year
 (URL-versioned), and this module governs the READ-ONLY API arm only —
 today the two KB-tree routes and ``GET /api/docs``. The SSE stream and
 every mutating route are untouched.
 """
 from __future__ import annotations
 import hashlib
 #: Hex chars of the SHA-256 digest the tag carries (see the module
 #: docstring's collision argument).
 _TAG_HEX_CHARS = 32
 #: The ``Cache-Control`` paired with every validator here: ``private``
 #: (no shared cache may store the admin-gated catalogue) +
 #: ``must-revalidate`` (a sync or an admin edit is seen on the NEXT
 #: load — the client may never serve these bytes without asking, which
 #: is what distinguishes this from the ``/assets/*`` posture).
 REVALIDATE_CACHE_CONTROL = "private, must-revalidate"
 def weak_etag(body: str | bytes) -> str:
    """Weak validator over the exact served bytes: ``W/"<sha256 hex[:32]>"``.
    Content-exact by construction (the endpoint hashes the SAME string
    it returns — see the module docstring for the A4 argument and the
    truncation convention). ``str`` bodies are hashed as UTF-8, so the
    str and its ``bytes`` form produce the SAME tag. Weak (``W/``)
    because the comparison is byte-equivalence under revalidation, not
    bit-identity — a differently gzipped encoding of the same content
    must still match (RFC 9110 §8.8.3).
    """
    raw = body.encode("utf-8") if isinstance(body, str) else body
    return f'W/"{hashlib.sha256(raw).hexdigest()[:_TAG_HEX_CHARS]}"'
 def _strip_weakness(candidate: str) -> str:
    """One comparable form: trimmed, with the ``W/`` weakness marker gone
    (weak/strong equivalence — ``W/"x"`` matches ``"x"`` in BOTH
    directions, RFC 9110 §8.8.3.2's weak comparison rule applied in both
    directions on purpose: our tag is over the exact bytes, so a client
    that stored either form HAS these bytes)."""
    value = candidate.strip()
    if value[:2].upper() == "W/":
        value = value[2:].strip()
    return value
 def etag_matches(header: str | None, tag: str) -> bool:
    """Does an inbound ``If-None-Match`` header value match our ``tag``?
    Tolerant the way real clients send it: ``*`` matches (RFC 9110
    §13.3.3 — the client holds nothing it trusts), a comma-separated
    candidate list matches on ANY entry, weak/strong forms are
    equivalent in both directions, and surrounding whitespace is
    ignored. ``None`` or an empty/whitespace-only header never matches
    (a request with no conditional header is an unconditional GET → 200).
    """
    if not header or not header.strip():
        return False
    candidates = header.split(",")
    if any(candidate.strip() == "*" for candidate in candidates):
        return True
    target = _strip_weakness(tag)
    return any(_strip_weakness(candidate) == target for candidate in candidates)
 def revalidate_headers(etag: str) -> dict[str, str]:
    """The ONE header pair every validated catalogue response carries —
    on the 200 AND on the 304 (RFC 9110 §8.8.3: a 304 must repeat the
    validator, or the client cannot extend the stored entry's life).
    A19 scope (module docstring): HTML pages stay ``no-cache`` with no
    validators, ``/assets/*`` stay ``immutable``; this pair is the
    read-only API arm — ``private, must-revalidate`` so a sync or an
    admin edit is seen on the next load, because the tag changes exactly
    when the body changes.
    """
    return {"ETag": etag, "Cache-Control": REVALIDATE_CACHE_CONTROL}
@@ -0,0 +1,211 @@
 """The byte budget: ONE log line per ``/api/*`` response (phase 167, task 05, D5, LOCKED A5).
 TODO L5 asked "which endpoint is heavy, right now, in production?" — and
 before this phase the app could not answer it. The 2 MB
 ``GET /api/docs/tree`` defect was found by a human eyeballing a browser
 network tab; every other candidate endpoint was a guess. This module makes
 the answer a ``grep`` away:
 .. code-block:: text
    api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41
 * ``bytes`` is the size of the body the ROUTE produced — the sum of the
  ``http.response.body`` chunk lengths as they pass through, uncompressed
  even when the wire carried gzip (the compression layer above us encodes
  AFTER we counted). ``enc`` reports what actually went on the wire
  (``content-encoding``, else ``identity``). The two together answer "how
  fat is it, and how fat did the client see it".
 * level: ``INFO`` when ``bytes >= settings.log_large_response_bytes``
  (``BOR_LOG_LARGE_RESPONSE_BYTES``, default 65536; ``0`` = every ``/api/*``
  response at INFO), ``DEBUG`` below it — so the default log stays quiet
  and an offender is loud without a re-deploy.
 * a ``304`` logs ``bytes=0`` deliberately: the revalidation win this phase
  bought is visible in the log, not hidden behind a missing line.
 * `` stream=1`` is appended for ``text/event-stream`` responses, where
  ``bytes`` is the streamed total and ``duration_ms`` is the stream's LIFE.
 Why **pure ASGI** (the phase-167 rule for anything on this path): the
 middleware must never buffer. It only reads message lengths, so the SSE
 chat stream (A15) passes through byte-identical — a
 ``BaseHTTPMiddleware`` here would deliver the whole turn at the end,
 which is exactly the failure mode phase 167 exists to avoid elsewhere.
 Logged on the ``http.response.body`` message with ``more_body=False`` —
 for a stream that is when the **stream ends**, so ``duration_ms`` is the
 stream's whole life. A stream the client walks away from (a cancelled SSE
 turn) writes exactly one line too, with the bytes that actually got out:
 the app unwinds without a completing body message and the ``finally`` path
 emits it. Half a response happened, so half a count is the honest number —
 and no line at all would make a chatty, abandoned poll look free.
 An app exception is handled the same way (one line, bytes as counted, status
 500 when the app never got to answer) and re-raises UNCHANGED; non-HTTP
 scopes and non-``/api/`` paths pass through silently (pages, assets and the
 static catch-all are not the catalogue surface this budget governs).
 """
 from __future__ import annotations
 import logging
 import time
 from starlette.datastructures import Headers
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 from app.config import get_settings
 #: The logger behind every line — renders as ``… :: api method=…`` with the
 #: §9 format, so ``grep ' :: api '`` isolates the budget log.
 logger = logging.getLogger("app.response_budget")
 #: Only paths under this prefix are measured. The admin catalogue (and
 #: whatever new read route joins it) is the surface TODO L5's "any any
 #: other endpoints that could load a ton of data" is about; pages, assets
 #: and the static catch-all are not.
 API_PATH_PREFIX = "/api/"
 #: The media type whose line carries `` stream=1`` (A15 SSE): the frame
 #: stream is where ``bytes`` is a total over a life rather than one body.
 STREAM_MEDIA_TYPE = "text/event-stream"
 #: Status logged when the app died before it could send
 #: ``http.response.start`` — the exception is propagating and Starlette's
 #: ``ServerErrorMiddleware`` (outermost) answers exactly that.
 UNKNOWN_STATUS = 500
 class _Tally:
    """What one measured response accumulates: status, the start message,
    bytes.
    ``start`` holds the ``http.response.start`` MESSAGE (the dict), not a
    snapshot of its headers, and its headers are read at emit time — after
    the final body chunk has been awaited. That is how ``enc`` reports the
    wire encoding from a layer sitting UNDER the compressor (the position
    that gives an uncompressed ``bytes=``): Starlette's gzip responder holds
    that same start message while it decides what to do with the body and
    decorates it in place (``MutableHeaders(raw=initial_message["headers"])``
    shares the list), so by the time the body completes the
    ``content-encoding`` the client sees is on it. Holding the message
    rather than the list is not pedantry — ``Headers(scope=message)``
    REBINDS ``message["headers"]`` to a fresh list (the compression
    responder and this stack's ``_RebufferSmallResponses`` both call it
    that way), so a captured list would be a stale one.
    The dependency is documented and pinned, not assumed:
    ``tests/integration/test_response_budget_api.py`` reads the real
    middleware stack for both ``enc=gzip`` and ``enc=identity``, so a
    future layer that copies the message instead of decorating it fails a
    test rather than quietly mislabeling the log (the byte count stays
    exact either way — it is the number the budget compares).
    """
    __slots__ = ("bytes", "logged", "start", "status")
    def __init__(self) -> None:
        self.bytes = 0
        self.status = UNKNOWN_STATUS
        self.start: Message | None = None
        self.logged = False
    def line(self, scope: Scope, started: float) -> str:
        """The §9-style single line — the exact shape README documents.
        ``duration_ms`` is rounded from the monotonic clock (never
        wall-clock: an NTP step must not produce a negative duration).
        """
        headers = Headers(raw=list(self.start["headers"]) if self.start else [])
        enc = (headers.get("content-encoding") or "identity").strip() or "identity"
        media = (headers.get("content-type") or "").partition(";")[0].strip().lower()
        duration_ms = int(round((time.monotonic() - started) * 1000))
        line = (
            f"api method={scope.get('method', '')} path={scope.get('path', '')} "
            f"status={self.status} bytes={self.bytes} enc={enc} "
            f"duration_ms={duration_ms}"
        )
        if media == STREAM_MEDIA_TYPE:
            line += " stream=1"
        return line
    def emit(self, scope: Scope, started: float, budget: int) -> None:
        """Write the line ONCE, at the level the budget decides.
        Idempotent on purpose: the completion path and the ``finally``
        path describe the same response and must never double-log it.
        """
        if self.logged:
            return
        self.logged = True
        line = self.line(scope, started)
        if self.bytes >= budget:
            logger.info(line)
        else:
            logger.debug(line)
 class ResponseBudgetMiddleware:
    """Pure-ASGI response-size logger — zero buffering, SSE-safe.
    Wraps ``send`` only (never ``receive``): the timing runs from
    ``time.monotonic()`` at entry to the ``more_body=False`` completion,
    so ``duration_ms`` is the response's (or the stream's) whole life
    without the middleware needing to see one inbound message. Every
    ``http.response.body`` chunk's length adds to the count and nothing is
    held, so a stream still flows frame by frame.
    ``threshold`` overrides the INFO/DEBUG cut for this instance; the
    default (``None``) reads ``Settings.log_large_response_bytes`` per
    response, so ``app/main.py`` wires it with no arguments and
    ``BOR_LOG_LARGE_RESPONSE_BYTES`` stays the operator's knob (``0`` = log
    every ``/api/*`` response at INFO, negative refuses startup in the
    validator). LOCKED A5: this is a log line and a script, never a 413 —
    no endpoint starts rejecting large payloads here.
    """
    def __init__(self, app: ASGIApp, *, threshold: int | None = None) -> None:
        self.app = app
        self.threshold = threshold
    def _budget(self) -> int:
        if self.threshold is not None:
            return self.threshold
        return get_settings().log_large_response_bytes
    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        if scope["type"] != "http" or not str(scope.get("path", "")).startswith(
            API_PATH_PREFIX
        ):
            # Nothing to measure: not HTTP (websocket/lifespan) or not the
            # API surface. Silent pass-through — the original send.
            await self.app(scope, receive, send)
            return
        started = time.monotonic()
        tally = _Tally()
        async def send_counting(message: Message) -> None:
            message_type = message["type"]
            if message_type == "http.response.start":
                tally.status = int(message["status"])
                # The MESSAGE, not a header snapshot — see _Tally: the
                # layers above decorate this same dict in place, so reading
                # it after the body completes yields the wire encoding.
                tally.start = message
            elif message_type == "http.response.body":
                tally.bytes += len(message.get("body", b""))
                if not message.get("more_body", False):
                    # Await FIRST: the layers above decorate the start
                    # headers while handling this final chunk, so `enc`
                    # reads what actually went on the wire.
                    await send(message)
                    tally.emit(scope, started, self._budget())
                    return
            await send(message)
        try:
            await self.app(scope, receive, send_counting)
        finally:
            # A normal response already emitted at its completion; this
            # covers an app that raised (or returned) without finishing a
            # body. The exception itself re-raises UNCHANGED.
            tally.emit(scope, started, self._budget())
@@ -45,8 +45,10 @@ from app.api.ui_settings import router as ui_settings_router
 from app.config import get_settings
 from app.core.auth import ensure_admin_configured
 from app.core.caching import configure_caching
 from app.core.compression import SelectiveGZipMiddleware
 from app.core.debugging import configure_debugging
 from app.core.logging import configure_logging
 from app.core.response_budget import ResponseBudgetMiddleware
 from app.core.security_headers import SecurityHeadersMiddleware
 configure_logging()
@@ -145,6 +147,34 @@ def create_app() -> FastAPI:
    # stream in particular — passes through untouched.
    configure_caching(app)
    # Phase 167 (task 05, TODO L5): the byte budget — ONE
    # ``api method=… path=… status=… bytes=… enc=… duration_ms=…`` line per
    # /api/* response (INFO at/over BOR_LOG_LARGE_RESPONSE_BYTES, DEBUG
    # below), so "which endpoint is heavy, right now" is a
    # ``grep ' :: api '`` away. Position: added BEFORE the gzip call below,
    # i.e. INSIDE it (SecurityHeaders → gzip → response-budget → caching →
    # session → routes), so ``bytes=`` is the bytes the ROUTE produced
    # (uncompressed) while ``enc=`` reports what went on the wire — the two
    # together answer "how fat is it, and how fat did the client see it".
    # Pure ASGI (never BaseHTTPMiddleware): the SSE stream is counted as it
    # passes and logged once, when the stream ends.
    app.add_middleware(ResponseBudgetMiddleware)
    # Phase 167 (TODO L5): selective gzip — every response except the
    # POST /api/chat SSE stream (skipped by EXACT path, A15) compresses
    # when it is >= compress_min_bytes (the 2 MB /api/docs/tree defect).
    # 0 = the feature is OFF: no middleware is added at all (the
    # kill-switch convention). Position: added AFTER configure_caching but
    # BEFORE the SecurityHeaders line below, so SecurityHeaders stays
    # OUTERMOST (A20, the last add_middleware is outermost) and gzip sits
    # outside the caching layer — an already-rewritten HTML page compresses
    # too, and /api/* still passes the caching layer byte-identically
    # before compression.
    if settings.compress_min_bytes > 0:
        app.add_middleware(
            SelectiveGZipMiddleware, minimum_size=settings.compress_min_bytes
        )
    # Phase 82: security headers (CSP + no-framing + nosniff) — outermost on
    # purpose (last add_middleware): every response carries them, including
    # the static catch-all's 404s below (audit SEC-04).
@@ -522,6 +522,17 @@ class KbTreeFolder(BaseModel):
    A 0-document folder cannot exist (a folder is a catalogue prefix
    only), so every existing folder with no stored row is pending —
    single-file folders included.
    ``truncated`` (phase 167, D2/A2): true iff the depth-limited build
    STOPPED here — the node's own children were not sent, so the RAG
    view drills via ``GET /api/docs/tree/node`` instead of reading
    ``children``. A truncated node still carries every field the level
    row renders: the recursive ``documents`` count, the recursive
    ``updated_at`` (the phase-106 D9 max — kept correct without the
    children), its ``summary``, and ``summary_pending``. Omitted from
    the wire when ``False`` (the :func:`_drop_image_fields` omission
    precedent) — an untruncated node's wire shape stays byte-identical
    to pre-phase-167.
    """
    kind: Literal["folder"] = "folder"
@@ -534,6 +545,13 @@ class KbTreeFolder(BaseModel):
    summary: str | None = None
    summary_pending: bool = False
    children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
    #: Phase 167 (task 02, D2/A2) — the children were not built at this
    #: depth (see the class docstring); OMITTED when ``False``.
    truncated: bool = False
    @model_serializer(mode="wrap")
    def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
        return _drop_truncated_when_false(self, handler)
 class KbTreeSource(BaseModel):
@@ -560,6 +578,14 @@ class KbTreeSource(BaseModel):
    A registered 0-document source is never pending (there is nothing
    to summarize); a single-document source IS pending while its root
    row is absent.
    ``truncated`` (phase 167, D2/A2) exists on the shape for symmetry
    with :class:`KbTreeFolder` (and the task-03 node route), but the
    depth budget never marks a SOURCE node truncated at ``depth ≥ 1``:
    the top level of the RAG view is source → level 1, so the source's
    direct children are always built. Omitted from the wire when
    ``False`` — the source node's wire shape stays byte-identical to
    pre-phase-167.
    """
    name: str
@@ -571,14 +597,52 @@ class KbTreeSource(BaseModel):
    summary: str | None = None
    summary_pending: bool = False
    children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
    #: Phase 167 (task 02, D2/A2) — never true at ``depth ≥ 1`` (see the
    #: class docstring); OMITTED from the wire when ``False``.
    truncated: bool = False
    @model_serializer(mode="wrap")
    def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
        return _drop_truncated_when_false(self, handler)
 class KbTreeTotals(BaseModel):
    """Catalogue-wide totals the RAG view's three stat cards read
    (phase 167, task 02, D2/A2).
    A depth-limited tree CANNOT be summed client-side — the phase-97
    ``treeStats()`` walk only sees the sent levels, so a truncated
    tree would under-count every card. The endpoint therefore computes
    the totals from the FULL catalogue rows it already fetched (no
    extra query) and ships them beside the tree.
    """
    #: Every indexed document (the whole catalogue, not one level).
    documents: int = Field(default=0, ge=0)
    #: Sum of the per-document chunk counts (content + ``is_summary``
    #: chunks — the same numbers ``GET /api/docs`` reports).
    chunks: int = Field(default=0, ge=0)
    #: The newest ``indexed_at`` stamp across the catalogue (ISO-8601),
    #: ``None`` for an empty KB.
    last_indexed: str | None = None
 class KbTree(BaseModel):
    """Response of ``GET /api/docs/tree`` (phase 97, task 02).
-    The FULL recursive tree in ONE fetch — the RAG view (admin) drills
+    Since phase 167 (task 02, LOCKED A2/A3 — supersedes the phase-97
-    client-side, zero per-level fetches (the ``00_phase.md`` "The tree
+    "full recursive tree in one fetch, drill client-side" contract) the
-    endpoint" contract).
+    endpoint is DEPTH-LIMITED by default (``depth=1``: the sources plus
    ONE level); a node whose children were not built carries
    ``truncated: true`` and the client drills level by level. An
    explicit ``depth`` up to 8 still returns the whole tree, and the
    pure builder's ``depth=None`` default keeps every pre-phase
    response byte-identical.
    ``totals`` (phase 167, additive top-level key) is the catalogue's
    true documents/chunks/last-indexed — the stat cards stay honest at
    every depth because the server, not the (partial) tree walk,
    computes them.
    Every SOURCE and FOLDER node carries ``summary_pending`` (phase
    98, D3): true iff its recursive document count ≥
@@ -592,6 +656,9 @@ class KbTree(BaseModel):
    """
    sources: list[KbTreeSource]
    #: Phase 167 (task 02): the server-computed catalogue totals beside
    #: any depth of tree (see :class:`KbTreeTotals`).
    totals: KbTreeTotals = Field(default_factory=KbTreeTotals)
 class DocContent(BaseModel):
@@ -1560,6 +1627,21 @@ def _drop_image_fields(model: KbTreeFile, handler: SerializerFunctionWrapHandler
    return data
 def _drop_truncated_when_false(
    model: KbTreeFolder | KbTreeSource, handler: SerializerFunctionWrapHandler
 ) -> Any:
    """The phase-167 (task 02, D2) truncation-omission rule for the
    :class:`KbTreeFolder` / :class:`KbTreeSource` nodes: only a node whose
    children were NOT built adds the ``truncated: true`` key — an
    untruncated node's wire shape stays byte-identical to pre-phase-167
    (every existing wire pin holds), the deliberate echo of the
    :func:`_drop_image_fields` omission idiom."""
    data = handler(model)
    if not model.truncated:
        data.pop("truncated", None)
    return data
 def _drop_absent_share_url(model: BaseModel, handler: SerializerFunctionWrapHandler) -> Any:
    """The ``share_url`` omission rule (phase 51, task 02): ``None`` →
    ABSENT from the JSON (not ``"share_url": null``) — an unshared chat
@@ -52,12 +52,14 @@
 *
 * Phase 97 (task 04) — the catalog becomes the DRILL-DOWN TREE the
 * agent's `ls` sees (the phase-94 concept, one end to end). The load
- * is now `loadTree()`: ONE fetch of GET /api/docs/tree returns the
+ * is `loadTree()` (sources → folders → files, each folder carrying
 * FULL recursive tree (sources → folders → files, each folder carrying
 * its STORED description — the rows the agent reads), and the view
- * drills CLIENT-SIDE from that single fetch — zero per-level fetches,
+ * drills level by level with no URL change (the navigation is a
- * no URL change (the navigation is a re-render of the already-fetched
+ * re-render). PHASE 167 SUPERSEDED the single-payload model here
- * tree).
+ * (TODO L5 — the 2 MB first load, LOCKED A3): ONE
 * `GET /api/docs/tree?depth=1` plus ONE node fetch per not-yet-loaded
 * level — see the Phase 167 (task 04) section; the drill UX itself
 * (breadcrumb, scroll panel, editing) is unchanged.
 *
 *   • the drill state is module-scoped: `current` —
 *     { source: null, folder: null } = the top level (the rows ARE the
@@ -76,8 +78,11 @@
 *     the same-page modal; the no-JS href escape hatch intact), hidden
 *     when the level has none (at the top level it is ALWAYS hidden —
 *     files are seen per source, as with ls(source)).
- *   • the stat cards walk the WHOLE tree (document count, chunks sum,
+ *   • the stat cards show the KB-wide numbers (document count, chunks
- *     max indexed_at) — the values identical to the former flat walk.
+ *     sum, max indexed_at) — identical to the former flat walk. Since
 *     phase 167 they read the server's `totals` (a depth-limited tree
 *     cannot be summed client-side); the whole-tree walk stays only as
 *     the defensive no-`totals` fallback.
 *   • the empty state (#sources-empty) is now the "zero SOURCES"
 *     semantic (nothing registered, nothing indexed) — the deliberate
 *     phase-97 change: a registered 0-document source renders its row
@@ -423,6 +428,35 @@
 * control's immediate disable keeps the never-stale contract). The
 * running label, the button title, and the button aria-label are
 * byte-identical to pre-phase; no copy changed.
 *
 * Phase 167 (task 04, D4/A3 — TODO L5) — the LAZY DRILL: the 2 MB
 * first payload is gone. The catalog load is ONE
 * `fetch("/api/docs/tree?depth=1")` (the sources plus one level plus
 * the server's `totals`), and a folder the payload only NAMES
 * (`node.truncated === true`, not yet in `levelCache`) fetches
 * `GET /api/docs/tree/node?source=…&path=…` the FIRST time it opens —
 * the merged children ride the node (Object.assign, so
 * `currentLevelNode()`/`renderLevel()` keep their pre-phase semantics
 * for every loaded level), the level is cached (key
 * `source + "/" + folder`), and re-opening it is free (zero network).
 * A NEW tree load clears the cache — the sync/upload/re-show refresh
 * call sites keep working unchanged — and rehydrates the user's OWN
 * level when the fresh payload truncates it (a refresh never paints a
 * half-empty level). §7.4 never-stale, in full: while the node fetch
 * runs the level the user is LEAVING stays rendered, the clicked link
 * carries aria-busy + aria-disabled (drillPending — a second click
 * cannot start a parallel fetch; adopted popstate navigations
 * rehydrate their level the same way, via ensureCurrentLevel) and
 * ONE role="status"
 * "Loading <folder>…" row sits in the folders panel; a failure shows
 * a role="alert" line with a real Retry <button> and keeps the
 * current level rendered; the phase-79 loadSeq token EXTENDS to the
 * drill (a response superseded by a newer load — or by a navigation —
 * never paints). The node route's 404 IS the vanished-folder signal:
 * the honest reset to the top level, nothing cached. The existence
 * rules stop walking a full tree — folderKnownIn() only claims what
 * the HELD levels prove — and treeStats() prefers kbTree.totals (the
 * walk is the pre-phase-167 fallback only).
 */
 import { fetchIsAdmin } from "./header.js";
@@ -1622,11 +1656,30 @@ export async function mount(root) {
  let loadSeq = 0;
  let levelEditor = null; // phase 97 (task 05): the level editor handle
  /* Phase 167 (task 04, D4/A3 — the TODO L5 2 MB fix): the client no
     longer holds the whole tree. `levelCache` marks the folder levels
     the drill has already fetched (key: source + "/" + the
     source-relative folder path); the merged children live ON THE
     NODES (Object.assign in fetchLevel), so currentLevelNode() and
     renderLevel() keep their pre-phase semantics for every loaded
     level. `drillPending` is the in-flight drill { source, folder } —
     a second drill click is a no-op while a node fetch runs (§7.4:
     every settle path re-enables the controls). `navSeq` bumps on
     every applyTarget so a navigation invalidates an in-flight drill
     exactly like a newer load does. A NEW tree load clears the cache
     (the sync/refresh call sites keep working unchanged) and any
     in-flight drill with it. */
  const levelCache = new Map();
  let drillPending = null;
  let drillRowEl = null; // the ONE in-flight row (loading OR alert)
  let navSeq = 0;
  async function loadTree() {
    const my = ++loadSeq;
    drillPending = null; // any in-flight drill is invalidated by this load
    let r;
    try {
-      r = await fetch("/api/docs/tree");
+      r = await fetch("/api/docs/tree?depth=1");
    } catch {
      if (my === loadSeq) renderEmpty();
      return;
@@ -1638,14 +1691,25 @@ export async function mount(root) {
    const tree = await r.json();
    if (my !== loadSeq) return; // a newer load owns the DOM now
    kbTree = tree && Array.isArray(tree.sources) ? tree : { sources: [] };
    levelCache.clear(); // a fresh tree invalidates every drilled level
    resetVanishedLocation();
    // Phase 167: the user's own level may be one the fresh depth-1
    // payload only NAMES (truncated, cache just cleared) — rehydrate
    // it through the node route BEFORE it paints, so a refresh never
    // shows a half-empty level.
    if (needsLevelFetch(currentLevelNode(), current.source)) {
      enterFolder({ source: current.source, folder: current.folder }, null, false);
      return;
    }
    renderLevel();
  }
  /* Never-stale (PLAN §7.4, phase 97): after a re-fetch, the drilled
   * location may no longer exist in the NEW tree (the source was
   * unregistered/pruned, the folder vanished) — reset to the top level
-   * BEFORE rendering: no stale breadcrumb, no stale block. */
+   * BEFORE rendering: no stale breadcrumb, no stale block. Since
   * phase 167 the check is folderKnownIn (what the HELD levels prove)
   * — the fetch path's own vanished-signal is the node route's 404. */
  function resetVanishedLocation() {
    if (current.source === null) return;
    const src = kbTree.sources.find((s) => s.name === current.source);
@@ -1653,39 +1717,48 @@ export async function mount(root) {
      current = { source: null, folder: null };
      return;
    }
-    if (current.folder !== null && !folderExistsIn(src, current.folder)) {
+    if (current.folder !== null && !folderKnownIn(src, current.folder)) {
      current = { source: null, folder: null };
    }
  }
-  /* The phase-94 existence rule, mirrored client-side over the tree's
+  /* Phase 167 (task 04, D4): the client no longer holds the whole
-   * file paths: a folder exists under the source iff some of the
+   * tree, so the phase-97 full-tree existence walk becomes KNOWN-ness:
-   * source's indexed paths starts with `folder + "/"` (the source root
+   * a folder is known iff every node of its ancestor chain exists in
-   * — `""` — always exists). */
+   * what the client holds (the depth-1 tree + the drilled levels —
-  function folderExistsIn(src, folderPath) {
+   * merged children ride the nodes, so one walk over the held children
-    if (folderPath === "") return true;
+   * answers). A broken chain means the folder is gone OR simply not
-    const paths = [];
+   * loaded that deep — for the never-stale RESET both are honestly
-    const collect = (node) => {
+   * "back to the top level". The drill path does not guess: the node
-      for (const child of node.children || []) {
+   * route's 404 IS the "the folder vanished" signal (fetchLevel). */
-        if (child.kind === "file") paths.push(child.path);
+  function folderKnownIn(src, folderPath) {
-        else collect(child);
+    if (folderPath === "") return true; // the source root always exists
      }
    };
    collect(src);
    return paths.some((p) => p.startsWith(folderPath + "/"));
  }
  /* The node for `current`: the source node at the root, or the folder
   * node found by walking the tree with the folder's cumulative
   * source-relative path (the builder's folder paths are
   * source-relative, so `one/two` resolves one level at a time). */
  function currentLevelNode() {
    const src = kbTree.sources.find((s) => s.name === current.source);
    if (!src) return null;
    if (current.folder === null || current.folder === "") return src;
    let node = src;
    let acc = "";
-    for (const part of current.folder.split("/")) {
+    for (const part of folderPath.split("/")) {
      acc = acc ? acc + "/" + part : part;
      const next = (node.children || []).find(
        (c) => c.kind === "folder" && c.path === acc
      );
      if (!next) return false;
      node = next;
    }
    return true;
  }
  /* The node for a drill target (phase 97, phase 167-generalized):
   * the source node at the root, or the folder node found by walking
   * the HELD levels with the folder's cumulative source-relative path
   * (the builder's folder paths are source-relative, so `one/two`
   * resolves one level at a time; a truncated node simply has no
   * children to walk into). */
  function nodeForTarget(target) {
    const src = kbTree.sources.find((s) => s.name === target.source);
    if (!src) return null;
    if (target.folder === null || target.folder === "") return src;
    let node = src;
    let acc = "";
    for (const part of target.folder.split("/")) {
      acc = acc ? acc + "/" + part : part;
      const next = (node.children || []).find(
        (c) => c.kind === "folder" && c.path === acc
@@ -1696,11 +1769,27 @@ export async function mount(root) {
    return node;
  }
-  /* The KB-wide stat cards (phase 97): walk the WHOLE tree — document
+  function currentLevelNode() {
-   * count, chunks sum, max indexed_at — the values identical to the
+    return nodeForTarget(current);
-   * former flat /api/docs walk (the same documents, one level deeper).
+  }
-   * fmtDate reuse for the max. */
+
  /* The KB-wide stat cards. Phase 167 (task 04, D2/D4): the server's
   * `totals` (computed from the FULL catalogue rows beside any depth
   * of tree) are the source of truth — a depth-limited tree CANNOT be
   * summed client-side. The phase-97 recursive walk (document count,
   * chunks sum, max indexed_at — the values identical to the former
   * flat /api/docs walk) stays ONLY as the defensive fallback for a
   * tree served without `totals` (a pre-phase-167 process) — the
   * no-crash path. fmtDate reuse for the max (renderLevel). */
  function treeStats() {
    const totals = kbTree.totals;
    if (totals && typeof totals === "object") {
      return {
        docs: totals.documents,
        totalChunks: totals.chunks,
        last: totals.last_indexed || "",
      };
    }
    let docs = 0;
    let totalChunks = 0;
    let last = "";
@@ -1722,8 +1811,10 @@ export async function mount(root) {
  /* The breadcrumb (phase 97): hidden at the top level; when drilled
   * in, one link per ancestor — the top level (back to the sources
   * list), the source, then the folder chain — the LAST segment a
-   * span with aria-current="page". Client-side only: no fetch, no URL
+   * span with aria-current="page". No URL change (the navigation is a
-   * change (the navigation is a re-render of the fetched tree). */
+   * re-render); phase 167: an ancestor is already loaded (you cannot
   * drill past it without holding it), so a breadcrumb step costs no
   * fetch in practice — goTo stays the ONE drill code either way. */
  function renderCrumb() {
    if (!crumbEl) return;
    crumbEl.replaceChildren();
@@ -1766,7 +1857,7 @@ export async function mount(root) {
    a.textContent = label; // document-derived text — never innerHTML
    a.addEventListener("click", (e) => {
      e.preventDefault();
-      goTo(target);
+      goTo(target, a);
    });
    return a;
  }
@@ -1779,9 +1870,9 @@ export async function mount(root) {
    return span;
  }
-  /* The drill navigation (phase 99, task 02, D2 — client-side, no
+  /* The drill navigation (phase 99, task 02, D2 — the drill state IS
-   * fetch; the drill state IS the history state). `applyTarget(target,
+   * the history state; since phase 167 a user drill may FIRST fetch
-   * push)`: when `push` is true the target DIFFERS from the current
+   * the level — see enterFolder). `applyTarget(target, push)`: when `push` is true the target DIFFERS from the current
   * level (a re-click of the current row/segment pushes NO duplicate
   * entry), it FIRST records a STATE-ONLY entry — history.pushState(
   * { view: "rag", kb: target }, "") — the second arg "" + no third:
@@ -1795,14 +1886,166 @@ export async function mount(root) {
      history.pushState({ view: "rag", kb: target }, "");
    }
    current = { source: target.source, folder: target.folder };
    navSeq++; // phase 167: any navigation invalidates an in-flight drill
    renderLevel();
  }
  /* Every user-initiated drill (source row, folder link, breadcrumb
   * segment) — push a history entry, then drill (D2, phase 99 task
-   * 02). */
+   * 02). Since phase 167 (task 04, A3) the drill may FIRST fetch the
-  function goTo(target) {
+   * level (enterFolder); the push lands when the level is in hand, so
-    applyTarget(target, true);
+   * a failed drill records nothing. `link` is the clicked link — it
   * carries the in-flight aria-busy/aria-disabled state. */
  function goTo(target, link) {
    enterFolder(target, link, true);
  }
  /* Phase 167 (task 04, D4/A3) — the lazy drill. The depth-1 tree
   * NAMES deeper folders (truncated: true, children not built); the
   * FIRST open of one fetches its level through the node route, merges
   * the children into the node, and caches the level — a re-open is
   * free (zero network). A level the client already holds (or a node
   * the payload fully built) drills with NO fetch, exactly as before:
   * applyTarget + render, nothing else. One drill at a time —
   * drillPending is the §7.4 double-click guard, and every settle
   * path (success, failure, 404, superseded) releases it. */
  function needsLevelFetch(node, source) {
    return (
      !!node &&
      node.truncated === true &&
      !levelCache.has(source + "/" + node.path)
    );
  }
  async function enterFolder(target, link, push) {
    if (drillPending) return; // one drill at a time (§7.4 double-click guard)
    const node = nodeForTarget(target);
    if (needsLevelFetch(node, target.source)) {
      drillPending = { source: target.source, folder: target.folder };
      const loaded = await fetchLevel(target, node, link);
      if (!loaded) return; // failed / vanished / superseded — handled inside
    }
    applyTarget(target, push);
  }
  /* ONE node fetch + its full §7.4 lifecycle. The fetch URL is
   * /api/docs/tree/node?source=…&path=… (path source-relative; the
   * route is documented in app/api/docs.py — one catalogue query per
   * drill, no N+1). The race convention EXTENDS from loadTree: the
   * response may only paint while NO newer load (loadSeq, phase 79)
   * and NO navigation (navSeq) have happened — a stale response never
   * touches the DOM. Success: Object.assign merges the node (children,
   * counts, dates, summary, pending flags — the wire shape the full
   * tree produces) and levelCache marks the level; only a 200 caches.
   * A 404 IS the vanished-folder signal — honest reset to the top
   * level, nothing cached. Any other failure: the alert + Retry row,
   * the current level staying rendered. */
  async function fetchLevel(target, targetNode, link) {
    const my = loadSeq;
    const nav = navSeq;
    if (link) {
      link.setAttribute("aria-busy", "true");
      link.setAttribute("aria-disabled", "true"); // a double click cannot refetch
    }
    showDrillLoading(target.folder.split("/").pop());
    let response = null;
    let vanished = false;
    try {
      response = await fetch("/api/docs/tree/node?source=" + encodeURIComponent(target.source) + "&path=" + encodeURIComponent(target.folder));
      if (response.status === 404) vanished = true;
    } catch {
      response = null; // network failure — the alert path below
    }
    // Every settle path re-enables the controls (§7.4) — the link is
    // only ours to clear while this drill is still THE pending one (a
    // newer load owns the flags the moment it starts).
    if (drillPending && drillPending.source === target.source && drillPending.folder === target.folder) {
      drillPending = null;
    }
    if (link) {
      link.removeAttribute("aria-busy");
      link.removeAttribute("aria-disabled");
    }
    if (my !== loadSeq || nav !== navSeq) return false; // superseded — never paints
    clearDrillRow();
    if (vanished) {
      // The folder vanished under us (pruned/unregistered since the
      // level was listed) — the phase-97 never-stale contract: reset
      // to the top level, no stale breadcrumb, nothing cached.
      current = { source: null, folder: null };
      renderLevel();
      return false;
    }
    if (response === null || !response.ok) {
      showDrillFailure(target, link); // the current level stays rendered
      return false;
    }
    const merged = await response.json();
    if (my !== loadSeq || nav !== navSeq) return false; // stale — never paints
    Object.assign(targetNode, merged);
    levelCache.set(target.source + "/" + target.folder, true);
    return true;
  }
  /* The ONE in-flight row of the folders panel — appended to the
   * level the user is LEAVING (renderLevel's tbody clear owns it: a
   * re-render always drops the row). Tokens-only .kb-level-loading
   * (styles.css); textContent only, the house rule. */
  function clearDrillRow() {
    if (drillRowEl) {
      drillRowEl.remove();
      drillRowEl = null;
    }
  }
  function showDrillLoading(name) {
    clearDrillRow();
    const row = document.createElement("tr");
    row.className = "kb-level-loading";
    const td = document.createElement("td");
    td.colSpan = 4;
    const note = document.createElement("span");
    note.setAttribute("role", "status");
    note.textContent = "Loading " + name + "…"; // never innerHTML
    td.appendChild(note);
    row.appendChild(td);
    drillRowEl = row;
    if (foldersWrap) foldersWrap.hidden = false; // the row needs its panel
    if (foldersTbody) foldersTbody.appendChild(row);
  }
  /* Phase 167: rehydrate the CURRENT level when the held tree only
   * names it — used by the ADOPTED navigations (popstate: the popped
   * entry can point at a level the client has not fetched since the
   * last load; the bor:view-refresh alignment is covered by loadTree's
   * own rehydration). A no-op for held levels and at the top. */
  function ensureCurrentLevel() {
    if (drillPending) return;
    if (needsLevelFetch(currentLevelNode(), current.source)) {
      enterFolder({ source: current.source, folder: current.folder }, null, false);
    }
  }
  function showDrillFailure(target, link) {
    clearDrillRow();
    const row = document.createElement("tr");
    row.className = "kb-level-loading";
    const td = document.createElement("td");
    td.colSpan = 4;
    const note = document.createElement("span");
    note.setAttribute("role", "alert");
    note.textContent =
      "Couldn't load " + target.folder.split("/").pop() + " — "; // never innerHTML
    const retry = document.createElement("button");
    retry.type = "button"; // a REAL button (the phase-49 .retry-btn pill)
    retry.className = "retry-btn";
    retry.textContent = "Retry";
    retry.addEventListener("click", () => enterFolder(target, link, true));
    td.append(note, retry);
    row.appendChild(td);
    drillRowEl = row;
    if (foldersWrap) foldersWrap.hidden = false;
    if (foldersTbody) foldersTbody.appendChild(row);
  }
  /* ONE table for every level (phase 97): at the TOP level the rows
@@ -1818,7 +2061,7 @@ export async function mount(root) {
    link.textContent = s.name; // document-derived text — never innerHTML
    link.addEventListener("click", (e) => {
      e.preventDefault();
-      goTo({ source: s.name, folder: "" });
+      goTo({ source: s.name, folder: "" }, link);
    });
    nameTd.appendChild(link);
    tr.appendChild(nameTd);
@@ -1853,7 +2096,7 @@ export async function mount(root) {
    link.textContent = f.path.split("/").pop(); // never innerHTML
    link.addEventListener("click", (e) => {
      e.preventDefault();
-      goTo({ source: current.source, folder: f.path });
+      goTo({ source: current.source, folder: f.path }, link);
    });
    nameTd.appendChild(link);
    tr.appendChild(nameTd);
@@ -2209,6 +2452,11 @@ export async function mount(root) {
      const kb = event.state && event.state.kb;
      if (kb) applyTarget(kb, false);
      else applyTarget({ source: null, folder: null }, false);
      // Phase 167 (task 04): a forward/back can adopt an entry whose
      // level the client has not fetched since — rehydrate it through
      // the node route (a no-op for held levels; a truly vanished one
      // takes the 404-reset / alert path inside).
      ensureCurrentLevel();
    });
    loadTree();
  }
@@ -2997,6 +2997,14 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
  background: var(--brand-soft);
  text-decoration: underline;
 }
 /* Phase 167 (task 04, D4): the lazy drill's ONE in-flight row — the
   role="status" loading note and the role="alert" failure note (+ the
   existing .retry-btn pill) shown in the folders panel of the level
   the user is LEAVING: the level never blanks and the breadcrumb never
   lies (PLAN §7.4). Tokens only — ink-soft on the wrap's --surface
   (the .kb-level p pair, 5.1:1 AA); no animation beyond the house
   idiom (the Retry button's :focus-visible is the global rule). */
 .kb-level-loading { color: var(--ink-soft); }
 /* Image-document rows (phase 122, task 04): the file table's Path cell
   carries a FIXED 48px thumbnail box before the path link (image rows
@@ -0,0 +1,378 @@
 """Audit the payload of every endpoint that could load a ton of data
 (phase 167, task 05, D5, LOCKED A5).
 TODO L5 is one sentence: "Fix that **any any other endpoints** that could
 load a ton of data." ``GET /api/docs/tree`` was the reported offender; this
 script is the measured answer for the rest of the API, against the app the
 operator actually runs:
    uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
    # (or against the deployed app: --base-url https://brain.reeseapps.com)
 It signs in through the real ``POST /api/login`` (the session cookie is the
 only credential — nothing is hardcoded: ``--password`` or the
 ``BOR_ADMIN_PASSWORD`` the app itself reads from ``.env``), walks the
 candidate read routes, and prints one row per endpoint with the IDENTITY and
 the GZIP transfer size, the ratio, the wall time and a verdict against the
 budget (default 256 KB — the phase's "look at it" bar; LOCKED A5: this is a
 report, never a gate, so the exit code is 0 whenever the measurement
 succeeded).
 **Credentials are password-only on purpose.** The single-admin session is
 ``POST /api/login {"password": …}`` (PLAN A10) — there is no username field
 to send, so there is no ``--username``; ``BOR_ADMIN_PASSWORD`` (or
 ``--password``) is the whole credential.
 Both variants are measured on every endpoint because that is the pair the
 byte-budget log line reports (``bytes=`` / ``enc=``): ``Accept-Encoding:
 identity`` gives the payload the route produced, ``Accept-Encoding: gzip``
 gives what the client puts on the wire (``response.num_bytes_downloaded``,
 the raw bytes — ``response.content`` is already decompressed by httpx). The
 routes that publish a validator (the three catalogue reads, phase 167 task
 03) get one extra conditional request each: the expected answer is
 ``304`` with an empty body, which is the row's verdict.
 Run it against the 1,000-document live-replica snapshot (the
 ``restore-test-db`` skill) so the numbers are the production order of
 magnitude and not the fixture KB's — the corpus the run measured is printed
 with the table for exactly that reason.
 Exit codes: **0** the table printed (an over-budget row is a FINDING, shown
 as ``LOOK``, not a failure), **2** a precondition failed — the app
 unreachable, the login rejected, or no credential available.
 """
 from __future__ import annotations
 import argparse
 import os
 import sys
 import time
 from dataclasses import dataclass
 from typing import Any
 import httpx
 from dotenv import load_dotenv
 load_dotenv()
 #: The phase's "look at it" bar (256 KB uncompressed — D5/A5). Rows over it
 #: are verdict ``LOOK`` and must be fixed in the phase or carry a named
 #: follow-up in ``AUDIT.md``; nothing here rejects a request.
 DEFAULT_BUDGET = 262_144
 #: Per-request wall-clock ceiling. A large catalogue read on the restored
 #: 1,000-doc replica is fast, but a cold first query can take seconds.
 DEFAULT_TIMEOUT = 60.0
@dataclass
 class Row:
    """One measured endpoint."""
    label: str
    path: str
    status: int = 0
    identity_bytes: int | None = None
    identity_ms: int | None = None
    gzip_bytes: int | None = None
    gzip_ms: int | None = None
    revalidated: bool | None = None
    note: str = ""
    @property
    def ratio(self) -> str:
        """identity / gzip — how much the compression layer bought (< 1 is
        a win; ``—`` when the route did not compress)."""
        if not self.gzip_bytes or not self.identity_bytes:
            return "—"
        return f"{self.gzip_bytes / self.identity_bytes:.3f}"
    def verdict(self, budget: int) -> str:
        if self.status == 0:
            # Nothing was measured — the corpus has no such row (an empty KB,
            # no saved chats). Honest gap, not a pass and not a finding.
            return "n/a"
        if self.status == 304:
            return "304"
        if self.status != 200:
            return f"HTTP {self.status}"
        if self.revalidated:
            return "304"
        if (self.identity_bytes or 0) > budget:
            return f"LOOK (> {budget})"
        return "ok"
 def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(
        prog="python -m scripts.api_payload_audit",
        description=(
            "Measure the response size (identity AND gzip) of every "
            "candidate large-payload endpoint, against a byte budget."
        ),
    )
    p.add_argument(
        "--base-url",
        default="http://localhost:8000",
        metavar="URL",
        help="app to measure (default: http://localhost:8000)",
    )
    p.add_argument(
        "--password",
        default=None,
        metavar="PASSWORD",
        help=(
            "admin password; falls back to $BOR_ADMIN_PASSWORD (the app's "
            "own variable). The single-admin login has no username (A10)."
        ),
    )
    p.add_argument(
        "--budget",
        type=int,
        default=DEFAULT_BUDGET,
        metavar="BYTES",
        help=f"the LOOK threshold in bytes (default: {DEFAULT_BUDGET})",
    )
    p.add_argument(
        "--timeout",
        type=float,
        default=DEFAULT_TIMEOUT,
        metavar="SECONDS",
        help=f"per-request timeout (default: {DEFAULT_TIMEOUT:g})",
    )
    return p
 def _fail(message: str) -> int:
    print(f"api_payload_audit: {message}", file=sys.stderr)
    return 2
 def _get(client: httpx.Client, path: str, accept: str) -> tuple[int, int, int, int]:
    """GET ``path`` with an explicit ``Accept-Encoding`` →
    ``(status, body bytes, wire bytes, ms)``.
    ``body`` is the decoded payload (what the route produced); ``wire`` is
    the raw transfer, which is what the browser pays. They differ exactly
    when the response came back gzipped.
    """
    started = time.perf_counter()
    response = client.get(path, headers={"Accept-Encoding": accept})
    ms = int(round((time.perf_counter() - started) * 1000))
    return response.status_code, len(response.content), response.num_bytes_downloaded, ms
 def measure(
    client: httpx.Client, label: str, path: str, *, conditional: bool = False
 ) -> Row:
    """One endpoint, both encodings, plus the conditional re-request on the
    routes that publish a validator (``conditional=True``)."""
    row = Row(label=label, path=path)
    row.status, row.identity_bytes, _, row.identity_ms = _get(client, path, "identity")
    gzip_status, _, wire, ms = _get(client, path, "gzip")
    row.gzip_bytes, row.gzip_ms = wire, ms
    if gzip_status != row.status:  # pragma: no cover - defensive
        row.status = gzip_status
    if conditional:
        # The ETag from a fresh unconditional GET, replayed (fetch does the
        # same thing on the second page view).
        etag = client.get(path, headers={"Accept-Encoding": "identity"}).headers.get("etag")
        if etag is None:
            row.revalidated = False
        else:
            again = client.get(
                path, headers={"Accept-Encoding": "identity", "If-None-Match": etag}
            )
            row.revalidated = again.status_code == 304 and len(again.content) == 0
    return row
 def _first_folder(tree: dict[str, Any]) -> tuple[str, str] | None:
    """``(source, folder path)`` of the first folder node the depth-1 tree
    reports — a representative drill target for the node route."""
    for source in tree.get("sources", []):
        for child in source.get("children", []):
            if child.get("kind") == "folder":
                return str(source.get("name", "")), str(child.get("path", ""))
    return None
 def run(args: argparse.Namespace) -> tuple[int, list[Row], list[str]]:
    """Do the whole audit; returns ``(exit code, rows, corpus lines)``."""
    password = args.password or os.environ.get("BOR_ADMIN_PASSWORD", "")
    if not password:
        return (
            _fail(
                "no admin password — pass --password or set BOR_ADMIN_PASSWORD "
                "(the app's own variable; it is never hardcoded here)"
            ),
            [],
            [],
        )
    client = httpx.Client(
        base_url=args.base_url, timeout=args.timeout, follow_redirects=False
    )
    try:
        login = client.post("/api/login", json={"password": password})
    except httpx.HTTPError as exc:
        return (
            _fail(
                f"cannot reach the app at {args.base_url} ({exc.__class__.__name__}: {exc}) "
                "— start it with `uv run uvicorn app.main:app`"
            ),
            [],
            [],
        )
    if login.status_code != 204:
        detail = ""
        try:
            detail = str(login.json().get("detail", ""))
        except Exception:  # noqa: BLE001 - a non-JSON error body is still an error
            detail = login.text[:120]
        hint = {
            401: "wrong password (BOR_ADMIN_PASSWORD of the app under test)",
            429: "rate limited (10 failed attempts / 15 min) — wait for the window",
        }.get(login.status_code, "unexpected answer from the real login route")
        return (
            _fail(
                f"login to {args.base_url} failed: HTTP {login.status_code} — "
                f"{hint} {detail}"
            ),
            [],
            [],
        )
    rows: list[Row] = []
    corpus: list[str] = []
    # --- the corpus the numbers describe (printed with the table) ---------
    docs_response = client.get("/api/docs", headers={"Accept-Encoding": "identity"})
    documents: list[dict[str, Any]] = list(docs_response.json().get("documents", []))
    by_chunks = sorted(documents, key=lambda d: int(d.get("chunks", 0)), reverse=True)
    biggest_doc = by_chunks[0] if by_chunks else None
    chunks_total = sum(int(d.get("chunks", 0)) for d in documents)
    corpus.append(f"documents={len(documents)} chunks_total={chunks_total}")
    chats: list[dict[str, Any]] = list(
        client.get("/api/chats", headers={"Accept-Encoding": "identity"}).json().get("chats", [])
    )
    biggest_chat = (
        sorted(chats, key=lambda c: int(c.get("message_count", 0)), reverse=True)[0]
        if chats
        else None
    )
    # --- the candidates ---------------------------------------------------
    rows.append(
        measure(client, "tree depth=1 (default)", "/api/docs/tree?depth=1", conditional=True)
    )
    rows.append(
        measure(client, "tree depth=8 (full)", "/api/docs/tree?depth=8", conditional=True)
    )
    depth1 = client.get(
        "/api/docs/tree?depth=1", headers={"Accept-Encoding": "identity"}
    ).json()
    folder = _first_folder(depth1)
    if folder is not None and folder[0]:
        node_path = f"/api/docs/tree/node?source={folder[0]}&path={folder[1]}&depth=1"
        rows.append(
            measure(client, f"tree node {folder[0]}/{folder[1]}", node_path, conditional=True)
        )
    else:  # the honest gap: an empty KB has no drill to measure
        rows.append(
            Row(label="tree node (drill)", path="/api/docs/tree/node", note="no folder in corpus")
        )
    rows.append(measure(client, "docs catalogue", "/api/docs", conditional=True))
    if biggest_doc is not None:
        content_path = (
            f"/api/documents/content?source={biggest_doc['source']}&path={biggest_doc['path']}"
        )
        row = measure(client, f"document content (biggest: {biggest_doc['path']})", content_path)
        body = client.get(content_path, headers={"Accept-Encoding": "identity"}).json()
        row.note = f"content field {len(str(body.get('content', '')))} chars"
        rows.append(row)
    else:
        rows.append(Row(label="document content", path="/api/documents/content", note="empty KB"))
    rows.append(measure(client, "saved chats list", "/api/chats"))
    if biggest_chat is not None:
        rows.append(
            measure(
                client,
                f"saved chat (biggest: {int(biggest_chat.get('message_count', 0))} msgs)",
                f"/api/chats/{biggest_chat['id']}",
            )
        )
    else:
        rows.append(Row(label="saved chat (one)", path="/api/chats/{id}", note="no saved chats"))
    rows.append(measure(client, "git sources", "/api/git-sources"))
    rows.append(measure(client, "suggestions", "/api/suggestions"))
    rows.append(measure(client, "sync status (polled)", "/api/sync/status"))
    rows.append(measure(client, "tokens", "/api/tokens"))
    rows.append(measure(client, "config", "/api/config"))
    rows.append(measure(client, "whoami", "/api/whoami"))
    corpus.append(f"sources={len(depth1.get('sources', []))}")
    totals = depth1.get("totals") or {}
    if totals:
        corpus.append(
            f"tree totals documents={totals.get('documents')} chunks={totals.get('chunks')}"
        )
    corpus.append(f"saved_chats={len(chats)}")
    corpus.append(f"biggest_doc={biggest_doc['path'] if biggest_doc else '—'}")
    return 0, rows, corpus
 def render(rows: list[Row], *, base_url: str, budget: int, corpus: list[str]) -> None:
    """The table: endpoint, identity bytes, gzip bytes, ratio, ms, verdict."""
    header: tuple[str, ...] = ("endpoint", "identity", "gzip", "ratio", "ms", "verdict")
    table: list[tuple[str, ...]] = [header]
    for row in rows:
        table.append(
            (
                row.label,
                f"{row.identity_bytes:,}" if row.identity_bytes is not None else "—",
                f"{row.gzip_bytes:,}" if row.gzip_bytes is not None else "—",
                row.ratio,
                f"{row.gzip_ms}" if row.gzip_ms is not None else "—",
                row.verdict(budget),
            )
        )
    widths = [max(len(str(r[i])) for r in table) for i in range(len(header))]
    def _line(values: tuple[Any, ...]) -> str:
        return (
            f"  {str(values[0]):<{widths[0]}}  {str(values[1]):>10}  "
            f"{str(values[2]):>10}  {str(values[3]):>{widths[3]}}  "
            f"{str(values[4]):>6}  {str(values[5])}"
        )
    print(f"api_payload_audit: base={base_url} budget={budget:,} bytes (LOOK threshold)")
    print(f"api_payload_audit: corpus {' '.join(corpus)}")
    print(_line(header))
    print(f"  {'-' * (widths[0] + 44)}")
    for row, source in zip(rows, table[1:], strict=True):
        print(_line(source))
        if row.note:
            print(f"      note: {row.note}")
    print(
        "api_payload_audit: ms is the gzip (real-client) variant; identity is the "
        "route's own bytes — the pair the ' :: api ' log line reports."
    )
 def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)
    code, rows, corpus = run(args)
    if code != 0 or not rows:
        return code
    render(rows, base_url=args.base_url, budget=args.budget, corpus=corpus)
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -98,8 +98,9 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
 def _drill(page: Page, *names: str) -> None:
    """Drill one level at a time: each name is the EXACT text of the
-    source/folder link at the current level (client-side — no fetch,
+    source/folder link at the current level (no URL change; since phase
-    no URL change)."""
+    167 the first open of a level the depth-1 payload only NAMES costs
    one /api/docs/tree/node fetch, a level already held costs none)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -301,7 +301,11 @@ def _docs_by_path(app_url: str, cookies: dict[str, str]) -> dict[str, dict[str,
 def _tree(app_url: str, cookies: dict[str, str]) -> dict[str, Any]:
-    r = httpx.get(f"{app_url}/api/docs/tree", timeout=10, cookies=cookies)
+    # Phase 167 (task 02): the callers below use ``_find_file``, which
    # recurses (every fixture file sits under backups/, legacy/ or
    # future/), so the WHOLE tree is requested — the endpoint's default
    # (depth=1) is the RAG view's first paint only.
    r = httpx.get(f"{app_url}/api/docs/tree?depth=8", timeout=10, cookies=cookies)
    assert r.status_code == 200, r.text
    return r.json()
@@ -122,9 +122,10 @@ def _delete_source_rows() -> None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
-    no URL change): each name is the EXACT text of the source/folder
+    costs exactly one /api/docs/tree/node fetch, a held level costs
-    link at the current level."""
+    nothing, and no click ever changes the URL): each name is the EXACT
    text of the source/folder link at the current level."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -502,9 +502,11 @@ def _assert_change_terminal(body: dict[str, Any]) -> None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
+    """Drill one level at a time (no URL change; phase 167: the first
-    change): each name is the EXACT text of the source/folder link at
+    open of a level the depth-1 payload only NAMES costs one
-    the current level (the row builders' link text: the source name,
+    /api/docs/tree/node fetch, a level already held costs nothing):
    each name is the EXACT text of the source/folder link at the
    current level (the row builders' link text: the source name,
    or the folder's last path segment)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -609,7 +611,7 @@ def test_rag_table_renders_the_stored_voice(
    login(page, app_url)  # → /sources.html (the RAG view)
    # The top level: the one source row (the ls() equivalence — the
-    # tree's single fetch settled) and its stored source-root voice in
+    # tree's ONE depth-1 fetch settled) and its stored source-root voice in
    # the Description cell (the exact canned line, text + title).
    page.wait_for_selector("#folders-tbody tr", timeout=30_000)
    expect(page.locator("#folders-tbody tr")).to_have_count(1)
@@ -416,7 +416,10 @@ def test_api_created_dates_are_true_per_file(
    # (old folder = the 2020 date alone, recent folder = the 2024
    # date alone, source = the whole subtree's 2024 max — D9, now over
    # TRUE dates).
-    t = page.request.get(f"{app_url}/api/docs/tree")
+    # Phase 167 (task 02): ``_find_file`` recurses by design (the files
    # sit under ``old/`` and ``recent/``), so ask for the WHOLE tree —
    # the endpoint's default (depth=1) is the RAG view's first paint.
    t = page.request.get(f"{app_url}/api/docs/tree?depth=8")
    assert t.status == 200, t.text
    source = _source_node(t.json())
    assert source["documents"] == 2
@@ -204,9 +204,11 @@ SOURCE_NAME = "docs"  # the fixtures dir's basename (the indexed-only source)
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
+    """Drill one level at a time (no URL change; phase 167: the first
-    change): each name is the EXACT text of the source/folder link at
+    open of a level the depth-1 payload only NAMES costs one
-    the current level (the test_kb_tree house pattern)."""
+    /api/docs/tree/node fetch, a level already held costs nothing):
    each name is the EXACT text of the source/folder link at the
    current level (the test_kb_tree house pattern)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -218,7 +220,7 @@ def _go_top(page: Page) -> None:
 def _wait_top_level(page: Page) -> None:
-    """The tree's single fetch settled: the source row is rendered
+    """The ONE ``?depth=1`` catalog fetch has settled: the source row is rendered
    (the catalog-rendered signal — the top-level file table is always
    hidden, so it is no longer a usable one)."""
    page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
@@ -92,9 +92,10 @@ def _delete_source_rows() -> None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
-    no URL change): each name is the EXACT text of the source/folder
+    costs exactly one /api/docs/tree/node fetch, a held level costs
-    link at the current level."""
+    nothing, and no click ever changes the URL): each name is the EXACT
    text of the source/folder link at the current level."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -9,6 +9,24 @@ the level's files — and the owner edits (or clears) any directory's
 description with the phase-57 inline affordance. Everything is pinned
 against the real app + the deterministic mock.
 PHASE 167 (task 06, LOCKED A3) — THE CATALOG LOAD, MIGRATED NOT DELETED:
 the view no longer downloads the whole tree. The boot is ONE
 ``GET /api/docs/tree?depth=1`` (the sources plus one level plus the
 server's ``totals``), and the FIRST time the owner opens a folder the
 payload only NAMES, the client fetches that one level through
 ``GET /api/docs/tree/node?source=…&path=…`` (the phase-97 "ONE fetch of
 /api/docs/tree, the drill is client-side" contract, superseded by the
 TODO L5 2 MB fix — see
 ``tests/e2e/test_kb_tree_lazy_load.py`` for the new load-shape suite).
 EVERY claim this suite owns stays and is re-pointed at the level the
 owner is ON: the breadcrumb text, the back navigation, the table
 columns, the dates, the summary-editor round trip, the pending markers,
 the empty state and the anonymous gate. A level the client already
 holds (the top level, a source root, a folder drilled into once — and
 the breadcrumb always walks back through a level it drilled) still
 renders with ZERO requests, so the pre-phase navigation claims below
 remain literally true at those levels.
 Run in isolation (DB must be up: ``podman compose up -d db``):
    uv run pytest tests/e2e/test_kb_tree.py -v --no-cov
@@ -498,18 +516,21 @@ def _run_sync(app_url: str) -> dict[str, Any]:
 def _wait_top_level(page: Page) -> None:
    """The admin boot has rendered the top level: the two source rows
-    are in the folders table (the tree's single fetch settled) and the
+    are in the folders table (the ONE ``?depth=1`` catalog fetch has
-    top-level file table is empty (files are per-source — always hidden
+    settled — phase 167) and the top-level file table is empty (files
-    at the top)."""
+    are per-source — always hidden at the top)."""
    expect(page.locator("#folders-tbody tr")).to_have_count(2, timeout=30_000)
    expect(page.locator("#docs-tbody tr")).to_have_count(0)
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
+    """Drill one level at a time (no URL change — the navigation is a
-    change): each name is the EXACT text of the source/folder link at
+    re-render; phase 167: a level the depth-1 payload already holds
-    the current level (the row builders' link text: the source name,
+    costs ZERO requests, the first open of one it only NAMES costs
-    or the folder's last path segment)."""
+    exactly one ``/api/docs/tree/node`` fetch): each name is the EXACT
    text of the source/folder link at the current level (the row
    builders' link text: the source name, or the folder's last path
    segment)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -668,7 +689,10 @@ def test_drill_into_folder(
    expect(page.locator("#folders-wrap")).to_be_hidden()
    # The breadcrumb link on the source goes back UP to the source
-    # level (client-side — no fetch, no URL change).
+    # level — a re-render with no fetch and no URL change, because
    # reaching ``two`` means the client holds ``alpha`` (phase 167:
    # the breadcrumb only ever walks back through a level it drilled,
    # so this stays the zero-request path it was pre-phase).
    page.click(f'#kb-crumb a.kb-crumb-link:text-is("{ALPHA}")')
    expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(ALPHA)
    expect(page.locator("#kb-crumb a.kb-crumb-link")).to_have_count(1)
@@ -446,12 +446,13 @@ def _wait_top_level(page: Page) -> None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, NO URL
+    """Drill one level at a time (no URL change; phase 167: an UNCACHED
-    change): each name is the EXACT text of the source/folder link at
+    level costs exactly one /api/docs/tree/node fetch, a level already
-    the current level (the row builders' link text: the source name, or
+    held costs nothing): each name is the EXACT text of the
-    the folder's last path segment). Phase 99 (D2): every drill click
+    source/folder link at the current level (the row builders' link
-    is a STATE-ONLY history entry (``goTo`` → ``applyTarget(target,
+    text: the source name, or the folder's last path segment). Phase 99
-    true)``)."""
+    (D2): every drill click is a STATE-ONLY history entry (``goTo`` →
    ``applyTarget(target, true)``)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -110,9 +110,11 @@ TITLE_E = "Phase77 popstate E"
 DATA_VIEWS: tuple[tuple[str, str, str], ...] = (
    ("#nav-history", "/history.html", "/api/chats"),
    # Phase 97: the RAG view's list endpoint is the drill-down tree
-    # (GET /api/docs/tree — the flat /api/docs is the unchanged API
+    # (the flat /api/docs is the unchanged API surface, no longer the
-    # surface, no longer the view's fetch).
+    # view's fetch). Phase 167: the load carries depth=1 (the payload
-    ("#nav-sources", "/sources.html", "/api/docs/tree"),
+    # diet) — the per-level drill fetches /api/docs/tree/node, which
    # these no-drill tests never trigger.
    ("#nav-sources", "/sources.html", "/api/docs/tree?depth=1"),
    ("#nav-git-sources", "/git-sources.html", "/api/git-sources"),
    ("#nav-tuning", "/tuning.html", "/api/steering"),
 )
@@ -107,9 +107,10 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
-    no URL change): each name is the EXACT text of the source/folder
+    costs exactly one /api/docs/tree/node fetch, a held level costs
-    link at the current level."""
+    nothing, and no click ever changes the URL): each name is the EXACT
    text of the source/folder link at the current level."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -119,9 +119,28 @@ def _run_in_thread(coro: Any) -> Any:
 def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
-    """Truncate the KB (and query log), then optionally re-import fixtures."""
+    """Truncate the KB (and query log), then optionally re-import fixtures.
    The truncate covers EVERY table that shapes the prompt, not just the
    document rows: this suite's story asserts the EXACT long-answer text,
    and the mock echoes a stored KB overview into the answer verbatim
    (`` (kb: <first bullet>)``, phase 31) plus a stored steering note
    (phase 15). Those rows survive a documents-only truncate — another
    suite in the shared E2E Postgres ran a real sync and left its
    overview behind — so the answer would arrive with a suffix this
    suite never seeded and the byte-exact pin would fail on the OTHER
    suite's corpus (``tests/e2e/test_kb_tree.py`` is the usual holder).
    Same list as that suite's ``_truncate_all`` (the house isolation
    pattern); no assertion changes — the corpus under test is exactly
    the one this fixture seeds.
    """
    with SessionLocal() as db:
-        db.execute(text("TRUNCATE chunks, documents, query_log"))
+        db.execute(
            text(
                "TRUNCATE chunks, documents, query_log, steering_notes, "
                "kb_overview, git_sources, folder_summaries"
            )
        )
        db.commit()
    if not seed:
        return None
@@ -593,9 +593,11 @@ def _wait_top_level(page: Page) -> None:
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
+    """Drill one level at a time (no URL change; phase 167: the first
-    change): each name is the EXACT text of the source/folder link at
+    open of a level the depth-1 payload only NAMES costs one
-    the current level (the row builders' link text: the source name,
+    /api/docs/tree/node fetch, a level already held costs nothing):
    each name is the EXACT text of the source/folder link at the
    current level (the row builders' link text: the source name,
    or the folder's last path segment — the phase-97 pattern)."""
    for name in names:
        page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -335,6 +335,39 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR
    assert "HONESTY GATE" in system["content"]
 def test_chat_turn_logs_one_budget_line_with_the_stream_flag(
    client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
 ) -> None:
    """Phase 167 (task 05, D5): the SSE turn is measured like every other
    ``/api/*`` response but with `` stream=1`` — ONE line, written when the
    stream ends, ``bytes=`` the whole streamed total (the frames, not one
    frame) and ``duration_ms`` the stream's life. The stream itself stays
    uncompressed (phase 167 A1, pinned above), so ``enc=identity`` here
    documents the wire, and the chat turn never bloats the log with one
    line per frame."""
    fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
    try:
        with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
            _, _, frames = _stream_chat(client, QUESTION)
    finally:
        fastapi_app.dependency_overrides.clear()
    assert frames[-1]["type"] == "done"  # the stream ran to completion
    lines = [
        record.getMessage()
        for record in caplog.records
        if record.name == "app.response_budget" and " path=/api/chat " in record.getMessage()
    ]
    assert len(lines) == 1, lines
    line = lines[0]
    assert line.endswith(" stream=1")
    match = re.search(r"status=(\d+) bytes=(\d+) enc=(\S+)", line)
    assert match is not None
    assert match.group(1) == "200"
    assert int(match.group(2)) > len(b"data: ")  # a real total, not a stub
    assert match.group(3) == "identity"
 def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM) -> None:
    """Phase 17: ``thinking`` frames precede every ``delta`` frame and
    reassemble to the model's reasoning; the ``done`` contract is
@@ -15,12 +15,20 @@ qualifying row, ABSENT otherwise — never ``null``), plus the phase-163
 for a stamped strip, ABSENT for a pre-phase row and for every
 non-qualifying document).
 Also the phase-167 (task 03) per-level drill route
 ``GET /api/docs/tree/node`` (the node shape, the 403/404 matrix, the
 pin that its children equal the same level of the unlimited
 ``/api/docs/tree?depth=8`` response) and the ETag revalidation of the
 three catalogue reads (200 + validator, conditional 304 with no body,
 the admin-PATCH bust — the LOCKED A4 proof — coexisting with gzip).
 Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
 """
 from __future__ import annotations
 import asyncio
 import base64
 import gzip
 import hashlib
 import inspect
 import itertools
@@ -38,8 +46,9 @@ import app.api.docs as docs_api
 import app.rag.importer as rag_importer
 from app.config import Settings
 from app.core import tokens as token_service
 from app.core.security_headers import CSP
 from app.main import app as fastapi_app
-from app.models import Chunk, Document, FolderSummary, GitSource
+from app.models import Chunk, Document, FolderSummary, GitSource, SourcesMeta
 from app.rag import git_sources as rag_git_sources
 from app.rag.folder_summaries import missing_folder_summaries
 from app.rag.importer import import_sources
@@ -218,10 +227,19 @@ def test_docs_tree_populated_shape_order_counts_summaries(admin_client, db) -> N
    )
    db.commit()
-    r = admin_client.get("/api/docs/tree")
+    # Phase 167 (task 02): this shape pin walks the FULL tree, so it
    # asks for the whole build explicitly — the endpoint's default is
    # the view's first paint (depth=1), depth=8 is the entire tree.
    r = admin_client.get("/api/docs/tree?depth=8")
    assert r.status_code == 200
    body = r.json()
-    assert set(body) == {"sources"}
+    # ``totals`` joins the top-level keys (phase 167, additive).
    assert set(body) == {"sources", "totals"}
    assert body["totals"] == {
        "documents": 6,
        "chunks": 11,
        "last_indexed": base.isoformat(),
    }
    sources = body["sources"]
    assert [s["name"] for s in sources] == ["Homelab", "Deployments"]
@@ -310,7 +328,12 @@ def test_docs_tree_empty_registry_and_catalog(
    _truncate_tree_tables(db)
    r = admin_client.get("/api/docs/tree")
    assert r.status_code == 200
-    assert r.json() == {"sources": []}
+    # Phase 167 (task 02): ``totals`` rides even the empty tree —
    # 0 documents, 0 chunks, no last-indexed stamp.
    assert r.json() == {
        "sources": [],
        "totals": {"documents": 0, "chunks": 0, "last_indexed": None},
    }
 def test_docs_tree_indexed_only_source_after_registered(admin_client, db) -> None:
@@ -367,7 +390,10 @@ def test_docs_tree_summary_pending_on_source_and_folder_nodes(admin_client, db)
    )
    db.commit()
-    r = admin_client.get("/api/docs/tree")
+    # Phase 167 (task 02): the nested ``k8s/helm`` node needs the full
    # build — depth=8 keeps this pin walking every level (the default
    # is the view's first paint only).
    r = admin_client.get("/api/docs/tree?depth=8")
    assert r.status_code == 200
    (homelab,) = r.json()["sources"]
    # Phase 106 (task 05): ``updated_at`` joins the source node keys.
@@ -458,7 +484,10 @@ def test_docs_tree_pending_set_equals_missing_folder_summaries(admin_client, db)
    )
    db.commit()
-    r = admin_client.get("/api/docs/tree")
+    # Phase 167 (task 02): the cross-check walks EVERY node, so it
    # requests the full tree (the default depth ships only the view's
    # first level — truncated nodes still carry their own markers).
    r = admin_client.get("/api/docs/tree?depth=8")
    assert r.status_code == 200
    pending = _tree_pending_keys(r.json()["sources"])
    # THE cross-check: the marker set IS the gap-fill's candidate set.
@@ -828,8 +857,14 @@ def test_docs_tree_stat_walk_equivalence_with_flat_list(admin_client, db) -> Non
    _seed_doc(db, "Homelab", "solo.md", "Solo", 0, base)
    db.commit()
-    tree = admin_client.get("/api/docs/tree").json()
+    # Phase 167 (task 02): the walk needs EVERY file node, so it asks
    # for the full tree; the server ``totals`` must equal the same walk
    # over the flat list (the stat cards cannot drift either way).
    tree = admin_client.get("/api/docs/tree?depth=8").json()
    flat = admin_client.get("/api/docs").json()
    assert tree["totals"]["documents"] == len(flat["documents"])
    assert tree["totals"]["chunks"] == sum(d["chunks"] for d in flat["documents"])
    assert tree["totals"]["last_indexed"] == max(d["indexed_at"] for d in flat["documents"])
    files = _tree_file_nodes(tree["sources"])
    assert len(files) == len(flat["documents"])  # document count
@@ -1768,3 +1803,636 @@ def test_tree_with_no_image_rows_is_byte_identical(admin_client: TestClient, db)
        }
    finally:
        _truncate_tree_tables(db)
 # ---------------------------------------------------------------------------
 # Phase 167 (task 01): transport gzip on the real app — the /api/docs/tree
 # payload (the TODO L5 2 MB defect) compresses, small responses stay
 # identity, and the A20 security-header + A19 page contracts survive the
 # new layer sitting between SecurityHeaders (outermost) and Caching.
 # ---------------------------------------------------------------------------
 def _seed_gzip_corpus(db, count: int = 50) -> None:
    """>= 50 indexed docs — the tree response lands well above the 860-byte
    floor (the production corpus that motivates the phase is ~1 k docs)."""
    _truncate_tree_tables(db)
    base = datetime.now(UTC)
    for i in range(count):
        _seed_doc(db, "gzip-corpus", f"docs/sub-{i % 5}/file-{i:02d}.md", f"Doc {i:02d}", 1, base)
    db.commit()  # the endpoint session reads committed data only
 def test_docs_tree_ships_gzip_and_decompresses_to_the_documented_json(
    admin_client: TestClient, db
 ) -> None:
    """Accept-Encoding: gzip → Content-Encoding: gzip + Vary, and the raw
    wire bytes decompress to EXACTLY the identity bytes (the documented
    tree JSON, 50 file nodes) — ~an order of magnitude smaller."""
    _seed_gzip_corpus(db)
    # Phase 167 (task 02): the 50-file corpus must ride the wire for
    # this byte pin, so the full tree is requested explicitly (the
    # endpoint default ships one level, which would fall under the
    # compression floor).
    try:
        identity = admin_client.get(
            "/api/docs/tree?depth=8", headers={"Accept-Encoding": "identity"}
        )
        assert identity.status_code == 200
        assert "content-encoding" not in identity.headers
        assert len(identity.content) >= 860  # the seed is above the floor
        r = admin_client.get(
            "/api/docs/tree?depth=8", headers={"Accept-Encoding": "gzip"}
        )
        assert r.status_code == 200
        assert r.headers["content-encoding"] == "gzip"
        assert "accept-encoding" in r.headers.get("vary", "").lower()
        # httpx decodes transparently: same documented JSON.
        sources = r.json()["sources"]
        assert len(_tree_file_nodes_all(sources)) == 50
        # The RAW stream is a real gzip of the identity body.
        with admin_client.stream(
            "GET", "/api/docs/tree?depth=8", headers={"Accept-Encoding": "gzip"}
        ) as streamed:
            raw = b"".join(streamed.iter_raw())
        assert gzip.decompress(raw) == identity.content
        assert len(raw) < len(identity.content) // 2  # repetitive JSON: >> 2x
    finally:
        _truncate_tree_tables(db)
 def test_gzipped_response_still_carries_the_a20_headers(
    admin_client: TestClient, db
 ) -> None:
    """The outer-layer invariant (A20): gzip sits INSIDE
    SecurityHeadersMiddleware, so a gzipped response still carries the
    CSP, nosniff and the no-framing headers."""
    _seed_gzip_corpus(db, 60)
    try:
        # depth=8 (phase 167, task 02): the whole tree stays above the
        # 860-byte floor — the point here is a GZIPPED response.
        r = admin_client.get("/api/docs/tree?depth=8", headers={"Accept-Encoding": "gzip"})
        assert r.headers["content-encoding"] == "gzip"
        assert r.headers["content-security-policy"] == CSP
        assert r.headers["x-content-type-options"] == "nosniff"
        assert r.headers["x-frame-options"] == "DENY"
    finally:
        _truncate_tree_tables(db)
 def test_small_api_response_stays_identity(client: TestClient) -> None:
    """A tiny /api/* response (under the 860-byte floor) is NOT compressed
    even when the client offers gzip — the minimum-size exemption."""
    r = client.get("/api/health", headers={"Accept-Encoding": "gzip"})
    assert r.status_code == 200
    assert "content-encoding" not in r.headers
    assert r.json()["status"] == "ok"
 def test_html_page_contract_survives_gzip(client: TestClient) -> None:
    """A19 through the new layer: the rewritten HTML page (>= the floor, so
    it compresses) STILL serves no-cache with the ?v= asset rewrite and no
    validators — and the contract holds AFTER decompression."""
    r = client.get("/", headers={"Accept-Encoding": "gzip"})
    assert r.status_code == 200
    assert r.headers["content-encoding"] == "gzip"  # 95 KB page, far above the floor
    assert r.headers["cache-control"] == "no-cache"
    assert "etag" not in r.headers
    assert "last-modified" not in r.headers
    assert r.text.count("?v=") >= 1  # the caching rewrite ran, gzip preserved it
 # ---------------------------------------------------------------------------
 # Phase 167 (task 02): the depth-limited tree + server totals (D2/A2) — the
 # default depth is the RAG view's first paint (sources + ONE level), an
 # explicit depth builds deeper, out-of-range values are rejected, and an
 # untruncated node keeps its pre-phase-167 wire shape.
 # ---------------------------------------------------------------------------
 def _seed_depth_corpus(db) -> datetime:
    """Four documents nested one level deeper each (root, ``a``, ``a/b``,
    ``a/b/c``) with DISTINCT ``indexed_at`` stamps, plus one stored
    summary on ``a`` — a fixture whose truncated nodes still carry a
    recursive count, and whose level rows carry a description and a
    pending marker."""
    _truncate_tree_tables(db)
    base = datetime.now(UTC)
    db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
    _seed_doc(db, "Homelab", "root.md", "Root", 1, base)
    _seed_doc(db, "Homelab", "a/top.md", "Top", 2, base + timedelta(hours=1))
    _seed_doc(db, "Homelab", "a/b/mid.md", "Mid", 3, base + timedelta(hours=2))
    _seed_doc(db, "Homelab", "a/b/c/deep.md", "Deep", 4, base + timedelta(hours=3))
    db.add(FolderSummary(source="Homelab", folder_path="a", summary="A desc."))
    db.commit()  # the app's endpoint session reads committed data only
    return base
 def test_docs_tree_default_depth_ships_one_level_with_totals(admin_client, db) -> None:
    """``GET /api/docs/tree`` (no param) is the view's first paint:
    sources + ONE level. The folder with unloaded children carries
    ``truncated: true`` + ``children: []`` AND its recursive count;
    ``totals`` matches ``GET /api/docs`` (documents, chunks, the newest
    ``indexed_at``) — the stat cards stay honest on a partial tree."""
    base = _seed_depth_corpus(db)
    try:
        r = admin_client.get("/api/docs/tree")
        assert r.status_code == 200
        body = r.json()
        assert set(body) == {"sources", "totals"}
        (homelab,) = body["sources"]
        assert homelab["documents"] == 4  # the whole recursive source count
        assert "truncated" not in homelab  # depth >= 1: never truncated
        folders = [c for c in homelab["children"] if c["kind"] == "folder"]
        files = [c for c in homelab["children"] if c["kind"] == "file"]
        assert [f["path"] for f in files] == ["root.md"]  # direct files ship
        (a,) = folders
        assert a["path"] == "a"
        assert a["truncated"] is True
        assert a["children"] == []
        assert a["documents"] == 3  # the recursive count, no children built
        assert a["summary"] == "A desc."  # the level row shows the description
        assert a["summary_pending"] is False  # the stored row covers it
        # Nothing deeper is on the wire (the 2 MB payload stays home).
        assert [f["path"] for f in _tree_file_nodes_all(body["sources"])] == ["root.md"]
        # Totals from the SAME catalogue rows (no extra query).
        assert body["totals"] == {
            "documents": 4,
            "chunks": 10,
            "last_indexed": (base + timedelta(hours=3)).isoformat(),
        }
        flat = admin_client.get("/api/docs").json()
        assert body["totals"]["documents"] == len(flat["documents"])
        assert body["totals"]["chunks"] == sum(d["chunks"] for d in flat["documents"])
        assert body["totals"]["last_indexed"] == max(
            d["indexed_at"] for d in flat["documents"]
        )
    finally:
        _truncate_tree_tables(db)
 def test_docs_tree_depth_three_builds_deeper_and_truncates_the_rest(
    admin_client, db
 ) -> None:
    """``?depth=3``: ``a`` and ``a/b`` are built (no ``truncated`` key),
    ``a/b``'s subfolder ``a/b/c`` is truncated — while ``a/b``'s OWN
    direct files still ship: the budget limits DEPTH, never the row set
    of the levels it does send. The phase-98 pending rule is unchanged
    on truncated nodes (``a/b/c``: 1 doc, no stored row → pending)."""
    _seed_depth_corpus(db)
    try:
        body = admin_client.get("/api/docs/tree?depth=3").json()
        (homelab,) = body["sources"]
        (a,) = [c for c in homelab["children"] if c["kind"] == "folder"]
        assert "truncated" not in a  # expanded at this depth
        a_children = {c["path"]: c for c in a["children"]}
        assert set(a_children) == {"a/b", "a/top.md"}
        a_b = a_children["a/b"]
        assert "truncated" not in a_b
        a_b_c, mid = a_b["children"]  # subfolder first, then the direct file
        assert (a_b_c["kind"], a_b_c["path"], a_b_c["truncated"]) == (
            "folder",
            "a/b/c",
            True,
        )
        assert (a_b_c["documents"], a_b_c["children"]) == (1, [])
        assert a_b_c["summary"] is None
        assert a_b_c["summary_pending"] is True  # 1 doc, no stored row
        assert (mid["kind"], mid["path"]) == ("file", "a/b/mid.md")
        assert {f["path"] for f in _tree_file_nodes_all(body["sources"])} == {
            "root.md",
            "a/top.md",
            "a/b/mid.md",
        }  # deep.md never reaches the wire at this depth
    finally:
        _truncate_tree_tables(db)
 def test_docs_tree_depth_validation_matrix(admin_client, db) -> None:
    """``depth`` is bounded (1..8) and typed: ``0`` and ``9`` are out of
    range and ``abc`` is not an int — all 422 before any build runs;
    the bounds themselves answer 200."""
    _seed_depth_corpus(db)
    try:
        for bad in ("0", "9", "abc"):
            r = admin_client.get(f"/api/docs/tree?depth={bad}")
            assert r.status_code == 422, bad
        assert admin_client.get("/api/docs/tree?depth=1").status_code == 200
        assert admin_client.get("/api/docs/tree?depth=8").status_code == 200
    finally:
        _truncate_tree_tables(db)
 def test_docs_tree_full_build_keeps_the_pre_phase_node_wire_shape(
    admin_client, db
 ) -> None:
    """The wire-shape pin: on a FULL (``depth=8``) build nothing is
    truncated, so NO node carries the ``truncated`` key — the source,
    folder and file key sets are exactly the pre-phase-167 ones (the
    omission serializer keeps every existing consumer byte-identical);
    the additive top-level ``totals`` is the only body change."""
    _seed_depth_corpus(db)
    try:
        body = admin_client.get("/api/docs/tree?depth=8").json()
        assert set(body) == {"sources", "totals"}
        (homelab,) = body["sources"]
        assert set(homelab) == {
            "name", "documents", "updated_at", "summary", "summary_pending", "children"
        }
        seen_folders = seen_files = 0
        stack = list(homelab["children"])
        while stack:
            node = stack.pop()
            if node["kind"] == "folder":
                seen_folders += 1
                assert set(node) == {
                    "kind", "path", "documents", "updated_at", "summary",
                    "summary_pending", "children",
                }
                stack.extend(node["children"])
            else:
                seen_files += 1
                assert set(node) == {
                    "kind", "path", "title", "chunks", "created_at", "indexed_at"
                }
        assert (seen_folders, seen_files) == (3, 4)
    finally:
        _truncate_tree_tables(db)
 # ---------------------------------------------------------------------------
 # Phase 167 (task 03): GET /api/docs/tree/node — the drill's per-level
 # fetch — and the ETag revalidation of the three catalogue reads
 # (D3, LOCKED A3/A4).
 # ---------------------------------------------------------------------------
 def _find_child(node: dict, path: str) -> dict:
    """The child of *node* whose ``path`` equals *path* (the level walk
    the drill does client-side, expressed once for the assertions)."""
    return next(child for child in node["children"] if child["path"] == path)
 def test_tree_node_source_root_is_the_kb_tree_source_shape(
    admin_client: TestClient, db
 ) -> None:
    """``?source=Homelab`` (``path`` omitted = ``""``; the explicit
    ``path=`` answers the identical node): the SOURCE shape — the full
    key set, never ``truncated``, the whole recursive source count —
    with the root level only: the direct file plus the subfolder
    flagged for the next drill."""
    _seed_depth_corpus(db)
    try:
        r = admin_client.get("/api/docs/tree/node", params={"source": "Homelab"})
        assert r.status_code == 200
        node = r.json()
        assert "truncated" not in node  # the node's OWN children are always built
        assert set(node) == {
            "name",
            "documents",
            "updated_at",
            "summary",
            "summary_pending",
            "children",
        }
        assert node["name"] == "Homelab"
        assert node["documents"] == 4  # the whole recursive source count
        assert node["summary_pending"] is True  # no stored (Homelab, "") row
        (a,) = [c for c in node["children"] if c["kind"] == "folder"]
        (root,) = [c for c in node["children"] if c["kind"] == "file"]
        assert (a["path"], a["truncated"], a["children"], a["documents"]) == (
            "a",
            True,
            [],
            3,
        )
        assert root["path"] == "root.md"
        # The explicit empty path is the SAME node as the omitted one.
        r2 = admin_client.get(
            "/api/docs/tree/node", params={"source": "Homelab", "path": ""}
        )
        assert r2.status_code == 200
        assert r2.json() == node
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_nested_folder_fields_and_depth_one_children(
    admin_client: TestClient, db
 ) -> None:
    """``?source=Homelab&path=a&depth=1``: the KbTreeFolder shape with
    every field the level row renders — recursive count, recursive
    ``updated_at``, ``summary``, ``summary_pending`` (phase 98 D3) — and
    its DIRECT children only: the deeper subfolder flagged
    ``truncated`` (the next drill's target), the direct file present."""
    _seed_depth_corpus(db)
    try:
        r = admin_client.get(
            "/api/docs/tree/node", params={"source": "Homelab", "path": "a"}
        )
        assert r.status_code == 200
        node = r.json()
        assert node["kind"] == "folder"
        assert node["path"] == "a"
        assert "truncated" not in node  # this node's children WERE built
        assert node["documents"] == 3  # recursive — the phase-94 count rule
        assert node["updated_at"] is not None  # recursive max (phase 106 D9)
        assert node["summary"] == "A desc."
        assert node["summary_pending"] is False  # the stored row covers it
        a_b, top = node["children"]  # subfolder first, then the direct file
        assert (a_b["kind"], a_b["path"], a_b["truncated"], a_b["children"]) == (
            "folder",
            "a/b",
            True,
            [],
        )
        assert a_b["documents"] == 2  # mid.md + deep.md, recursive
        assert a_b["summary_pending"] is True  # docs but no stored row
        assert (top["kind"], top["path"]) == ("file", "a/top.md")
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_depth_two_builds_one_level_deeper(
    admin_client: TestClient, db
 ) -> None:
    """``&depth=2`` spends one more level: ``a/b`` arrives BUILT (no
    ``truncated`` key, its direct file present) and only its own
    subfolder stays truncated — the same budget semantics the full tree
    applies, answered for one node."""
    _seed_depth_corpus(db)
    try:
        r = admin_client.get(
            "/api/docs/tree/node",
            params={"source": "Homelab", "path": "a", "depth": 2},
        )
        assert r.status_code == 200
        a_b = _find_child(r.json(), "a/b")
        assert "truncated" not in a_b  # expanded at this depth
        a_b_c, mid = a_b["children"]
        assert (a_b_c["path"], a_b_c["truncated"], a_b_c["documents"]) == (
            "a/b/c",
            True,
            1,
        )
        assert (mid["kind"], mid["path"]) == ("file", "a/b/mid.md")
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_404_unknown_source_folder_and_file_path(
    admin_client: TestClient, db
 ) -> None:
    """404 ``{"detail": "unknown node"}`` for a source that is neither
    registered nor indexed, a folder failing the phase-94 existence
    rule, a FILE's path (a document's OWN path is not a folder), and
    traversal strings — which have no filesystem surface to hit: the
    node is derived from catalogue ROWS alone (the row-only rule, the
    phase-10 traversal argument reused)."""
    _seed_depth_corpus(db)
    try:
        for params in (
            {"source": "nope"},  # neither registered nor indexed
            {"source": "Homelab", "path": "nope"},  # no indexed path starts with it
            {"source": "Homelab", "path": "a/top.md"},  # a FILE's own path
            {"source": "Homelab", "path": "../Homelab"},  # traversal is not a folder
            {"source": "..", "path": ""},  # nor is it a source
        ):
            r = admin_client.get("/api/docs/tree/node", params=params)
            assert r.status_code == 404, params
            assert r.json() == {"detail": "unknown node"}
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_403_anonymous(client: TestClient, db) -> None:
    """The same gate as ``/api/docs/tree`` — anonymous 403
    ``{"detail": "admin only"}`` (the RAG view never fetches for guests,
    the endpoint must not lean on that)."""
    _seed_depth_corpus(db)
    try:
        r = client.get("/api/docs/tree/node", params={"source": "Homelab"})
        assert r.status_code == 403
        assert r.json() == {"detail": "admin only"}
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_validation_matrix(admin_client: TestClient, db) -> None:
    """``source`` is required and ``depth`` keeps the tree route's
    bounds (``ge=1, le=8``): a missing source, ``depth=0``, ``depth=9``
    and ``depth=abc`` all 422 before anything is built."""
    _seed_depth_corpus(db)
    try:
        assert admin_client.get("/api/docs/tree/node").status_code == 422
        for bad in ("0", "9", "abc"):
            r = admin_client.get(
                "/api/docs/tree/node", params={"source": "Homelab", "depth": bad}
            )
            assert r.status_code == 422, bad
    finally:
        _truncate_tree_tables(db)
 def test_tree_node_children_equal_the_full_tree_level(
    admin_client: TestClient, db
 ) -> None:
    """ONE concept, two routes, pinned equal: at the same budget the
    node route answers EXACTLY what ``GET /api/docs/tree`` carries at
    the same level — the source root at the default budget AND the full
    one, and the nested ``a`` folder on the full build — the JSON
    compared whole, so the drill can never paint a node the tree would
    not have produced."""
    _seed_depth_corpus(db)
    try:
        for depth in (1, 8):
            tree = admin_client.get(f"/api/docs/tree?depth={depth}").json()
            (source_tree,) = tree["sources"]
            node_root = admin_client.get(
                "/api/docs/tree/node", params={"source": "Homelab", "depth": depth}
            )
            assert node_root.status_code == 200
            assert node_root.json() == source_tree
        full = admin_client.get("/api/docs/tree?depth=8").json()
        a_tree = _find_child(full["sources"][0], "a")
        node_a = admin_client.get(
            "/api/docs/tree/node",
            params={"source": "Homelab", "path": "a", "depth": 8},
        )
        assert node_a.status_code == 200
        assert node_a.json() == a_tree
    finally:
        _truncate_tree_tables(db)
 _CATALOGUE_READ_URLS = (
    "/api/docs",
    "/api/docs/tree",
    "/api/docs/tree/node?source=Homelab&path=a",
 )
 def test_catalogue_reads_carry_etag_and_answer_304(
    admin_client: TestClient, db
 ) -> None:
    """The ETag matrix on all three catalogue reads (A4): the first GET
    → 200 + weak ``ETag`` + ``Cache-Control: private, must-revalidate``;
    the re-send with ``If-None-Match`` → 304 with an EMPTY body and
    BOTH headers (RFC 9110 §8.8.3 — a 304 repeats the validator); the
    strong form of the weak tag, a comma-separated list and ``*`` all
    revalidate; a wrong tag is a plain 200 carrying the CURRENT tag."""
    _seed_depth_corpus(db)
    try:
        for url in _CATALOGUE_READ_URLS:
            first = admin_client.get(url)
            assert first.status_code == 200, url
            tag = first.headers["etag"]
            assert tag.startswith('W/"'), url
            assert first.headers["cache-control"] == "private, must-revalidate", url
            hit = admin_client.get(url, headers={"If-None-Match": tag})
            assert hit.status_code == 304, url
            assert hit.content == b"", url
            assert hit.headers["etag"] == tag, url
            assert hit.headers["cache-control"] == "private, must-revalidate", url
            strong = tag[2:]  # the same tag WITHOUT the weakness marker
            assert admin_client.get(
                url, headers={"If-None-Match": strong}
            ).status_code == 304, url
            assert admin_client.get(
                url, headers={"If-None-Match": f'W/"stale-tag", {strong}'}
            ).status_code == 304, url
            assert admin_client.get(url, headers={"If-None-Match": "*"}).status_code == (
                304
            ), url
            miss = admin_client.get(url, headers={"If-None-Match": 'W/"not-the-tag"'})
            assert miss.status_code == 200, url
            assert miss.headers["etag"] == tag, url
            assert miss.json() == first.json(), url
    finally:
        _truncate_tree_tables(db)
 def _seed_bust_corpus(db) -> datetime:
    """The depth corpus with DETERMINISTIC, distinct ``created_at``
    stamps (the default seeding's subtree max is insert-time
    microseconds): ``a/b/c/deep.md`` IS the max of ``a``, so editing its
    date provably MOVES every answer that carries ``a``'s
    ``updated_at`` — which is what the bust test below asserts."""
    _truncate_tree_tables(db)
    base = datetime.now(UTC)
    dates = {
        "root.md": base - timedelta(days=30),
        "a/top.md": base - timedelta(days=20),
        "a/b/mid.md": base - timedelta(days=10),
        "a/b/c/deep.md": base - timedelta(days=1),  # the subtree max of "a"
    }
    db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
    for i, (path, created) in enumerate(dates.items()):
        doc = Document(
            source="Homelab",
            path=path,
            full_path=f"/tmp/Homelab/{path}",
            title=path,
            content=f"# {path}\n\nBody.",
            content_hash=f"{next(_counter):064d}",  # unique per row
            indexed_at=base,
            created_at=created,
        )
        db.add(doc)
        db.flush()
        db.add(
            Chunk(document_id=doc.id, position=0, content=f"chunk {i}", embedding=[0.01] * 768)
        )
    db.add(FolderSummary(source="Homelab", folder_path="a", summary="A desc."))
    db.commit()  # the app's endpoint session reads committed data only
    return base
 def test_etag_busts_on_folder_summary_and_document_date(
    admin_client: TestClient, db
 ) -> None:
    """The LOCKED A4 proof: the admin PATCHes mutate the tree WITHOUT
    bumping ``sources_meta.version`` (asserted around the edits — a
    version-derived tag would keep answering 304 here). After
    ``PATCH /api/folders/summary`` AND after ``PATCH
    /api/documents/date`` the tag CHANGES and the conditional GET with
    the OLD tag answers a fresh 200 carrying the new data."""
    _seed_bust_corpus(db)
    try:
        version_before = db.scalar(select(func.max(SourcesMeta.version)))
        tree_url = "/api/docs/tree"
        node_url = "/api/docs/tree/node?source=Homelab&path=a"
        old_tree = admin_client.get(tree_url).headers["etag"]
        old_node = admin_client.get(node_url).headers["etag"]
        r = admin_client.patch(
            "/api/folders/summary",
            json={"source": "Homelab", "folder_path": "a", "summary": "Handwritten."},
        )
        assert r.status_code == 200
        t = admin_client.get(tree_url, headers={"If-None-Match": old_tree})
        assert t.status_code == 200 and t.headers["etag"] != old_tree
        assert _find_child(t.json()["sources"][0], "a")["summary"] == "Handwritten."
        n = admin_client.get(node_url, headers={"If-None-Match": old_node})
        assert n.status_code == 200 and n.headers["etag"] != old_node
        assert n.json()["summary"] == "Handwritten."
        # The date edit moves the subtree max (deep.md is "a"'s max in
        # this corpus) — the tree, the node AND /api/docs all change.
        old_tree2 = admin_client.get(tree_url).headers["etag"]
        old_docs = admin_client.get("/api/docs").headers["etag"]
        deep_max_before = _find_child(
            admin_client.get(tree_url).json()["sources"][0], "a"
        )["updated_at"]
        r = admin_client.patch(
            "/api/documents/date",
            json={"source": "Homelab", "path": "a/b/c/deep.md", "date": "2020-01-01"},
        )
        assert r.status_code == 200
        t2 = admin_client.get(tree_url, headers={"If-None-Match": old_tree2})
        assert t2.status_code == 200 and t2.headers["etag"] != old_tree2
        deep_max_after = _find_child(t2.json()["sources"][0], "a")["updated_at"]
        assert deep_max_after != deep_max_before  # the body really moved
        d2 = admin_client.get("/api/docs", headers={"If-None-Match": old_docs})
        assert d2.status_code == 200 and d2.headers["etag"] != old_docs
        deep = next(d for d in d2.json()["documents"] if d["path"] == "a/b/c/deep.md")
        assert deep["created_at"].startswith("2020-01-01")
        # A4's other half: NEITHER PATCH bumped the KB generation (that
        # would mark every saved chat stale — phase 53).
        assert db.scalar(select(func.max(SourcesMeta.version))) == version_before
    finally:
        _truncate_tree_tables(db)
 def test_gzipped_catalogue_200_carries_etag_and_conditional_304(
    admin_client: TestClient, db
 ) -> None:
    """The compression layer and the validator coexist: the gzipped 200
    of the tree still carries the ``ETag`` (the header survives the
    encoding), and the conditional request answers a body-less 304 —
    no ``Content-Encoding`` on a response with no body."""
    _seed_gzip_corpus(db)
    try:
        # depth=8: the whole 50-doc tree stays above the 860-byte floor
        # (the default depth-1 body would ride under it, identity).
        r = admin_client.get("/api/docs/tree?depth=8", headers={"Accept-Encoding": "gzip"})
        assert r.status_code == 200
        assert r.headers["content-encoding"] == "gzip"
        tag = r.headers["etag"]
        assert tag.startswith('W/"')
        assert r.headers["cache-control"] == "private, must-revalidate"
        hit = admin_client.get(
            "/api/docs/tree?depth=8",
            headers={"Accept-Encoding": "gzip", "If-None-Match": tag},
        )
        assert hit.status_code == 304
        assert hit.content == b""
        assert "content-encoding" not in hit.headers
        assert hit.headers["etag"] == tag
        assert hit.headers["cache-control"] == "private, must-revalidate"
    finally:
        _truncate_tree_tables(db)
@@ -154,7 +154,10 @@ def test_tree_file_dates_and_derived_subtree_max(admin_client, db) -> None:
    _seed_doc(db, "Homelab", "a/top.md", "Top", 1, base, C2)
    _seed_doc(db, "Homelab", "root.md", "Root", 0, base, C0)
    try:
-        r = admin_client.get("/api/docs/tree")
+        # Phase 167 (task 02): this D9 walk visits ``a`` → ``a/b`` →
        # ``a/b/c``, so it asks for the WHOLE tree (depth=8); the
        # endpoint's default (depth=1) is the RAG view's first paint.
        r = admin_client.get("/api/docs/tree?depth=8")
        assert r.status_code == 200
        sources = r.json()["sources"]
        # Registry order leads; the 0-document registered source lists.
@@ -198,6 +201,49 @@ def test_tree_file_dates_and_derived_subtree_max(admin_client, db) -> None:
        _truncate(db)
 def test_tree_truncated_folder_keeps_the_recursive_date_at_default_depth(
    admin_client, db
 ) -> None:
    """Phase 167 (task 02, D2/A2) — the truncation-proof counterpart of
    the D9 walk above: at the DEFAULT depth (the view's first paint)
    the deep levels are not sent, but the truncated folder node still
    carries the phase-106 D9 contract — ``updated_at`` = the subtree's
    recursive MAX (deep C3) and ``documents`` = the recursive count —
    alongside ``truncated: true`` and ``children: []`` (the builder's
    max-date map is what buys this: the payload diet never loses the
    date column). The server ``totals`` ride the same response.
    """
    _truncate(db)
    base = datetime.now(UTC)
    db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
    _seed_doc(db, "Homelab", "a/b/c/deep.md", "Deep", 1, base, C3)
    _seed_doc(db, "Homelab", "a/b/shallow.md", "Shallow", 0, base, C1)
    _seed_doc(db, "Homelab", "a/top.md", "Top", 1, base, C2)
    _seed_doc(db, "Homelab", "root.md", "Root", 0, base, C0)
    try:
        r = admin_client.get("/api/docs/tree")  # the DEFAULT depth = 1
        assert r.status_code == 200
        body = r.json()
        homelab = _tree_node(body["sources"], "Homelab")
        # One level only: the folder ``a`` is present but NOT built out.
        (a,) = [c for c in homelab["children"] if c["kind"] == "folder"]
        assert (a["path"], a["truncated"], a["children"]) == ("a", True, [])
        # The truncation-proof fields: recursive count + recursive date.
        assert a["documents"] == 3
        assert a["updated_at"] == C3.isoformat()  # the DEEP max, unbuilt children
        # The source node's own max still spans the whole (unsent) subtree.
        assert homelab["updated_at"] == C3.isoformat()
        assert "truncated" not in homelab  # depth >= 1: never truncated, key omitted
        # Server totals beside the depth-limited tree (the stat cards).
        assert body["totals"] == {
            "documents": 4,
            "chunks": 2,
            "last_indexed": base.isoformat(),
        }
    finally:
        _truncate(db)
 # ---------------------------------------------------------------------------
 # PATCH /api/documents/date (D7) — the admin document-date editor.
 # ---------------------------------------------------------------------------
@@ -0,0 +1,238 @@
 """Integration: the byte-budget log line on the REAL app stack (phase 167, task 05).
 ``tests/unit/test_response_budget.py`` pins the middleware on a bare ASGI
 app; this file pins what only the real stack can prove —
 * the line describes the REAL ``GET /api/docs/tree``: ``bytes=`` equals the
  JSON the route produced, ``enc=identity`` for an identity client and
  ``enc=gzip`` through the actual ``SelectiveGZipMiddleware`` (the
  reading that works because the middleware sits INSIDE the compressor),
 * a conditional revalidation of that route logs ``status=304 bytes=0`` —
  the phase-167 win visible in the log, and
 * the level follows ``log_large_response_bytes`` (DEBUG under it, INFO at
  or over it, and ``/api/health`` under the 64 KB default), while
  non-``/api/`` paths (the shell page through the caching + compression
  layers) write nothing.
 Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
 """
 from __future__ import annotations
 import itertools
 import logging
 import re
 from datetime import UTC, datetime
 from typing import Any
 import pytest
 from fastapi.testclient import TestClient
 from sqlalchemy import text
 import app.core.response_budget as budget_mod
 from app.config import Settings
 from app.main import app as fastapi_app
 from app.models import Chunk, Document
 _LINE_RE = re.compile(
    r"^api method=(?P<method>[A-Z]+) path=(?P<path>\S+) status=(?P<status>\d+) "
    r"bytes=(?P<bytes>\d+) enc=(?P<enc>\S+) duration_ms=(?P<ms>\d+)$"
 )
 _counter = itertools.count()
 def _clean(db) -> None:
    db.execute(text("TRUNCATE chunks, documents"))
    db.commit()
 def _seed_corpus(db, count: int = 50) -> None:
    """``count`` indexed docs — the tree response lands well above the
    860-byte gzip floor (the corpus that motivates the phase is ~1 k docs)."""
    _clean(db)
    base = datetime.now(UTC)
    for i in range(count):
        doc = Document(
            source="budget-corpus",
            path=f"docs/sub-{i % 5}/file-{i:02d}.md",
            full_path=f"/tmp/budget-corpus/docs/sub-{i % 5}/file-{i:02d}.md",
            title=f"Doc {i:02d}",
            content=f"# Doc {i:02d}\n\nBody of document {i:02d}.",
            content_hash=f"{next(_counter):064d}",
            indexed_at=base,
        )
        db.add(doc)
        db.flush()
        db.add(
            Chunk(
                document_id=doc.id,
                position=0,
                content=f"chunk of doc {i:02d}",
                embedding=[0.01] * 768,
            )
        )
    db.commit()  # the endpoint session reads committed data only
 def _budget_lines(
    caplog: pytest.LogCaptureFixture, path: str | None = None
 ) -> list[tuple[int, str]]:
    """``(levelno, message)`` for the budget logger, optionally narrowed to
    one request path (the ``admin_client`` fixture's own login is a real
    ``/api/*`` response too)."""
    lines = [
        (record.levelno, record.getMessage())
        for record in caplog.records
        if record.name == "app.response_budget"
    ]
    if path is None:
        return lines
    return [(level, message) for level, message in lines if f" path={path} " in message]
 def test_tree_response_logs_the_exact_line_for_the_route_bytes(
    admin_client: TestClient, db, caplog: pytest.LogCaptureFixture
 ) -> None:
    """``bytes=`` is what the route produced — with an identity client that
    is exactly the body the caller received."""
    _seed_corpus(db)
    try:
        with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
            response = admin_client.get(
                "/api/docs/tree?depth=8", headers={"Accept-Encoding": "identity"}
            )
        assert response.status_code == 200
        lines = _budget_lines(caplog, "/api/docs/tree")
        assert len(lines) == 1
        match = _LINE_RE.match(lines[0][1])
        assert match is not None, lines[0][1]
        assert match.group("method") == "GET"
        assert match.group("status") == "200"
        assert int(match.group("bytes")) == len(response.content)
        assert match.group("enc") == "identity"
        assert int(match.group("ms")) >= 0
    finally:
        _clean(db)
 def test_gzipped_tree_logs_enc_gzip_with_the_uncompressed_bytes(
    admin_client: TestClient, db, caplog: pytest.LogCaptureFixture
 ) -> None:
    """The D5 reading, end to end: the client received gzip and the log says
    ``enc=gzip`` while ``bytes=`` stays the ROUTE's size — which is why the
    middleware is wired inside the compression layer."""
    _seed_corpus(db)
    try:
        with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
            identity = admin_client.get(
                "/api/docs/tree?depth=8", headers={"Accept-Encoding": "identity"}
            )
            response = admin_client.get(
                "/api/docs/tree?depth=8", headers={"Accept-Encoding": "gzip"}
            )
        assert response.status_code == 200
        assert response.headers["content-encoding"] == "gzip"
        assert response.content == identity.content
        lines = _budget_lines(caplog, "/api/docs/tree")
        assert len(lines) == 2
        identity_line, gzip_line = (
            _LINE_RE.match(lines[0][1]),
            _LINE_RE.match(lines[1][1]),
        )
        assert identity_line is not None and identity_line.group("enc") == "identity"
        assert gzip_line is not None and gzip_line.group("enc") == "gzip"
        # Same route bytes in both lines…
        assert identity_line.group("bytes") == gzip_line.group("bytes")
        # …and that is the ROUTE's size, not the compressed transfer.
        assert int(gzip_line.group("bytes")) == len(identity.content)
    finally:
        _clean(db)
 def test_conditional_revalidation_logs_zero_bytes(
    admin_client: TestClient, db, caplog: pytest.LogCaptureFixture
 ) -> None:
    """The phase-167 win is visible, not hidden: the 304 costs one line with
    ``bytes=0``."""
    _seed_corpus(db)
    try:
        fresh = admin_client.get("/api/docs/tree?depth=1")
        assert fresh.status_code == 200
        etag = fresh.headers["etag"]
        with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
            revalidated = admin_client.get(
                "/api/docs/tree?depth=1", headers={"If-None-Match": etag}
            )
        assert revalidated.status_code == 304
        lines = _budget_lines(caplog, "/api/docs/tree")
        assert len(lines) == 1
        match = _LINE_RE.match(lines[0][1])
        assert match is not None
        assert match.group("status") == "304"
        assert match.group("bytes") == "0"
    finally:
        _clean(db)
 def test_level_follows_the_configured_budget(
    admin_client: TestClient,
    monkeypatch: pytest.MonkeyPatch,
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """``/api/health`` sits under the 65536 default → DEBUG; the same
    response with ``log_large_response_bytes=0`` is INFO (the loud mode). The
    middleware reads the setting per response, so this is the operator's
    runtime knob, not a boot-time one."""
    with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
        assert admin_client.get("/api/health").status_code == 200
    assert [level for level, _ in _budget_lines(caplog, "/api/health")] == [logging.DEBUG]
    monkeypatch.setattr(
        budget_mod,
        "get_settings",
        lambda: Settings(_env_file=None, log_large_response_bytes=0),  # pyright: ignore[reportCallIssue]
    )
    caplog.clear()
    with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
        assert admin_client.get("/api/health").status_code == 200
    assert [level for level, _ in _budget_lines(caplog, "/api/health")] == [logging.INFO]
 def test_html_page_writes_no_budget_line(
    admin_client: TestClient, caplog: pytest.LogCaptureFixture
 ) -> None:
    """The shell page rides the caching rewrite AND the compression layer
    and still writes nothing — the budget measures ``/api/*`` only."""
    with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
        response = admin_client.get("/sources.html", headers={"Accept-Encoding": "gzip"})
        assert response.status_code == 200
        assert response.headers["content-encoding"] == "gzip"
    assert not any(" path=/sources.html " in message for _, message in _budget_lines(caplog))
 def test_anonymous_403_is_measured_like_any_other_response(
    client: TestClient, caplog: pytest.LogCaptureFixture
 ) -> None:
    """The gate is a response too — an unauthenticated catalogue read logs
    its status (and the byte line is what makes a chatty 403 visible)."""
    with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
        assert client.get("/api/docs").status_code == 403
    lines = _budget_lines(caplog, "/api/docs")
    assert len(lines) == 1
    match: Any = _LINE_RE.match(lines[0][1])
    assert match is not None
    assert match.group("status") == "403"
 def test_the_module_level_app_carries_the_middleware() -> None:
    """Smoke: the wiring pin in ``tests/unit/test_response_budget.py`` covers
    the ORDER on a fresh ``create_app()``; this proves the module-level ``app``
    the whole suite (and the dev server) runs against has the layer too."""
    names = [getattr(mw.cls, "__name__", "") for mw in fastapi_app.user_middleware]
    assert "ResponseBudgetMiddleware" in names
@@ -0,0 +1,497 @@
 """Unit tests: ``app.core.compression.SelectiveGZipMiddleware`` (phase 167, D1, LOCKED A1).
 Pins the three contracts of the selective-gzip layer on a tiny ASGI app —
 * a ~50 KB JSON route compresses (``Content-Encoding: gzip``, the body
  ``gzip.decompress``es to the original, ``Vary`` mentions
  ``accept-encoding``), an ``Accept-Encoding: identity`` client gets
  identity bytes, a ~50-byte route stays under the floor, and a response
  that already set ``content-encoding`` is forwarded byte-identical (never
  double-encoded — the fronting-nginx contract);
 * the EXACT-path skip: a route named ``/api/chat`` is NOT compressed and
  streams frame-by-frame (the raw-ASGI interleaving proof: every frame
  reaches the client's ``send`` BEFORE the app yields the next one — a
  buffering middleware fails this deterministically — plus the
  ``list(stream)`` client idiom the existing SSE tests use), while
  ``/api/chat-images/x`` shares the prefix and MUST compress;
 * non-HTTP scopes (``lifespan``/``websocket``) pass straight through.
 Plus the config kill switch: ``compress_min_bytes=0`` → ``create_app``
 adds no gzip layer at all; ``-1`` → the validator refuses startup naming
 the field; the middleware sits INSIDE ``SecurityHeadersMiddleware`` (A20)
 and OUTSIDE ``CachingMiddleware``.
 """
 from __future__ import annotations
 import asyncio
 import gzip
 import json
 from typing import Any
 import pytest
 from fastapi.testclient import TestClient
 from pydantic import ValidationError
 from starlette.applications import Starlette
 from starlette.middleware.gzip import GZipMiddleware
 from starlette.requests import Request
 from starlette.responses import Response, StreamingResponse
 from starlette.routing import Route
 from starlette.types import Message, Receive, Scope, Send
 import app.main as main_mod
 from app.config import Settings
 from app.core.compression import DEFAULT_SKIP_PATHS, SelectiveGZipMiddleware
 def _settings(**kwargs: Any) -> Settings:
    """Build Settings without reading a .env file (the test_config pattern)."""
    kwargs.setdefault("_env_file", None)
    return Settings(**kwargs)  # pyright: ignore[reportCallIssue]  (kwarg exists at runtime)
 # ~50 KB of highly compressible JSON — the /api/docs/tree shape class.
 _BIG_BODY = json.dumps(
    {
        "items": [
            {"n": i, "path": f"source/docs/file-{i}.md", "title": f"Doc {i}"}
            for i in range(1000)
        ]
    }
 ).encode()
 # ~50 bytes — comfortably under the 860-byte floor (compression would inflate).
 _SMALL_BODY = json.dumps({"ok": True, "note": "intentionally tiny body"}).encode("utf-8")
 # Five SSE frames (~2 KB total — ABOVE the floor, so a missing skip cannot
 # hide behind the minimum_size exemption).
 _SSE_FRAMES = [
    f"event: thinking\ndata: {'x' * 400}\n\n".encode() for _ in range(5)
 ]
 _PRE_ENCODED_BODY = gzip.compress(_BIG_BODY)
 def _inner_app() -> Starlette:
    async def big(request: Request) -> Response:
        return Response(_BIG_BODY, media_type="application/json")
    async def small(request: Request) -> Response:
        return Response(_SMALL_BODY, media_type="application/json")
    async def chat(request: Request) -> StreamingResponse:
        async def frames():
            for frame in _SSE_FRAMES:
                yield frame
        return StreamingResponse(frames(), media_type="text/event-stream")
    async def chat_images(request: Request) -> Response:
        # Shares the /api/chat PREFIX but is a different endpoint — the
        # skip must be EXACT, so this one compresses.
        return Response(_BIG_BODY, media_type="application/json")
    async def pre_encoded(request: Request) -> Response:
        # A body that arrived already encoded (e.g. from a future CDN
        # layer): the responder skips it, never double-encodes.
        return Response(
            _PRE_ENCODED_BODY,
            media_type="application/json",
            headers={"Content-Encoding": "gzip"},
        )
    return Starlette(
        routes=[
            Route("/big", big),
            Route("/small", small),
            Route("/api/chat", chat, methods=["POST"]),
            Route("/api/chat-images/x", chat_images),
            Route("/pre-encoded", pre_encoded),
        ]
    )
 def _client(minimum_size: int = 860) -> TestClient:
    return TestClient(
        SelectiveGZipMiddleware(_inner_app(), minimum_size=minimum_size)
    )
 # ---------------------------------------------------------------------------
 # The gzip path (Accept-Encoding negotiation, the floor, the pre-encoded skip)
 # ---------------------------------------------------------------------------
 def test_large_json_compresses_decompresses_and_sets_vary() -> None:
    client = _client()
    response = client.get("/big", headers={"Accept-Encoding": "gzip"})
    assert response.status_code == 200
    assert response.headers["content-encoding"] == "gzip"
    assert "accept-encoding" in response.headers.get("vary", "").lower()
    # httpx transparently decodes: the documented JSON arrives whole.
    assert response.content == _BIG_BODY
    assert len(response.content) > 860  # the seed really is above the floor
    # The RAW wire bytes are a real gzip stream of the original body.
    with client.stream("GET", "/big", headers={"Accept-Encoding": "gzip"}) as streamed:
        raw = b"".join(list(streamed.iter_raw()))
    assert gzip.decompress(raw) == _BIG_BODY
    assert len(raw) < len(_BIG_BODY)  # ~8-12x on this JSON; order of magnitude suffices
 def test_identity_client_is_not_compressed() -> None:
    client = _client()
    response = client.get("/big", headers={"Accept-Encoding": "identity"})
    assert response.status_code == 200
    assert "content-encoding" not in response.headers
    assert response.content == _BIG_BODY
 def test_small_response_stays_under_the_floor() -> None:
    client = _client()
    response = client.get("/small", headers={"Accept-Encoding": "gzip"})
    assert response.status_code == 200
    assert "content-encoding" not in response.headers
    assert response.content == _SMALL_BODY
 def test_pre_encoded_body_is_never_double_encoded() -> None:
    client = _client()
    with client.stream("GET", "/pre-encoded", headers={"Accept-Encoding": "gzip"}) as streamed:
        assert streamed.headers["content-encoding"] == "gzip"
        raw = b"".join(list(streamed.iter_raw()))
    # Byte-identical passthrough of the already-encoded body (Starlette's
    # "skip when content-encoding is set" rule — the nginx coexistence pin).
    assert raw == _PRE_ENCODED_BODY
    assert gzip.decompress(raw) == _BIG_BODY
 # ---------------------------------------------------------------------------
 # The EXACT-path skip (A15: the SSE stream is never compressed, never buffered)
 # ---------------------------------------------------------------------------
 def test_default_skip_paths_constant() -> None:
    assert DEFAULT_SKIP_PATHS == ("/api/chat",)
 def test_chat_route_is_not_compressed_and_streams_whole() -> None:
    """The client-level pin (the ``list(stream)`` idiom the SSE tests use):
    a route named /api/chat carries NO Content-Encoding and its raw bytes
    are exactly the SSE frames, concatenated, above the size floor."""
    client = _client()
    with client.stream(
        "POST", "/api/chat", json={"message": "hi"}, headers={"Accept-Encoding": "gzip"}
    ) as streamed:
        assert streamed.status_code == 200
        assert "content-encoding" not in streamed.headers
        chunks = list(streamed.iter_raw())
    assert b"".join(chunks) == b"".join(_SSE_FRAMES)
    assert sum(len(c) for c in chunks) > 860  # the skip beats the floor, not size
 def test_skipped_stream_arrives_frame_by_frame_uncompressed() -> None:
    """The frame-by-frame proof (deterministic, no timing): every frame
    reaches the client's ``send`` BEFORE the app produces the next one.
    A buffering middleware (BaseHTTPMiddleware, or a gzip pass) yields
    all ``app:*`` events first — this strict interleaving fails then."""
    events: list[str] = []  # shared: the app logs a yield, send logs delivery
    sent: list[Message] = []
    async def sse_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/event-stream")],
            }
        )
        for i, frame in enumerate(_SSE_FRAMES):
            events.append(f"app:frame-{i}")
            await send({"type": "http.response.body", "body": frame, "more_body": True})
        events.append("app:final")
        await send({"type": "http.response.body", "body": b"", "more_body": False})
    async def recording_send(message: Message) -> None:
        events.append(f"client:{message['type']}")
        sent.append(message)
    async def receive() -> Message:
        return {"type": "http.disconnect"}
    middleware = SelectiveGZipMiddleware(sse_app, minimum_size=860)
    asyncio.run(
        middleware(
            {"type": "http", "path": "/api/chat", "headers": [(b"accept-encoding", b"gzip")]},
            receive,
            recording_send,
        )
    )
    expected = ["client:http.response.start"]
    for i in range(len(_SSE_FRAMES)):
        expected += [f"app:frame-{i}", "client:http.response.body"]
    expected += ["app:final", "client:http.response.body"]
    assert events == expected
    start = sent[0]
    headers = {k.decode().lower() for k, _ in start["headers"]}  # pyright: ignore[reportGeneralTypeIssues]
    assert "content-encoding" not in headers
    body = b"".join(m.get("body", b"") for m in sent[1:])  # pyright: ignore[reportGeneralTypeIssues]
    assert body == b"".join(_SSE_FRAMES)
 def _raw_harness(
    *,
    path: str,
    app: Any,
 ) -> list[Message]:
    """Drive the middleware at the raw ASGI level; return the exact
    messages that reached the client's ``send``."""
    sent: list[Message] = []
    async def recording_send(message: Message) -> None:
        sent.append(message)
    async def receive() -> Message:
        return {"type": "http.disconnect"}
    scope: Scope = {
        "type": "http",
        "path": path,
        "headers": [(b"accept-encoding", b"gzip")],
    }
    middleware = SelectiveGZipMiddleware(app, minimum_size=860)
    asyncio.run(middleware(scope, receive, recording_send))
    return sent
 def test_same_stream_at_a_non_skipped_path_goes_through_gzip() -> None:
    """The skip is PATH-driven: the identical streaming app mounted at a
    non-skipped path is compressed by the composed GZipMiddleware."""
    async def json_stream_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"application/json")],
            }
        )
        for i, frame in enumerate(_SSE_FRAMES):
            await send(
                {"type": "http.response.body", "body": frame, "more_body": i < len(_SSE_FRAMES) - 1}
            )
    sent = _raw_harness(path="/api/other-stream", app=json_stream_app)
    start_headers = {
        k.decode().lower(): v.decode() for k, v in sent[0]["headers"]  # pyright: ignore[reportGeneralTypeIssues]
    }
    assert start_headers.get("content-encoding") == "gzip"
    raw = b"".join(m.get("body", b"") for m in sent[1:])  # pyright: ignore[reportGeneralTypeIssues]
    assert gzip.decompress(raw) == b"".join(_SSE_FRAMES)
 def test_reframed_small_body_stays_identity_under_the_floor() -> None:
    """The CachingMiddleware seam (starlette 1.x ``BaseHTTPMiddleware``
    re-frames every response it passes up: chunk with ``more_body=True``
    + empty terminator): a body whose ``content-length`` is below the
    floor is re-buffered into ONE buffered message so Starlette's own
    small-response exemption fires — identity bytes, no ``Vary`` append."""
    payload = b'{"status":"ok","version":"0.1.0"}'  # 33 bytes — the /api/health class
    async def reframed_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [
                    (b"content-type", b"application/json"),
                    (b"content-length", str(len(payload)).encode()),
                ],
            }
        )
        await send({"type": "http.response.body", "body": payload, "more_body": True})
        await send({"type": "http.response.body", "body": b"", "more_body": False})
    sent = _raw_harness(path="/api/health", app=reframed_app)
    start_headers = {
        k.decode().lower(): v.decode() for k, v in sent[0]["headers"]  # pyright: ignore[reportGeneralTypeIssues]
    }
    assert "content-encoding" not in start_headers
    assert "vary" not in start_headers  # the exemption adds nothing
    body = b"".join(m.get("body", b"") for m in sent[1:])  # pyright: ignore[reportGeneralTypeIssues]
    assert body == payload
 def test_lying_content_length_that_outgrows_the_floor_compresses() -> None:
    """A body that outgrows the floor despite a small ``content-length``
    stops the hold and streams through the normal compressing path — the
    buffer is bounded by the floor and correctness never depends on the
    header being honest."""
    async def lying_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [
                    (b"content-type", b"application/json"),
                    (b"content-length", b"5"),  # a lie
                ],
            }
        )
        await send({"type": "http.response.body", "body": _BIG_BODY, "more_body": True})
        await send({"type": "http.response.body", "body": b"", "more_body": False})
    sent = _raw_harness(path="/api/lias", app=lying_app)
    start_headers = {
        k.decode().lower(): v.decode() for k, v in sent[0]["headers"]  # pyright: ignore[reportGeneralTypeIssues]
    }
    assert start_headers.get("content-encoding") == "gzip"
    raw = b"".join(m.get("body", b"") for m in sent[1:])  # pyright: ignore[reportGeneralTypeIssues]
    assert gzip.decompress(raw) == _BIG_BODY
 def test_non_body_message_while_holding_flushes_untouched() -> None:
    """``http.response.pathsend`` (or anything unexpected) arriving while
    the hold is active: the held messages and the message itself are
    forwarded byte-for-byte — the hold never swallows a frame."""
    async def pathsend_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [
                    (b"content-type", b"text/plain"),
                    (b"content-length", b"4"),
                ],
            }
        )
        await send({"type": "http.response.pathsend", "path": "/tmp/file"})
    sent = _raw_harness(path="/assets/file", app=pathsend_app)
    assert [m["type"] for m in sent] == ["http.response.start", "http.response.pathsend"]
    start_headers = {k.decode().lower() for k, _ in sent[0]["headers"]}  # pyright: ignore[reportGeneralTypeIssues]
    assert "content-encoding" not in start_headers
 def test_chat_images_prefix_still_compresses() -> None:
    """The exact-match pin: /api/chat-images/x shares the /api/chat
    prefix and MUST go through the gzip path."""
    client = _client()
    response = client.get("/api/chat-images/x", headers={"Accept-Encoding": "gzip"})
    assert response.status_code == 200
    assert response.headers["content-encoding"] == "gzip"
    assert response.content == _BIG_BODY
 # ---------------------------------------------------------------------------
 # Non-HTTP scopes pass straight through
 # ---------------------------------------------------------------------------
 def test_non_http_scopes_pass_through_untouched() -> None:
    seen: list[str] = []
    async def probe_app(scope: Scope, receive: Receive, send: Send) -> None:
        seen.append(scope["type"])
    async def receive() -> Message:
        return {"type": "http.disconnect"}
    async def send(message: Message) -> None:  # pragma: no cover - app sends nothing
        raise AssertionError("no ASGI messages expected")
    middleware = SelectiveGZipMiddleware(probe_app, minimum_size=860)
    for scope_type in ("lifespan", "websocket"):
        asyncio.run(
            middleware({"type": scope_type, "path": "/api/anything"}, receive, send)
        )
    assert seen == ["lifespan", "websocket"]
 def test_composes_starlette_gzip_with_minimum_size() -> None:
    """The middleware reuses Starlette's GZipMiddleware, it does not fork
    the compression math (D1)."""
    async def app(scope: Scope, receive: Receive, send: Send) -> None: ...
    middleware = SelectiveGZipMiddleware(app, minimum_size=1234)
    assert isinstance(middleware.gzip, GZipMiddleware)
    assert middleware.gzip.minimum_size == 1234
    assert middleware.skip_paths == DEFAULT_SKIP_PATHS
 # ---------------------------------------------------------------------------
 # Config: the BOR_COMPRESS_MIN_BYTES kill switch and the loud negative
 # ---------------------------------------------------------------------------
 def _middleware_names(app: Any) -> list[str]:
    """Middleware class names, outermost first (``user_middleware`` order)."""
    return [getattr(mw.cls, "__name__", "") for mw in app.user_middleware]
 def test_compress_min_bytes_defaults_to_the_nginx_floor(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    monkeypatch.delenv("BOR_COMPRESS_MIN_BYTES", raising=False)
    assert _settings().compress_min_bytes == 860
 def test_compress_min_bytes_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
    monkeypatch.setenv("BOR_COMPRESS_MIN_BYTES", "2048")
    assert _settings().compress_min_bytes == 2048
    monkeypatch.setenv("BOR_COMPRESS_MIN_BYTES", "0")  # the kill switch is legal
    assert _settings().compress_min_bytes == 0
 def test_compress_min_bytes_negative_refuses_startup_naming_the_field() -> None:
    with pytest.raises(ValidationError, match="compress_min_bytes must be >= 0"):
        _settings(compress_min_bytes=-1)
 def test_create_app_wires_gzip_inside_security_headers(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    """A20 holds: SecurityHeaders stays outermost (index 0 — starlette's
    last add_middleware is the outermost layer), gzip sits between it and
    the caching layer, so an already-rewritten HTML page compresses and
    the /api/* caching pass-through happens before compression.
    Phase 167 task 05 adds ``ResponseBudgetMiddleware`` between gzip and
    caching — INSIDE the compressor on purpose, so its ``bytes=`` counts the
    body the ROUTE produced while its ``enc=`` still reads the encoding the
    compressor put on the wire (see tests/unit/test_response_budget.py)."""
    monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 860)
    app2 = main_mod.create_app()
    assert _middleware_names(app2) == [
        "SecurityHeadersMiddleware",
        "SelectiveGZipMiddleware",
        "ResponseBudgetMiddleware",
        "CachingMiddleware",
        "SessionMiddleware",
    ]
    # The app still serves.
    assert TestClient(app2).get("/api/health").status_code == 200
 def test_create_app_adds_no_gzip_layer_when_disabled(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    """BOR_COMPRESS_MIN_BYTES=0 removes the middleware entirely (the kill
    switch): the gzip class is absent from app.user_middleware."""
    monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 0)
    app2 = main_mod.create_app()
    names = _middleware_names(app2)
    assert "SelectiveGZipMiddleware" not in names
    assert names == [
        "SecurityHeadersMiddleware",
        # Phase 167 task 05: the byte-budget line is not optional.
        "ResponseBudgetMiddleware",
        "CachingMiddleware",
        "SessionMiddleware",
    ]
@@ -572,11 +572,13 @@ def test_rag_view_refetches_on_reshow() -> None:
    """Phase 77 task 02 (+ phase 97 task 04, + phase 99 task 02):
    the RAG (knowledge base) view re-fetches on a user-initiated
    re-show — sources.js listens and re-runs ``loadTree()`` (the
-    phase-97 catalog load: ONE fetch of ``GET /api/docs/tree``), now
+    catalog load — ONE fetch of ``GET /api/docs/tree?depth=1`` since
-    AFTER the phase-99 history alignment (adopt the entry's kb / reset
+    phase 167's payload diet; the view loads the catalogue from the
-    to the top — a block-arrow listener, hence the ``});`` close in
+    TREE endpoint, not the flat one), now AFTER the phase-99 history
-    the pin helper below). The load is race-tokened (phase 79) and
+    alignment (adopt the entry's kb / reset to the top — a block-arrow
-    the RE-ENTRANT render — ``renderLevel`` clears BOTH row containers
+    listener, hence the ``});`` close in the pin helper below). The load
    is race-tokened (phase 79) and the RE-ENTRANT render —
    ``renderLevel`` clears BOTH row containers
    at the TOP before filling them (the History pattern from task 01,
    extended to the folders table) — so a refresh from a populated
    level into a sparser (or empty) result replaces the rows instead
@@ -588,7 +590,9 @@ def test_rag_view_refetches_on_reshow() -> None:
    load = js.find("async function loadTree()")
    assert load != -1, "loadTree must exist"
    body = js[load : js.find("\n  }", load)]
-    assert 'fetch("/api/docs/tree")' in body, "the load must fetch the tree endpoint"
+    assert 'fetch("/api/docs/tree?depth=1")' in body, (
        "the load must fetch the tree endpoint (depth-limited, phase 167)"
    )
    assert "++loadSeq" in body, "the race token stays (phase 79)"
    render = js.find("function renderLevel()")
    assert render != -1, "renderLevel must exist"
@@ -0,0 +1,110 @@
 """Unit tests: ``app/core/http_cache.py`` (phase 167, task 03).
 The three pure helpers behind the catalogue reads' revalidation
 (LOCKED A4): the weak content-hash tag (shape, stability, the bytes-vs-
 str equality), the tolerant ``If-None-Match`` evaluator (``None`` /
 empty / ``*`` / single / comma-list / weak↔strong both directions /
 whitespace / the prefix-is-not-a-match pin), and the ONE header pair.
 """
 from __future__ import annotations
 import re
 from app.core.http_cache import (
    REVALIDATE_CACHE_CONTROL,
    etag_matches,
    revalidate_headers,
    weak_etag,
 )
 _TAG_RE = re.compile(r'W/"[0-9a-f]{32}"')
 def test_weak_etag_shape_is_weak_32_hex() -> None:
    """``W/"<sha256 hexdigest[:32]>"`` — the prefix, the 32-hex-char body,
    the total 36-char length (the file's truncation convention)."""
    tag = weak_etag("hello")
    assert _TAG_RE.fullmatch(tag), tag
    assert len(tag) == 36
    assert tag.startswith('W/"') and tag.endswith('"')
 def test_weak_etag_is_deterministic_and_content_sensitive() -> None:
    assert weak_etag("same body") == weak_etag("same body")
    assert weak_etag("same body") != weak_etag("same body!")
    assert weak_etag("") != weak_etag(" ")
 def test_weak_etag_bytes_and_str_forms_agree() -> None:
    """The hash is over the UTF-8 bytes either way — the endpoint may
    hash the ``str`` and serve its encoding without diverging."""
    text = '{"tree": "héllo — ünïcode"}'
    assert weak_etag(text) == weak_etag(text.encode("utf-8"))
 def test_weak_etag_truncation_matches_the_sha256_prefix() -> None:
    import hashlib
    body = "exact served bytes"
    expected = hashlib.sha256(body.encode("utf-8")).hexdigest()[:32]
    assert weak_etag(body) == f'W/"{expected}"'
 def test_etag_matches_none_and_empty_never_match() -> None:
    """No conditional header (or an empty/whitespace-only one) is an
    unconditional GET → the route must answer 200."""
    assert etag_matches(None, 'W/"x"') is False
    assert etag_matches("", 'W/"x"') is False
    assert etag_matches("   ", 'W/"x"') is False
 def test_etag_matches_star() -> None:
    """RFC 9110 §13.3.3: ``*`` matches anything the server holds."""
    assert etag_matches("*", 'W/"x"') is True
    assert etag_matches(" * ", 'W/"anything"') is True
 def test_etag_matches_single_candidate() -> None:
    assert etag_matches('W/"x"', 'W/"x"') is True
    assert etag_matches('W/"x"', 'W/"y"') is False
 def test_etag_matches_comma_separated_list() -> None:
    header = 'W/"a", W/"b", W/"c"'
    assert etag_matches(header, 'W/"a"') is True
    assert etag_matches(header, 'W/"c"') is True
    assert etag_matches(header, 'W/"d"') is False
    # no space after the comma — a real client sends both shapes
    assert etag_matches('W/"a",W/"b"', 'W/"b"') is True
 def test_etag_matches_weak_strong_equivalence_both_directions() -> None:
    """``W/"x"`` matches ``"x"`` whichever side carries the weakness
    marker (the stored form must not decide the outcome)."""
    assert etag_matches('"x"', 'W/"x"') is True  # client strong, ours weak
    assert etag_matches('W/"x"', '"x"') is True  # client weak, ours strong
 def test_etag_matches_is_whitespace_tolerant() -> None:
    assert etag_matches('  W/"x"  ', 'W/"x"') is True
    assert etag_matches('W/"a" ,  "x" ', 'W/"x"') is True
    assert etag_matches('W/ "x"', 'W/"x"') is True
 def test_etag_matches_prefix_is_not_a_match() -> None:
    """A tag that merely SHARES a prefix with ours is a mismatch — the
    comparison is the full 32-hex-char digest, not ``startswith``."""
    ours = "abcdef0123456789abcdef0123456789"
    shorter = ours[:-1]
    assert etag_matches(f'W/"{shorter}"', f'W/"{ours}"') is False
    assert etag_matches(f'"{shorter}"', f'W/"{ours}"') is False
 def test_revalidate_headers_is_the_one_pair() -> None:
    """The 200 AND the 304 carry exactly this pair (A19's read-only API
    arm: private + must-revalidate beside the validator)."""
    assert revalidate_headers('W/"x"') == {
        "ETag": 'W/"x"',
        "Cache-Control": "private, must-revalidate",
    }
    assert REVALIDATE_CACHE_CONTROL == "private, must-revalidate"
@@ -11,8 +11,18 @@ file metadata, and the ``group_folder_listing`` cross-check property
 """
 from __future__ import annotations
-from app.api.docs import TreeDocRow, build_kb_tree
+import pytest
 from pydantic import ValidationError
 from app.api.docs import (
    TreeDocRow,
    _folder_counts,
    _kb_totals,
    _subtree_max,
    build_kb_tree,
 )
 from app.rag.agent import group_folder_listing
 from app.schemas import KbTree, KbTreeFile, KbTreeTotals
 T0 = "2026-09-01T08:00:00+00:00"
 T1 = "2026-09-02T08:00:00+00:00"
@@ -633,3 +643,284 @@ def test_image_affordance_absent_without_the_map() -> None:
    assert set(file.model_dump()) == {
        "kind", "path", "title", "chunks", "created_at", "indexed_at"
    }
 # ---------------------------------------------------------------------------
 # Phase 167 (task 02, D2/A2) — the depth budget: a node whose children were
 # NOT built says so (``truncated``) while keeping every field the Sources
 # view renders (recursive count, recursive updated_at — the D9 max via the
 # _folder_counts max-date map, summary, summary_pending), the untruncated
 # wire shape stays byte-identical, and the catalogue totals ride the
 # response (``_kb_totals`` — the stat cards cannot be summed client-side
 # off a truncated tree).
 # ---------------------------------------------------------------------------
 #: The depth fixture: one chain four levels deep, distinct created dates
 #: (C0 < C1 < C2 < C3 — the DEEPEST file holds the max, the D9 trap) and
 #: distinct indexed stamps.
 DEPTH_ROWS: list[TreeDocRow] = [
    ("S", "root.md", "Root", 1, T0, C0),
    ("S", "a/top.md", "Top", 2, T1, C1),
    ("S", "a/b/mid.md", "Mid", 3, T1, C2),
    ("S", "a/b/c/deep.md", "Deep", 4, T2, C3),
 ]
 DEPTH_SUMMARIES = {("S", "a"): "A desc."}  # a/b and a/b/c stay un-stored
 def _all_nodes(node):
    """The node plus every descendant node (folders and files)."""
    yield node
    for child in getattr(node, "children", []):
        if child.kind == "folder":
            yield from _all_nodes(child)
        else:
            yield child
 def _all_folder_nodes(node):
    """Every FOLDER descendant (the SOURCE node itself is excluded — it
    carries ``name``, not ``path``, and has no ``kind``)."""
    return [n for n in _all_nodes(node) if getattr(n, "kind", None) == "folder"]
 def _paths_under(node) -> set[str]:
    """Paths of every descendant (folders and files); the source node
    has no path of its own."""
    out: set[str] = set()
    for child in getattr(node, "children", []):
        out.add(child.path)
        if child.kind == "folder":
            out |= _paths_under(child)
    return out
 def test_depth_none_is_byte_identical_to_the_unlimited_build() -> None:
    """``depth=None`` (the builder default) reproduces today's tree
    byte-for-byte: the default call, the explicit None, and an
    over-deep budget all serialize identically, and NO node carries the
    truncated key (the omission rule)."""
    default = [s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)]
    explicit = [
        s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=None)
    ]
    over_deep = [
        s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=8)
    ]
    assert default == explicit == over_deep
    (source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
    assert all("truncated" not in n.model_dump() for n in _all_nodes(source))
    # The full chain IS built: every level of the fixture is present.
    assert _paths_under(source) == {
        "root.md",
        "a",
        "a/top.md",
        "a/b",
        "a/b/mid.md",
        "a/b/c",
        "a/b/c/deep.md",
    }
 def test_depth_one_ships_sources_plus_one_level_with_truncated_flags() -> None:
    """``depth=1``: the source's direct children are built (direct file
    ``root.md`` included) and the subfolder with unloaded contents
    carries the FULL truncation field set — ``truncated is True``,
    ``children == []``, the RECURSIVE count, the RECURSIVE max date
    (the deep C3 — D9 survives without the children), the stored
    ``summary`` and the ``summary_pending`` flag (rule unchanged)."""
    (source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=1)
    assert [f.path for f in _file_nodes(source)] == ["root.md"]
    (a,) = _folder_nodes(source)
    assert a.path == "a"
    assert a.truncated is True
    assert a.model_dump()["truncated"] is True
    assert a.children == []
    assert a.documents == 3  # recursive — the count rule is depth-proof
    assert a.updated_at == C3  # the DEEP max, no built children to derive from
    assert a.summary == "A desc."  # the level row shows a subfolder's description
    assert a.summary_pending is False  # the stored row covers it (phase 98)
    # Nothing deeper is built anywhere.
    assert _paths_under(source) == {"root.md", "a"}
 def test_depth_two_builds_two_levels_and_truncates_the_deeper_one() -> None:
    """``depth=2``: level 1 (``a``) is expanded UNTRUNCATED (its direct
    file ``a/top.md`` ships), level 2 (``a/b``) is the truncated node —
    the pending rule applies to it (2 docs, no stored row →
    ``summary_pending`` true with ``summary None``), and neither its
    own file (``a/b/mid.md``) nor the third level appear: the budget
    removes the LEVELS below it, level by level."""
    (source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=2)
    (a,) = _folder_nodes(source)
    assert "truncated" not in a.model_dump()  # untruncated: key absent
    assert [f.path for f in _file_nodes(a)] == ["a/top.md"]
    (a_b,) = _folder_nodes(a)
    assert a_b.truncated is True
    assert a_b.children == []
    assert (a_b.documents, a_b.updated_at) == (2, C3)  # mid + deep, max C3
    assert a_b.summary is None
    assert a_b.summary_pending is True  # the phase-98 rule rides truncated nodes
    assert _paths_under(source) == {
        "root.md",
        "a",
        "a/top.md",
        "a/b",
    }
 def test_source_node_is_never_truncated_at_any_depth() -> None:
    """The view's top level is source → level 1, so the budget never
    marks the SOURCE node truncated (at any depth, empty sources
    included) — its wire shape never carries the key."""
    (ref,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
    for depth in (1, 2, 3, 8):
        full, empty = build_kb_tree(
            ["S", "Empty"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=depth
        )
        assert full.truncated is False
        assert "truncated" not in full.model_dump()
        assert empty.truncated is False
        assert "truncated" not in empty.model_dump()
        assert empty.children == []
        # Direct children always built: the source's level is complete.
        assert [c.path for c in full.children] == [c.path for c in ref.children]
 def test_truncated_key_never_touches_the_file_node_shape() -> None:
    """A FILE node has no children to truncate: the field does not even
    exist on the class, and no dump ever carries it (the phase-122 text
    wire shape stays byte-identical at every depth)."""
    assert "truncated" not in KbTreeFile.model_fields
    for depth in (1, 2, 8):
        (source,) = build_kb_tree(["S"], DEPTH_ROWS, {}, depth=depth)
        file_nodes = [n for n in _all_nodes(source) if getattr(n, "kind", None) == "file"]
        assert file_nodes  # the fixture has files at every shipped level
        for node in file_nodes:
            assert "truncated" not in node.model_dump()
 def test_max_dates_map_agrees_with_subtree_max_at_every_depth() -> None:
    """One concept, two derivations, cross-pinned: the
    ``_folder_counts`` max-date map equals ``_subtree_max`` over the
    FULLY expanded children for every folder (the expanded builder path
    stays ``_subtree_max``, so a divergence fails loudly) — and every
    node present in a depth-limited tree carries the SAME
    ``updated_at`` as the full build, truncated or not (the phase-106
    D9 column survives the payload diet)."""
    file_rows = [
        (path, title, chunks, indexed, created)
        for _source, path, title, chunks, indexed, created in DEPTH_ROWS
    ]
    _folders, _counts, max_dates = _folder_counts(file_rows)
    (full,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
    full_dates = {n.path: n.updated_at for n in _all_folder_nodes(full)}
    assert full_dates == max_dates  # the cross-pin at full expansion
    for node in _all_folder_nodes(full):
        # The expanded node's derivation equals the aggregate map.
        assert _subtree_max(node.children) == max_dates[node.path]
    for depth in (1, 2, 3):
        (limited,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=depth)
        for node in _all_folder_nodes(limited):
            assert node.updated_at == max_dates[node.path] == full_dates[node.path]
        # The source max also spans the UNSENT levels.
        assert limited.updated_at == C3
 def test_truncation_keeps_the_phase_94_and_98_invariants_at_every_depth() -> None:
    """Existence rule, ordering, counts, summaries, pending markers and
    dates: every node a limited tree DOES send is field-for-field the
    node the full tree sends (the depth budget removes depth only — it
    never changes what a shipped level says), and the top level is
    identical at every depth."""
    (full,) = build_kb_tree(["S"], CROSS_ROWS, CROSS_SUMMARIES)
    full_map = {
        n.path: (n.documents, n.summary_pending, n.updated_at, n.summary)
        for n in _all_folder_nodes(full)
    }
    for depth in (1, 2, 3):
        (limited,) = build_kb_tree(["S"], CROSS_ROWS, CROSS_SUMMARIES, depth=depth)
        # Same level-1 rows, same order, at every depth.
        assert [c.path for c in limited.children] == [c.path for c in full.children]
        for node in _all_folder_nodes(limited):
            assert (
                node.documents,
                node.summary_pending,
                node.updated_at,
                node.summary,
            ) == full_map[node.path]
            # Existence rule: a node is present ⟺ a catalogue prefix.
            assert node.path in full_map
        # Every truncated node is exactly the truncation contract.
        for node in _all_folder_nodes(limited):
            if node.truncated:
                assert node.children == []
            else:
                assert node.model_dump().get("truncated") is None
 def test_truncated_nodes_keep_the_image_affordance_at_their_level() -> None:
    """Phase 122's affordance is row-driven, not depth-driven: a DIRECT
    image file of the source root still carries the thumbnail keys at
    ``depth=1``; the truncated folder below simply does not list its
    own rows (they arrive with the drill)."""
    rows = [
        ("S", "pic.png", "pic", 1, T0, C0),
        ("S", "d/x.png", "x", 1, T0, C0),
    ]
    images = {
        ("S", "pic.png"): (DOC_ID, "A red square."),
        ("S", "d/x.png"): (DOC_ID, "A blue circle."),
    }
    (source,) = build_kb_tree(["S"], rows, {}, images, depth=1)
    (pic,) = _file_nodes(source)
    assert pic.is_image is True
    assert pic.model_dump()["image_url"] == f"/api/documents/{DOC_ID}/image"
    assert pic.model_dump()["summary"] == "A red square."
    (d,) = _folder_nodes(source)
    assert d.truncated is True
    assert d.children == []
 # ---------------------------------------------------------------------------
 # _kb_totals (phase 167, task 02) — the stat-card totals the endpoint ships
 # beside any depth of tree (no extra query, empty-KB honest).
 # ---------------------------------------------------------------------------
 def test_kb_totals_math_over_the_catalogue_rows() -> None:
    """documents = the row count, chunks = the sum, last_indexed = the
    MAX indexed stamp (lexicographic over one ISO shape = chronological
    — the same rule as the D9 max)."""
    totals = _kb_totals(DEPTH_ROWS)
    assert totals.documents == 4
    assert totals.chunks == 10  # 1 + 2 + 3 + 4
    assert totals.last_indexed == T2  # the newest indexed_at
 def test_kb_totals_empty_kb_is_zero_and_none() -> None:
    """The empty KB: 0 documents, 0 chunks, ``last_indexed is None`` —
    the stat cards render the designed empty state, never a crash."""
    totals = _kb_totals([])
    assert totals == KbTreeTotals(documents=0, chunks=0, last_indexed=None)
    assert totals.model_dump() == {
        "documents": 0,
        "chunks": 0,
        "last_indexed": None,
    }
 def test_kb_tree_totals_default_and_floor_validation() -> None:
    """``KbTree.totals`` is an additive top-level key with an honest
    empty default (an older call shape still serializes), and the
    ge=0 bounds refuse negative totals loudly (the endpoint computes
    them — a negative is a bug, never data)."""
    assert KbTree(sources=[]).model_dump()["totals"] == {
        "documents": 0,
        "chunks": 0,
        "last_indexed": None,
    }
    with pytest.raises(ValidationError):
        KbTreeTotals(documents=-1, chunks=0)
    with pytest.raises(ValidationError):
        KbTreeTotals(documents=0, chunks=-1)
@@ -5,10 +5,13 @@ The RAG (Knowledge base) view lists the catalog the way the agent's
 top, then per level the subfolders (recursive count + the STORED
 description) and the level's files (the ``#docs-table`` — phase 106,
 task 08, D8, added the Created column BEFORE Indexed, making it the
-6-column contract). The view's catalog load is now
+6-column contract). The view's catalog load is ``loadTree()``: ONE
-``loadTree()``: ONE fetch of ``GET /api/docs/tree`` (the full recursive
+fetch of ``GET /api/docs/tree?depth=1`` (sources + one level + the
-tree in a single payload — task 02), then client-side drill navigation
+server's ``totals``) plus ONE ``GET /api/docs/tree/node`` fetch per
-(no per-level fetch, no URL change).
+not-yet-loaded level — phase 167 (task 04, LOCKED A3 — the TODO L5
 2 MB payload fix) SUPERSEDED the phase-97 "one full-tree fetch, drill
 purely client-side" contract; the drill UX (breadcrumb, no URL change)
 is unchanged.
 This module pins the source-level contract (the house pattern of
 ``tests/unit/test_sync_button.py`` — read the frontend files as text,
@@ -24,8 +27,14 @@ suite, task 06):
  ``#stat-cards`` and BEFORE the file table (now the 6-column
  Source | Path | Title | Chunks | Created | Indexed head — phase 106
  (task 08, D8) added the Created column BEFORE Indexed);
-* the exact ``/api/docs/tree`` fetch (and the flat ``/api/docs`` fetch
+* the exact ``/api/docs/tree?depth=1`` load fetch + the single
-  is gone from the view module);
+  ``/api/docs/tree/node`` drill fetch (the flat ``/api/docs`` fetch is
  gone from the view module), the level cache (declared once, cleared
  by every load, consulted by ``needsLevelFetch``, written only after a
  200), the §7.4 drill states (in-flight ``aria-busy``/``aria-disabled``
  + one ``role="status"`` row, ``role="alert"`` failure with a real
  Retry ``<button>``, the node 404 → top-level reset, the race token
  extended to the drill), and the ``totals``-first stat cards;
 * the drill state (``current`` / ``kbTree``), the state transitions
  (source row click, folder row click, breadcrumb up, top reset), and
  the aria-current last segment;
@@ -239,18 +248,25 @@ def test_rag_view_ships_the_tree_skeleton_in_order() -> None:
        assert f">{column}</th>" in doc_head.group(1), f"#docs-table head must keep {column!r}"
-# ---------- the tree load: the ONE fetch + the race token ----------
+# ---------- the tree load: the depth-1 fetch, the node fetch, the token ----------
 def test_sources_js_fetches_the_tree_endpoint_only() -> None:
-    """The catalog load is `loadTree()`: ONE fetch of the tree endpoint.
+    """Phase 167 (task 04, A3 — supersedes the phase-97 "ONE full-tree
-    The flat `GET /api/docs` fetch is GONE from the view module (the
+    fetch, zero per-level fetches" pin): the catalog load is ONE
-    endpoint itself stays — the API surface is untouched, task 02)."""
+    `fetch("/api/docs/tree?depth=1")` and the drill has exactly ONE
    node-fetch call site (`fetch("/api/docs/tree/node?…`, reached only
    through fetchLevel). The flat `GET /api/docs` fetch stays GONE from
    the view module (the endpoint itself stays — task 02)."""
    js = _js()
-    assert 'fetch("/api/docs/tree")' in js, "the load must fetch /api/docs/tree"
+    code = _code(js)
-    assert js.count('fetch("/api/docs/tree")') == 1, (
+    assert code.count('fetch("/api/docs/tree?depth=1")') == 1, (
-        "the tree is the view's SINGLE catalog fetch (zero per-level fetches)"
+        "the load is ONE depth=1 request (the 2 MB first-payload fix)"
    )
    assert code.count('fetch("/api/docs/tree/node?') == 1, (
        "exactly ONE per-level fetch call site — every drill routes through it"
    )
    assert 'fetch("/api/docs/tree")' not in js, "the depth-less load URL is gone"
    assert 'fetch("/api/docs")' not in js, "the flat /api/docs fetch must be gone"
@@ -301,8 +317,9 @@ def test_source_row_click_drills_into_the_source_root() -> None:
    body = js[js.find("function makeSourceRow(") : js.find("function makeFolderRow(")]
    assert 'link.className = "folder-link"' in body, "the source row uses the row drill link"
    assert "link.textContent = s.name" in body
-    assert "goTo({ source: s.name, folder: \"\" })" in body, (
+    assert "goTo({ source: s.name, folder: \"\" }, link)" in body, (
-        "a source row drills to the source ROOT (folder \"\")"
+        "a source row drills to the source ROOT (folder \"\") — the link "
        "rides the phase-167 in-flight state"
    )
    assert "countTd.textContent = String(s.documents)" in body, "the recursive count"
    assert 'makeDescCell(s, s.name, "", s.name)' in body, (
@@ -326,8 +343,9 @@ def test_folder_row_click_drills_into_the_folder() -> None:
        "the label is the last segment (the breadcrumb language)"
    )
    assert "link.title = current.source + \"/\" + f.path" in body, "full path on hover"
-    assert "goTo({ source: current.source, folder: f.path })" in body, (
+    assert "goTo({ source: current.source, folder: f.path }, link)" in body, (
-        "a folder row drills to that folder"
+        "a folder row drills to that folder — the link rides the "
        "phase-167 in-flight state"
    )
    assert "countTd.textContent = String(f.documents)" in body
    assert (
@@ -370,19 +388,24 @@ def test_breadcrumb_links_go_up_and_the_top_reset() -> None:
 def test_drill_navigation_pushes_state_only_entries() -> None:
    """Phase 99 (task 02, D2): every user-initiated drill is a history
-    entry — ``goTo`` is the push variant of ``applyTarget``: when the
+    entry — ``applyTarget(target, push)``: when the target DIFFERS
-    target DIFFERS from the current level (a re-click of the current
+    from the current level (a re-click of the current row/segment
-    row/segment pushes NO duplicate entry), it FIRST records a
+    pushes NO duplicate entry), it FIRST records a STATE-ONLY entry —
-    STATE-ONLY entry — ``history.pushState({ view: "rag", kb: target },
+    ``history.pushState({ view: "rag", kb: target }, "")`` — the second
-    "")`` — the second arg is ``""`` and there is no third (the URL
+    arg is ``""`` and there is no third (the URL stays the shell's
-    stays the shell's pathname: no new route, the phase-76 deep-link
+    pathname: no new route, the phase-76 deep-link surface is
-    surface is untouched) — then sets ``current`` + re-renders. The
+    untouched) — then sets ``current`` + re-renders. The module never
-    drill still never fetches (the tree is in memory) and the module
+    ``replaceState``s. Phase 167 (task 04, A3 — supersedes the "the
-    never ``replaceState``s."""
+    drill never fetches" pin): ``goTo`` hands the target to
    ``enterFolder`` with ``push=true``, so the push lands only when the
    level is IN HAND (a failed drill records nothing) — ``applyTarget``
    and ``goTo`` themselves stay fetch-free; the drill's only fetch
    lives in ``fetchLevel`` (pinned below)."""
    js = _js()
-    start = js.find("function applyTarget(")
+    assert js.find("function applyTarget(") != -1, (
-    assert start != -1, "applyTarget must exist (the two-step drill)"
+        "applyTarget must exist (the two-step drill)"
-    body = js[start : js.find("function makeSourceRow(")]
+    )
    body = _fn(js, "applyTarget")
    gate_i = body.find(
        "if (push && (target.source !== current.source || target.folder !== current.folder)) {"
    )
@@ -394,14 +417,191 @@ def test_drill_navigation_pushes_state_only_entries() -> None:
    )
    # STATE-ONLY: exactly ONE pushState call site — the "" second arg,
    # no third (no URL change).
-    assert _code(body).count("pushState") == 1, (
+    assert _code(js).count("pushState") == 1, (
        "ONE pushState call site — the state-only drill entry"
    )
    assert "replaceState" not in _code(js), "no replaceState anywhere in the module"
    # goTo is the push variant — every drill call site keeps calling it.
-    go = body[body.find("function goTo(") :]
+    go = _fn(js, "goTo")
-    assert "applyTarget(target, true)" in go, "goTo pushes (the user-initiated drill)"
+    assert "enterFolder(target, link, true)" in go, (
-    assert "fetch(" not in body, "the drill never fetches (client-side only)"
+        "goTo pushes (the user-initiated drill, phase 167 via enterFolder)"
    )
    assert "fetch(" not in _code(body), "applyTarget stays a pure state transition"
    assert "fetch(" not in _code(go), "goTo delegates the fetch to enterFolder only"
 # ---------- phase 167 (task 04): the lazy drill (cache, states, reset) ----------
 def test_level_cache_declared_cleared_and_consulted() -> None:
    """The level cache (phase 167, D4): ONE module-scoped Map keyed
    `source + "/" + folder`; EVERY successful load clears it (a refresh
    invalidates every drilled level — the sync/upload/re-show call
    sites keep working unchanged) BEFORE the never-stale reset (the
    reset re-derives known-ness from the NEW tree); `needsLevelFetch`
    consults it (the second visit to a folder is free — zero network);
    the write sits behind the ok gate (only a 200 caches, never the
    404/failure arms) and follows the merge (children ride the node
    before the level is marked loaded)."""
    js = _js()
    assert js.count("const levelCache = new Map();") == 1, (
        "ONE module-scoped level cache"
    )
    load = _fn(js, "loadTree")
    clear_i = load.find("levelCache.clear();")
    reset_i = load.find("resetVanishedLocation();")
    assert 0 < clear_i < reset_i, "clear the cache BEFORE the reset (new tree)"
    needs = _fn(js, "needsLevelFetch")
    assert "node.truncated === true" in needs, "only a TRUNCATED node needs a fetch"
    assert "levelCache.has(" in needs, "a cached level never refetches"
    fetch = _fn(js, "fetchLevel")
    merge_i = fetch.find("Object.assign(targetNode, merged);")
    set_i = fetch.find('levelCache.set(target.source + "/" + target.folder, true);')
    ok_gate = fetch.find("if (response === null || !response.ok) {")
    vanish_i = fetch.find("if (vanished) {")
    assert -1 < vanish_i < ok_gate < merge_i < set_i, (
        "404 arm → ok gate → merge → cache: only a 200 caches, after the merge"
    )
 def test_drill_race_token_extends_to_the_node_fetch() -> None:
    """Phase 79 carried over (task 04): fetchLevel captures the
    loadSeq token (plus a navSeq bumped by every applyTarget) and a
    response may only paint while NEITHER a newer load NOR a newer
    navigation happened — pinned twice (after the fetch, after the
    json read). A new loadTree invalidates the in-flight drill the
    moment it starts, and rehydrates the user's own truncated level
    through the node route BEFORE it paints (a refresh never shows a
    half-empty level)."""
    js = _js()
    fetch = _fn(js, "fetchLevel")
    assert "const my = loadSeq;" in fetch, "the phase-79 token extends to the drill"
    assert "const nav = navSeq;" in fetch, "a navigation invalidates the drill too"
    assert fetch.count("if (my !== loadSeq || nav !== navSeq) return false;") == 2, (
        "a stale drill response never paints (checked after the fetch and the json)"
    )
    load = _fn(js, "loadTree")
    assert load.find("drillPending = null;") < load.find("await fetch"), (
        "a new load invalidates the in-flight drill immediately"
    )
    assert "if (needsLevelFetch(currentLevelNode(), current.source)) {" in load, (
        "the user's own truncated level is REHYDRATED through the node route"
    )
    assert "let navSeq = 0;" in js
    # …and the ADOPTED navigation (popstate) rehydrates the same way:
    # a forward/back entry can point at a level the client has not
    # fetched since the last load (ensureCurrentLevel, after the adopt).
    pop = js[js.find('window.addEventListener("popstate"') :]
    pop = pop[: pop.find("});")]
    adopt_i = pop.find("applyTarget(kb, false)")
    ensure_i = pop.find("ensureCurrentLevel();")
    assert -1 < adopt_i < ensure_i, "adopt the entry, THEN rehydrate its level"
    ensure = _fn(js, "ensureCurrentLevel")
    assert "if (drillPending) return;" in ensure
    assert "if (needsLevelFetch(currentLevelNode(), current.source)) {" in ensure
 def test_drill_pending_guards_and_aria_inflight_state() -> None:
    """§7.4 in-flight: enterFolder early-returns while a drill is
    pending (a double click cannot start a second fetch), sets the
    drillPending state BEFORE awaiting fetchLevel, and applies the
    target only once the level is in hand. fetchLevel marks the
    clicked link aria-busy + aria-disabled BEFORE the fetch and clears
    both on every settle path (the control never stays disabled — the
    failure path re-enables)."""
    js = _js()
    enter = _fn(js, "enterFolder")
    guard_i = enter.find("if (drillPending) return;")
    pending_i = enter.find(
        "drillPending = { source: target.source, folder: target.folder };"
    )
    fetch_i = enter.find("await fetchLevel(target, node, link)")
    apply_i = enter.find("applyTarget(target, push);")
    assert -1 < guard_i < pending_i < fetch_i < apply_i, (
        "guard → in-flight state → fetch → apply (only when the level is in hand)"
    )
    fetch = _fn(js, "fetchLevel")
    busy_i = fetch.find('link.setAttribute("aria-busy", "true")')
    disabled_i = fetch.find('link.setAttribute("aria-disabled", "true")')
    call_i = fetch.find("response = await fetch(")
    clear_busy = fetch.find('link.removeAttribute("aria-busy")')
    clear_disabled = fetch.find('link.removeAttribute("aria-disabled")')
    assert -1 < busy_i < call_i and -1 < disabled_i < call_i, (
        "the busy state rides the row link BEFORE the fetch"
    )
    assert call_i < clear_busy and call_i < clear_disabled, (
        "every settle path re-enables the control (§7.4)"
    )
 def test_drill_inflight_status_and_failure_alert_rows() -> None:
    """The ONE in-flight row of the folders panel (phase 167, D4): the
    loading row is a role="status" "Loading <folder>…" line while the
    level the user is LEAVING stays rendered; the failure row is a
    role="alert" "Couldn't load <folder> — " line with a REAL
    <button type="button">Retry</button> whose click re-runs the SAME
    enterFolder fetch. Neither builder calls renderLevel (the current
    level never blanks); both clear the previous drill row first (ONE
    row at a time) and textContent only (the house rule)."""
    js = _js()
    loading = _fn(js, "showDrillLoading")
    assert "clearDrillRow();" in loading
    assert 'note.setAttribute("role", "status");' in loading
    assert 'note.textContent = "Loading " + name + "…"' in loading
    assert 'row.className = "kb-level-loading"' in loading
    assert "foldersTbody.appendChild(row)" in loading, "the row lives in the folders panel"
    assert "renderLevel" not in _code(loading), "the leaving level stays rendered"
    fail = _fn(js, "showDrillFailure")
    assert "clearDrillRow();" in fail
    assert 'note.setAttribute("role", "alert");' in fail
    assert '"Couldn\'t load " + target.folder.split("/").pop() + " — "' in fail
    assert 'retry.type = "button";' in fail, "Retry is a REAL button"
    assert 'retry.textContent = "Retry";' in fail
    assert (
        "retry.addEventListener(\"click\", () => enterFolder(target, link, true));" in fail
    ), "Retry re-runs the SAME drill fetch"
    assert "renderLevel" not in _code(fail), "the failure keeps the current level rendered"
    clear = _fn(js, "clearDrillRow")
    assert "drillRowEl.remove();" in clear and "drillRowEl = null;" in clear
 def test_vanished_node_resets_to_the_top_level() -> None:
    """The node route's 404 IS the "the folder vanished" signal (task
    04, D4): the drill resets `current` to the TOP level and re-renders
    (no stale breadcrumb), caches NOTHING for the failed level, and
    never applies the vanished target (enterFolder returns on false).
    A network/non-OK failure takes the alert arm instead — the current
    level stays rendered (pinned above)."""
    js = _js()
    fetch = _fn(js, "fetchLevel")
    assert "response.status === 404" in fetch, "the 404 is detected"
    vanish = fetch[fetch.find("if (vanished) {") : fetch.find(
        "if (response === null || !response.ok) {"
    )]
    assert "current = { source: null, folder: null };" in vanish, (
        "the 404 resets to the top level (the phase-97 never-stale contract)"
    )
    assert "renderLevel();" in vanish, "the reset renders — no stale breadcrumb"
    assert "return false;" in vanish, "the vanished target is never applied"
    assert "levelCache" not in vanish, "the failing level is NOT cached"
    enter = _fn(js, "enterFolder")
    assert "if (!loaded) return;" in enter, "enterFolder applies only on a real merge"
 def test_module_docstring_documents_the_lazy_drill() -> None:
    """The house per-phase module-note convention: the Phase 167
    (task 04) section records the depth-1 load, the per-level node
    fetch + cache, the §7.4 drill states, the extended race token, the
    404 reset and the totals-first stat cards — citing TODO L5."""
    js = _js()
    header = js[: js.find("import { fetchIsAdmin }")]
    assert "Phase 167 (task 04, D4/A3" in header
    assert "TODO L5" in header, "the 2 MB payload defect is documented"
    assert "LAZY DRILL" in header
    assert "/api/docs/tree?depth=1" in header
    assert "levelCache" in header
    assert "loadSeq token EXTENDS" in header, "the race-token narrative stays accurate"
    assert "folderKnownIn" in header, "the existence-rule change is documented"
 # ---------- the level rendering ----------
@@ -507,12 +707,39 @@ def test_render_clears_both_row_containers_before_filling() -> None:
        assert clear_docs < body.find(append), f"{append} must come after the clears"
-# ---------- the stat cards: the whole-tree walk ----------
+# ---------- the stat cards: server totals, walk as the fallback ----------
 def test_stat_cards_prefer_the_server_totals() -> None:
    """Phase 167 (task 04, D2/D4): a depth-limited tree CANNOT be
    summed client-side, so treeStats() reads the server's `totals`
    (documents / chunks / last_indexed) FIRST; the phase-97 recursive
    walk stays only as the defensive no-`totals` fallback (a tree
    served by a pre-phase process — the no-crash path). The stat-card
    rendering (String(st.docs)/String(st.totalChunks)/fmtDate(st.last)
    with the "–" null idiom) is UNCHANGED."""
    js = _js()
    body = _fn(js, "treeStats")
    totals_i = body.find("const totals = kbTree.totals;")
    guard_i = body.find('if (totals && typeof totals === "object") {')
    docs_i = body.find("docs: totals.documents,")
    chunks_i = body.find("totalChunks: totals.chunks,")
    last_i = body.find('last: totals.last_indexed || "",')
    walk_i = body.find("const walk = (node) => {")
    assert -1 < totals_i < guard_i < docs_i < chunks_i < last_i < walk_i, (
        "totals first, the recursive walk ONLY as the no-totals fallback"
    )
    render = js[js.find("function renderLevel(") : js.find("function renderEmpty(")]
    assert "const st = treeStats();" in render
    assert 'st.last ? fmtDate(st.last) : "–"' in render, (
        "the exact fmtDate formatting call is unchanged at every level"
    )
 def test_stat_cards_walk_the_whole_tree() -> None:
-    """The KB-wide stat cards walk the WHOLE tree (every file node at
+    """The no-`totals` fallback walk (phase 97, kept defensive since
-    any depth): document count, chunks sum, max indexed_at (fmtDate
+    phase 167): walk the WHOLE held tree (every file node at any
    depth): document count, chunks sum, max indexed_at (fmtDate
    reuse) — the values identical to the former flat walk. A
    0-document registered source contributes nothing (and still lists
    its row — the ls invariant)."""
@@ -577,21 +804,30 @@ def test_module_docstring_documents_the_semantic_change() -> None:
 def test_reset_to_top_when_the_location_vanished() -> None:
    """PLAN §7.4: after a re-fetch, if the current source is no longer
-    in the tree (unregistered/pruned) OR the current folder no longer
+    in the tree (unregistered/pruned) OR the current folder is no
-    exists under it, `current` RESETS to the top level BEFORE rendering
+    longer KNOWN under it, `current` RESETS to the top level BEFORE
-    — no stale breadcrumb, no stale block. The folder existence check
+    rendering — no stale breadcrumb, no stale block. Phase 167 (task
-    mirrors the phase-94 rule: some indexed path starts with
+    04, D4): the phase-94 full-tree path-prefix walk becomes
-    `folder + "/"` (the source root — \"\" — always exists)."""
+    folderKnownIn — a folder is known iff its WHOLE ancestor chain
    exists in the levels the client HOLDS (the depth-1 tree + the
    drilled-and-merged levels; a broken chain — gone OR simply not
    loaded that deep — resets honestly; the node route's 404 is the
    drill path's own vanished-signal, pinned separately)."""
    js = _js()
-    body = js[js.find("function resetVanishedLocation()") : js.find("function folderExistsIn(")]
+    body = _fn(js, "resetVanishedLocation")
    assert "if (current.source === null) return;" in body, "the top level never vanishes"
    assert "kbTree.sources.find((s) => s.name === current.source)" in body
    assert body.count("current = { source: null, folder: null };") == 2, (
        "both vanishing arms reset to the top level"
    )
-    exists = js[js.find("function folderExistsIn(") : js.find("function currentLevelNode(")]
+    assert "folderKnownIn(src, current.folder)" in body, (
-    assert 'if (folderPath === "") return true;' in exists, "the source root always exists"
+        "the reset consults KNOWN-ness (the held levels), never a full-tree walk"
-    assert "p.startsWith(folderPath + \"/\")" in exists, "the phase-94 existence rule"
+    )
    known = _fn(js, "folderKnownIn")
    assert 'if (folderPath === "") return true;' in known, "the source root always exists"
    assert "if (!next) return false;" in known, "a broken ancestor chain is not known"
    assert "collect" not in _code(known), "the recursive path-collection walk is gone"
    assert "folderExistsIn" not in js, "the old full-tree existence walker is gone"
    # The reset runs on the success path of the load, before the render
    # (pinned in the load-order test — here: it is defined and called).
    assert "resetVanishedLocation();" in js[js.find("async function loadTree()"):]
@@ -736,6 +972,7 @@ def test_styles_carry_the_tree_classes_with_no_new_hue() -> None:
        ".kb-crumb-current",
        ".kb-crumb-sep",
        ".kb-level",
        ".kb-level-loading",
        ".kb-folders-table",
        ".folder-link",
    ):
@@ -0,0 +1,702 @@
 """Unit tests: ``app.core.response_budget`` — the byte-budget log line
 (phase 167, task 05, D5, LOCKED A5).
 Pins the contract of the line, the level cut, and the pass-through rules
 on a tiny ASGI app (the raw-ASGI interleaving harness of
 ``test_compression.py`` where ordering matters, ``TestClient`` where the
 composed stack matters):
 * the EXACT shape ``api method=… path=… status=… bytes=… enc=…
  duration_ms=…`` for a large 200 at INFO and a small 200 at DEBUG;
 * a 304 logs ``bytes=0`` (the revalidation win stays visible);
 * a non-``/api/`` path and a non-HTTP scope log NOTHING;
 * a stream logs ONE line with ``stream=1``, ``bytes`` = the sum of the
  chunks, emitted when the stream COMPLETES (never mid-flight) — and the
  frames are never buffered;
 * ``enc`` reads the WIRE encoding (``gzip`` through the real compression
  layer, ``identity`` for a client that does not accept it) while
  ``bytes`` stays the size the ROUTE produced;
 * an app exception is logged (finally-style, bytes as counted) and
  re-raised UNCHANGED;
 * the config knob: ``0`` logs every ``/api/*`` response at INFO, a
  negative value refuses startup naming the field, and the middleware sits
  INSIDE ``SelectiveGZipMiddleware`` so ``bytes=`` is the route's size.
 """
 from __future__ import annotations
 import asyncio
 import gzip
 import json
 import logging
 import re
 from typing import Any
 import pytest
 from fastapi.testclient import TestClient
 from pydantic import ValidationError
 from starlette.applications import Starlette
 from starlette.responses import Response
 from starlette.routing import Route
 from starlette.types import Message, Receive, Scope, Send
 import app.core.response_budget as budget_mod
 import app.main as main_mod
 from app.config import Settings
 from app.core.compression import SelectiveGZipMiddleware
 from app.core.response_budget import (
    API_PATH_PREFIX,
    UNKNOWN_STATUS,
    ResponseBudgetMiddleware,
 )
 #: One line, fully anchored — the shape README documents and the
 #: ``grep ' :: api '`` workflow parses.
 LINE_RE = re.compile(
    r"^api method=(?P<method>[A-Z]+) path=(?P<path>\S+) status=(?P<status>\d+) "
    r"bytes=(?P<bytes>\d+) enc=(?P<enc>\S+) duration_ms=(?P<ms>\d+)$"
 )
 #: ~28 KB of tree-shaped JSON.
 _BODY = json.dumps(
    {
        "sources": [
            {
                "name": "src",
                "documents": i,
                "children": [{"path": f"d/{i}/{n}.md"} for n in range(20)],
            }
            for i in range(200)
        ]
    }
 ).encode()
 _SMALL_BODY = b'{"ok":true}'
 #: Four SSE-class frames, each comfortably above the gzip floor.
 _FRAMES = [f"data: {'x' * 500}\n\n".encode() for _ in range(4)]
 _LOGGER_NAME = "app.response_budget"
 def _settings(**kwargs: Any) -> Settings:
    """Build Settings without reading a .env file (the test_config pattern)."""
    kwargs.setdefault("_env_file", None)
    return Settings(**kwargs)  # pyright: ignore[reportCallIssue]  (kwarg exists at runtime)
 def _api_lines(caplog: pytest.LogCaptureFixture) -> list[tuple[int, str]]:
    """``(levelno, message)`` for this module's logger only, in order."""
    return [
        (record.levelno, record.getMessage())
        for record in caplog.records
        if record.name == _LOGGER_NAME
    ]
 class _Counter(logging.Handler):
    """Counts emitted lines live, so an app can observe "has the line been
    written yet?" from INSIDE the response — the ordering proof."""
    def __init__(self) -> None:
        super().__init__()
        self.count = 0
    def emit(self, record: logging.LogRecord) -> None:
        self.count += 1
 def _http_scope(path: str, method: str = "GET") -> Scope:
    return {"type": "http", "method": method, "path": path, "headers": []}
 async def _receive() -> Message:
    return {"type": "http.disconnect"}
 async def _send(message: Message) -> None:
    return None
 def _raw_json_app(body: bytes, *, status: int = 200, media_type: str = "application/json") -> Any:
    """The smallest ASGI app that answers one buffered body — the raw-ASGI
    level is where the WIRE bytes (post-compression) are observable."""
    async def app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": status,
                "headers": [(b"content-type", media_type.encode())],
            }
        )
        await send({"type": "http.response.body", "body": body, "more_body": False})
    return app
 def _raw_harness(middleware_app: Any, *, path: str, accept: str) -> list[Message]:
    """Drive a middleware stack at the raw ASGI level; return the exact
    messages that reached the client's ``send``."""
    sent: list[Message] = []
    async def recording_send(message: Message) -> None:
        sent.append(message)
    asyncio.run(
        middleware_app(
            {
                "type": "http",
                "method": "GET",
                "path": path,
                "headers": [(b"accept-encoding", accept.encode())],
            },
            _receive,
            recording_send,
        )
    )
    return sent
 def _inner_app(
    *,
    body: bytes = _BODY,
    status: int = 200,
    path: str = "/api/docs/tree",
    media_type: str = "application/json",
 ) -> Starlette:
    async def respond(request: Any) -> Response:
        return Response(body, status_code=status, media_type=media_type)
    async def boom(request: Any) -> Response:
        raise RuntimeError("kaboom")
    return Starlette(
        routes=[
            Route(path, respond, methods=["GET", "POST"]),
            Route("/api/boom", boom, methods=["GET"]),
            Route("/sources.html", respond, methods=["GET"]),
        ]
    )
 def _client(app: Any, *, threshold: int = 1024) -> TestClient:
    return TestClient(ResponseBudgetMiddleware(app, threshold=threshold))
 # ---------------------------------------------------------------------------
 # The line's exact shape, and the level the budget decides
 # ---------------------------------------------------------------------------
 def test_large_response_logs_one_info_line_in_the_exact_shape(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        client = _client(_inner_app(), threshold=1024)
        assert client.get("/api/docs/tree").status_code == 200
    lines = _api_lines(caplog)
    assert len(lines) == 1
    level, message = lines[0]
    assert level == logging.INFO
    match = LINE_RE.match(message)
    assert match is not None, message
    assert match.group("method") == "GET"
    assert match.group("path") == "/api/docs/tree"
    assert match.group("status") == "200"
    assert int(match.group("bytes")) == len(_BODY)
    assert match.group("enc") == "identity"
    assert int(match.group("ms")) >= 0
 def test_small_response_logs_the_same_line_at_debug(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        client = _client(_inner_app(body=_SMALL_BODY), threshold=1024)
        assert client.get("/api/docs/tree").status_code == 200
    lines = _api_lines(caplog)
    assert len(lines) == 1
    level, message = lines[0]
    assert level == logging.DEBUG
    assert f"bytes={len(_SMALL_BODY)}" in message
 def test_budget_is_the_inclusive_cut(caplog: pytest.LogCaptureFixture) -> None:
    """``bytes >= budget`` is INFO, one byte less is DEBUG — the boundary the
    README and ``.env.example`` state."""
    async def exactly(request: Any) -> Response:
        return Response(b"x" * 100)
    async def one_less(request: Any) -> Response:
        return Response(b"x" * 99)
    app = Starlette(
        routes=[
            Route("/api/a", exactly, methods=["GET"]),
            Route("/api/b", one_less, methods=["GET"]),
        ]
    )
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        client = _client(app, threshold=100)
        client.get("/api/a")
        client.get("/api/b")
    assert [level for level, _ in _api_lines(caplog)] == [logging.INFO, logging.DEBUG]
 def test_threshold_defaults_to_the_settings_value(
    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
 ) -> None:
    """No ``threshold`` argument — the way ``create_app`` wires it — so the
    live ``log_large_response_bytes`` decides per response."""
    monkeypatch.setattr(
        budget_mod, "get_settings", lambda: _settings(log_large_response_bytes=5)
    )
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        TestClient(ResponseBudgetMiddleware(_inner_app(body=_SMALL_BODY))).get(
            "/api/docs/tree"
        )
    assert [level for level, _ in _api_lines(caplog)] == [logging.INFO]
 def test_zero_budget_logs_every_api_response_at_info(
    monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
 ) -> None:
    """``BOR_LOG_LARGE_RESPONSE_BYTES=0`` is the loud mode: even an 11-byte
    body is INFO (the operator's one-off investigation setting)."""
    monkeypatch.setattr(
        budget_mod, "get_settings", lambda: _settings(log_large_response_bytes=0)
    )
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        TestClient(ResponseBudgetMiddleware(_inner_app(body=_SMALL_BODY))).get(
            "/api/docs/tree"
        )
    # 11 bytes, budget 0 -> INFO (the same body is DEBUG at any positive
    # budget: see test_small_response_logs_the_same_line_at_debug).
    assert [level for level, _ in _api_lines(caplog)] == [logging.INFO]
 # ---------------------------------------------------------------------------
 # Statuses: a 304 logs 0 bytes, an error is counted like any other response
 # ---------------------------------------------------------------------------
 def test_304_logs_zero_bytes(caplog: pytest.LogCaptureFixture) -> None:
    """The revalidation win stays in the log — a bodiless 304 is a line with
    ``bytes=0``, not a missing line."""
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        client = _client(_inner_app(body=b"", status=304), threshold=1024)
        assert client.get("/api/docs/tree").status_code == 304
    lines = _api_lines(caplog)
    assert len(lines) == 1
    match = LINE_RE.match(lines[0][1])
    assert match is not None
    assert match.group("status") == "304"
    assert match.group("bytes") == "0"
    assert lines[0][0] == logging.DEBUG
 def test_error_status_is_logged_like_any_other_response(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        client = _client(_inner_app(body=b'{"detail":"admin only"}', status=403))
        assert client.get("/api/docs/tree").status_code == 403
    match = LINE_RE.match(_api_lines(caplog)[0][1])
    assert match is not None and match.group("status") == "403"
 # ---------------------------------------------------------------------------
 # What is NOT measured
 # ---------------------------------------------------------------------------
 def test_non_api_path_logs_nothing(caplog: pytest.LogCaptureFixture) -> None:
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        # budget 0: ANY /api line would show, so silence is the whole pin.
        client = _client(_inner_app(), threshold=0)
        assert client.get("/sources.html").status_code == 200
    assert _api_lines(caplog) == []
 def test_non_http_scopes_pass_through_silently(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """``lifespan`` / ``websocket`` scopes reach the app untouched and write
    no line — there is no response to measure."""
    seen: list[str] = []
    async def app(scope: Scope, receive: Receive, send: Send) -> None:
        seen.append(scope["type"])
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        for scope_type in ("lifespan", "websocket"):
            asyncio.run(
                ResponseBudgetMiddleware(app, threshold=0)(
                    {"type": scope_type, "path": "/api/x"}, _receive, _send
                )
            )
    assert seen == ["lifespan", "websocket"]
    assert _api_lines(caplog) == []
 # ---------------------------------------------------------------------------
 # The stream: one line, stream=1, counted total, emitted at completion
 # ---------------------------------------------------------------------------
 def test_stream_logs_once_at_completion_with_stream_flag_and_total_bytes(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """The SSE contract of the line: nothing while frames are in flight, one
    ``stream=1`` line when the final chunk completes, ``bytes`` = the sum of
    the frames (NOT one frame)."""
    counter = _Counter()
    logger = logging.getLogger(_LOGGER_NAME)
    logger.addHandler(counter)
    in_flight: list[int] = []
    async def sse_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/event-stream")],
            }
        )
        for frame in _FRAMES:
            await send(
                {"type": "http.response.body", "body": frame, "more_body": True}
            )
            in_flight.append(counter.count)  # after each frame is delivered
        await send({"type": "http.response.body", "body": b"", "more_body": False})
        in_flight.append(counter.count)
    try:
        with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
            asyncio.run(
                ResponseBudgetMiddleware(sse_app, threshold=1)(
                    _http_scope("/api/chat", "POST"), _receive, _send
                )
            )
    finally:
        logger.removeHandler(counter)
    # Every in-flight probe except the last saw zero lines; the completion
    # is what emitted.
    assert in_flight == [0] * len(_FRAMES) + [1]
    lines = _api_lines(caplog)
    assert len(lines) == 1
    message = lines[0][1]
    assert message.endswith(" stream=1")
    match = LINE_RE.match(message.removesuffix(" stream=1"))
    assert match is not None
    assert match.group("method") == "POST"
    assert match.group("path") == "/api/chat"
    assert int(match.group("bytes")) == sum(len(frame) for frame in _FRAMES)
 def test_stream_frames_are_never_buffered() -> None:
    """Zero-buffering proof (the interleaving idiom of
    ``test_compression.py``): every frame reaches the client's ``send``
    before the app produces the next one — a ``BaseHTTPMiddleware`` here
    would emit all ``app:*`` events first."""
    events: list[str] = []
    async def sse_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/event-stream")],
            }
        )
        for i, frame in enumerate(_FRAMES):
            events.append(f"app:frame-{i}")
            await send(
                {"type": "http.response.body", "body": frame, "more_body": True}
            )
        await send({"type": "http.response.body", "body": b"", "more_body": False})
    async def recording_send(message: Message) -> None:
        events.append(f"client:{message['type']}")
    asyncio.run(
        ResponseBudgetMiddleware(sse_app, threshold=0)(
            _http_scope("/api/chat", "POST"), _receive, recording_send
        )
    )
    expected = ["client:http.response.start"]
    for i in range(len(_FRAMES)):
        expected += [f"app:frame-{i}", "client:http.response.body"]
    expected += ["client:http.response.body"]
    assert events == expected
 # ---------------------------------------------------------------------------
 # enc= reads the wire, bytes= stays the route's size
 # ---------------------------------------------------------------------------
 def _stack(*, threshold: int = 1024) -> TestClient:
    """response-budget INSIDE the selective-gzip layer — the order
    ``create_app`` builds."""
    return TestClient(
        SelectiveGZipMiddleware(
            ResponseBudgetMiddleware(_inner_app(), threshold=threshold),
            minimum_size=860,
        )
    )
 def test_enc_gzip_is_reported_through_the_compression_layer(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """The two halves of the line, measured against the wire: the client
    receives a gzipped body, the log says ``enc=gzip`` and still counts the
    ROUTE's uncompressed bytes (the position inside the compressor is what
    buys that)."""
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        sent = _raw_harness(
            SelectiveGZipMiddleware(
                ResponseBudgetMiddleware(_raw_json_app(_BODY), threshold=1024),
                minimum_size=860,
            ),
            path="/api/docs/tree",
            accept="gzip",
        )
    wire = b"".join(m.get("body", b"") for m in sent if m["type"] == "http.response.body")
    assert gzip.decompress(wire) == _BODY  # the transfer really was gzip
    match = LINE_RE.match(_api_lines(caplog)[-1][1])
    assert match is not None
    assert match.group("enc") == "gzip"
    assert int(match.group("bytes")) == len(_BODY) > len(wire)
 def test_enc_identity_when_the_client_does_not_accept_gzip(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        response = _stack().get(
            "/api/docs/tree", headers={"Accept-Encoding": "identity"}
        )
    assert "content-encoding" not in response.headers
    match = LINE_RE.match(_api_lines(caplog)[-1][1])
    assert match is not None and match.group("enc") == "identity"
    assert int(match.group("bytes")) == len(_BODY)
 def test_enc_reports_a_pre_encoded_response(caplog: pytest.LogCaptureFixture) -> None:
    """A route that encodes its own body (the fronting-proxy case) is
    reported as such — the header on the wire is the answer."""
    async def encoded(request: Any) -> Response:
        return Response(
            gzip.compress(_BODY),
            media_type="application/json",
            headers={"Content-Encoding": "gzip"},
        )
    app = Starlette(routes=[Route("/api/encoded", encoded, methods=["GET"])])
    with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME):
        TestClient(ResponseBudgetMiddleware(app, threshold=1024)).get("/api/encoded")
    match = LINE_RE.match(_api_lines(caplog)[-1][1])
    assert match is not None and match.group("enc") == "gzip"
 # ---------------------------------------------------------------------------
 # The app raising: logged, and re-raised unchanged
 # ---------------------------------------------------------------------------
 def test_app_exception_logs_and_reraises_unchanged(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """An app that dies before it can answer still writes one line (nothing
    was counted, status 500 — what the outer error layer will answer) and the
    exception propagates UNCHANGED."""
    async def dying_app(scope: Scope, receive: Receive, send: Send) -> None:
        raise RuntimeError("kaboom")
    with (
        caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME),
        pytest.raises(RuntimeError, match="kaboom"),
    ):
        asyncio.run(
            ResponseBudgetMiddleware(dying_app, threshold=0)(
                _http_scope("/api/docs/tree"), _receive, _send
            )
        )
    lines = _api_lines(caplog)
    assert len(lines) == 1
    match = LINE_RE.match(lines[0][1])
    assert match is not None
    assert match.group("status") == str(UNKNOWN_STATUS)
    assert match.group("bytes") == "0"
 def test_a_500_produced_by_the_framework_is_counted_like_a_response(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """Through a real app the unhandled exception becomes a 500 WITH a body
    (Starlette's ``Internal Server Error`` text): the line then reports that
    status and those bytes — one line either way, never zero."""
    with (
        caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME),
        pytest.raises(RuntimeError, match="kaboom"),
    ):
        _client(_inner_app(), threshold=0).get("/api/boom")
    lines = _api_lines(caplog)
    assert len(lines) == 1
    match = LINE_RE.match(lines[0][1])
    assert match is not None
    assert match.group("status") == "500"
    assert int(match.group("bytes")) == len(b"Internal Server Error")
 def test_exception_mid_stream_logs_the_bytes_counted_so_far(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """A stream that dies after two frames still writes ONE line with the
    bytes it actually got out (finally-style) — and the exception is not
    swallowed."""
    async def dying_stream_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/event-stream")],
            }
        )
        await send({"type": "http.response.body", "body": _FRAMES[0], "more_body": True})
        await send({"type": "http.response.body", "body": _FRAMES[1], "more_body": True})
        raise RuntimeError("stream died")
    with (
        caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME),
        pytest.raises(RuntimeError, match="stream died"),
    ):
        asyncio.run(
            ResponseBudgetMiddleware(dying_stream_app, threshold=0)(
                _http_scope("/api/chat", "POST"), _receive, _send
            )
        )
    lines = _api_lines(caplog)
    assert len(lines) == 1
    message = lines[0][1]
    assert message.endswith(" stream=1")
    assert f"bytes={len(_FRAMES[0]) + len(_FRAMES[1])}" in message
 def test_a_cancelled_stream_still_logs_one_line_with_what_got_out(
    caplog: pytest.LogCaptureFixture,
 ) -> None:
    """The client-walked-away path (the live audit hit it: an SSE turn cut
    mid-answer): the app unwinds with ``CancelledError`` and no completing
    body, the ``finally`` writes ONE line with the bytes that actually got
    out, and the cancellation propagates (the middleware never swallows it)."""
    async def cancelled_stream_app(scope: Scope, receive: Receive, send: Send) -> None:
        await send(
            {
                "type": "http.response.start",
                "status": 200,
                "headers": [(b"content-type", b"text/event-stream")],
            }
        )
        await send({"type": "http.response.body", "body": _FRAMES[0], "more_body": True})
        raise asyncio.CancelledError  # what the server does on a client disconnect
    with (
        caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME),
        pytest.raises(asyncio.CancelledError),
    ):
        asyncio.run(
            ResponseBudgetMiddleware(cancelled_stream_app, threshold=0)(
                _http_scope("/api/chat", "POST"), _receive, _send
            )
        )
    lines = _api_lines(caplog)
    assert len(lines) == 1
    assert lines[0][1].endswith(" stream=1")
    assert f"bytes={len(_FRAMES[0])}" in lines[0][1]
 # ---------------------------------------------------------------------------
 # Config + wiring
 # ---------------------------------------------------------------------------
 def test_log_large_response_bytes_defaults_to_64_kb() -> None:
    assert _settings().log_large_response_bytes == 65536
 def test_log_large_response_bytes_env_override(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    monkeypatch.setenv("BOR_LOG_LARGE_RESPONSE_BYTES", "4096")
    assert _settings().log_large_response_bytes == 4096
    monkeypatch.setenv("BOR_LOG_LARGE_RESPONSE_BYTES", "0")  # loud mode is legal
    assert _settings().log_large_response_bytes == 0
 def test_negative_budget_refuses_startup_naming_the_field() -> None:
    with pytest.raises(ValidationError, match="log_large_response_bytes must be >= 0"):
        _settings(log_large_response_bytes=-1)
 def test_middleware_sits_inside_gzip_outside_caching(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    """The documented stack (README + ``app/main.py``): SecurityHeaders
    outermost → gzip → response-budget → caching → session. That position is
    what makes ``bytes=`` the ROUTE's size while ``enc=`` still names the
    wire encoding."""
    monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 860)
    names = [
        getattr(mw.cls, "__name__", "") for mw in main_mod.create_app().user_middleware
    ]
    assert names == [
        "SecurityHeadersMiddleware",
        "SelectiveGZipMiddleware",
        "ResponseBudgetMiddleware",
        "CachingMiddleware",
        "SessionMiddleware",
    ]
 def test_budget_is_wired_even_with_compression_off(
    monkeypatch: pytest.MonkeyPatch,
 ) -> None:
    """``BOR_COMPRESS_MIN_BYTES=0`` removes gzip only — the budget line is
    not optional (LOCKED A5: measure everything, reject nothing), so the
    measurement survives the compression kill switch."""
    monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 0)
    names = [
        getattr(mw.cls, "__name__", "") for mw in main_mod.create_app().user_middleware
    ]
    assert "SelectiveGZipMiddleware" not in names
    assert "ResponseBudgetMiddleware" in names
 def test_api_path_prefix_constant() -> None:
    assert API_PATH_PREFIX == "/api/"