phase: 167_api_payload_diet

All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.

tests/e2e/test_admin_auth.py

@@ -98,8 +98,9 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
 
 def _drill(page: Page, *names: str) -> None:
     """Drill one level at a time: each name is the EXACT text of the
-    source/folder link at the current level (client-side — no fetch,
-    no URL change)."""
+    source/folder link at the current level (no URL change; since phase
+    167 the first open of a level the depth-1 payload only NAMES costs
+    one /api/docs/tree/node fetch, a level already held costs none)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 

tests/e2e/test_document_dates.py

@@ -301,7 +301,11 @@ def _docs_by_path(app_url: str, cookies: dict[str, str]) -> dict[str, dict[str,
 
 
 def _tree(app_url: str, cookies: dict[str, str]) -> dict[str, Any]:
-    r = httpx.get(f"{app_url}/api/docs/tree", timeout=10, cookies=cookies)
+    # Phase 167 (task 02): the callers below use ``_find_file``, which
+    # recurses (every fixture file sits under backups/, legacy/ or
+    # future/), so the WHOLE tree is requested — the endpoint's default
+    # (depth=1) is the RAG view's first paint only.
+    r = httpx.get(f"{app_url}/api/docs/tree?depth=8", timeout=10, cookies=cookies)
     assert r.status_code == 200, r.text
     return r.json()
 

tests/e2e/test_extensionless_import.py

@@ -122,9 +122,10 @@ def _delete_source_rows() -> None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
-    no URL change): each name is the EXACT text of the source/folder
-    link at the current level."""
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
+    costs exactly one /api/docs/tree/node fetch, a held level costs
+    nothing, and no click ever changes the URL): each name is the EXACT
+    text of the source/folder link at the current level."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 

tests/e2e/test_folder_summary_voice.py

@@ -502,9 +502,11 @@ def _assert_change_terminal(body: dict[str, Any]) -> None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
-    change): each name is the EXACT text of the source/folder link at
-    the current level (the row builders' link text: the source name,
+    """Drill one level at a time (no URL change; phase 167: the first
+    open of a level the depth-1 payload only NAMES costs one
+    /api/docs/tree/node fetch, a level already held costs nothing):
+    each name is the EXACT text of the source/folder link at the
+    current level (the row builders' link text: the source name,
     or the folder's last path segment)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
@@ -609,7 +611,7 @@ def test_rag_table_renders_the_stored_voice(
 
     login(page, app_url)  # → /sources.html (the RAG view)
     # The top level: the one source row (the ls() equivalence — the
-    # tree's single fetch settled) and its stored source-root voice in
+    # tree's ONE depth-1 fetch settled) and its stored source-root voice in
     # the Description cell (the exact canned line, text + title).
     page.wait_for_selector("#folders-tbody tr", timeout=30_000)
     expect(page.locator("#folders-tbody tr")).to_have_count(1)

tests/e2e/test_git_source_dates.py

@@ -416,7 +416,10 @@ def test_api_created_dates_are_true_per_file(
     # (old folder = the 2020 date alone, recent folder = the 2024
     # date alone, source = the whole subtree's 2024 max — D9, now over
     # TRUE dates).
-    t = page.request.get(f"{app_url}/api/docs/tree")
+    # Phase 167 (task 02): ``_find_file`` recurses by design (the files
+    # sit under ``old/`` and ``recent/``), so ask for the WHOLE tree —
+    # the endpoint's default (depth=1) is the RAG view's first paint.
+    t = page.request.get(f"{app_url}/api/docs/tree?depth=8")
     assert t.status == 200, t.text
     source = _source_node(t.json())
     assert source["documents"] == 2

tests/e2e/test_import_documents.py

@@ -204,9 +204,11 @@ SOURCE_NAME = "docs"  # the fixtures dir's basename (the indexed-only source)
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
-    change): each name is the EXACT text of the source/folder link at
-    the current level (the test_kb_tree house pattern)."""
+    """Drill one level at a time (no URL change; phase 167: the first
+    open of a level the depth-1 payload only NAMES costs one
+    /api/docs/tree/node fetch, a level already held costs nothing):
+    each name is the EXACT text of the source/folder link at the
+    current level (the test_kb_tree house pattern)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 
@@ -218,7 +220,7 @@ def _go_top(page: Page) -> None:
 
 
 def _wait_top_level(page: Page) -> None:
-    """The tree's single fetch settled: the source row is rendered
+    """The ONE ``?depth=1`` catalog fetch has settled: the source row is rendered
     (the catalog-rendered signal — the top-level file table is always
     hidden, so it is no longer a usable one)."""
     page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")

tests/e2e/test_import_extensions_env.py

@@ -92,9 +92,10 @@ def _delete_source_rows() -> None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
-    no URL change): each name is the EXACT text of the source/folder
-    link at the current level."""
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
+    costs exactly one /api/docs/tree/node fetch, a held level costs
+    nothing, and no click ever changes the URL): each name is the EXACT
+    text of the source/folder link at the current level."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 

tests/e2e/test_kb_tree.py

@@ -9,6 +9,24 @@ the level's files — and the owner edits (or clears) any directory's
 description with the phase-57 inline affordance. Everything is pinned
 against the real app + the deterministic mock.
 
+PHASE 167 (task 06, LOCKED A3) — THE CATALOG LOAD, MIGRATED NOT DELETED:
+the view no longer downloads the whole tree. The boot is ONE
+``GET /api/docs/tree?depth=1`` (the sources plus one level plus the
+server's ``totals``), and the FIRST time the owner opens a folder the
+payload only NAMES, the client fetches that one level through
+``GET /api/docs/tree/node?source=…&path=…`` (the phase-97 "ONE fetch of
+/api/docs/tree, the drill is client-side" contract, superseded by the
+TODO L5 2 MB fix — see
+``tests/e2e/test_kb_tree_lazy_load.py`` for the new load-shape suite).
+EVERY claim this suite owns stays and is re-pointed at the level the
+owner is ON: the breadcrumb text, the back navigation, the table
+columns, the dates, the summary-editor round trip, the pending markers,
+the empty state and the anonymous gate. A level the client already
+holds (the top level, a source root, a folder drilled into once — and
+the breadcrumb always walks back through a level it drilled) still
+renders with ZERO requests, so the pre-phase navigation claims below
+remain literally true at those levels.
+
 Run in isolation (DB must be up: ``podman compose up -d db``):
 
     uv run pytest tests/e2e/test_kb_tree.py -v --no-cov
@@ -498,18 +516,21 @@ def _run_sync(app_url: str) -> dict[str, Any]:
 
 def _wait_top_level(page: Page) -> None:
     """The admin boot has rendered the top level: the two source rows
-    are in the folders table (the tree's single fetch settled) and the
-    top-level file table is empty (files are per-source — always hidden
-    at the top)."""
+    are in the folders table (the ONE ``?depth=1`` catalog fetch has
+    settled — phase 167) and the top-level file table is empty (files
+    are per-source — always hidden at the top)."""
     expect(page.locator("#folders-tbody tr")).to_have_count(2, timeout=30_000)
     expect(page.locator("#docs-tbody tr")).to_have_count(0)
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
-    change): each name is the EXACT text of the source/folder link at
-    the current level (the row builders' link text: the source name,
-    or the folder's last path segment)."""
+    """Drill one level at a time (no URL change — the navigation is a
+    re-render; phase 167: a level the depth-1 payload already holds
+    costs ZERO requests, the first open of one it only NAMES costs
+    exactly one ``/api/docs/tree/node`` fetch): each name is the EXACT
+    text of the source/folder link at the current level (the row
+    builders' link text: the source name, or the folder's last path
+    segment)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 
@@ -668,7 +689,10 @@ def test_drill_into_folder(
     expect(page.locator("#folders-wrap")).to_be_hidden()
 
     # The breadcrumb link on the source goes back UP to the source
-    # level (client-side — no fetch, no URL change).
+    # level — a re-render with no fetch and no URL change, because
+    # reaching ``two`` means the client holds ``alpha`` (phase 167:
+    # the breadcrumb only ever walks back through a level it drilled,
+    # so this stays the zero-request path it was pre-phase).
     page.click(f'#kb-crumb a.kb-crumb-link:text-is("{ALPHA}")')
     expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(ALPHA)
     expect(page.locator("#kb-crumb a.kb-crumb-link")).to_have_count(1)

tests/e2e/test_kb_tree_nav.py

@@ -446,12 +446,13 @@ def _wait_top_level(page: Page) -> None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, NO URL
-    change): each name is the EXACT text of the source/folder link at
-    the current level (the row builders' link text: the source name, or
-    the folder's last path segment). Phase 99 (D2): every drill click
-    is a STATE-ONLY history entry (``goTo`` → ``applyTarget(target,
-    true)``)."""
+    """Drill one level at a time (no URL change; phase 167: an UNCACHED
+    level costs exactly one /api/docs/tree/node fetch, a level already
+    held costs nothing): each name is the EXACT text of the
+    source/folder link at the current level (the row builders' link
+    text: the source name, or the folder's last path segment). Phase 99
+    (D2): every drill click is a STATE-ONLY history entry (``goTo`` →
+    ``applyTarget(target, true)``)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 

tests/e2e/test_navbar_refresh.py

@@ -110,9 +110,11 @@ TITLE_E = "Phase77 popstate E"
 DATA_VIEWS: tuple[tuple[str, str, str], ...] = (
     ("#nav-history", "/history.html", "/api/chats"),
     # Phase 97: the RAG view's list endpoint is the drill-down tree
-    # (GET /api/docs/tree — the flat /api/docs is the unchanged API
-    # surface, no longer the view's fetch).
-    ("#nav-sources", "/sources.html", "/api/docs/tree"),
+    # (the flat /api/docs is the unchanged API surface, no longer the
+    # view's fetch). Phase 167: the load carries depth=1 (the payload
+    # diet) — the per-level drill fetches /api/docs/tree/node, which
+    # these no-drill tests never trigger.
+    ("#nav-sources", "/sources.html", "/api/docs/tree?depth=1"),
     ("#nav-git-sources", "/git-sources.html", "/api/git-sources"),
     ("#nav-tuning", "/tuning.html", "/api/steering"),
 )

tests/e2e/test_quadlet_jinja_import.py

@@ -107,9 +107,10 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (phase 97 — client-side, no fetch,
-    no URL change): each name is the EXACT text of the source/folder
-    link at the current level."""
+    """Drill one level at a time (phase 97; phase 167: an UNCACHED level
+    costs exactly one /api/docs/tree/node fetch, a held level costs
+    nothing, and no click ever changes the URL): each name is the EXACT
+    text of the source/folder link at the current level."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
 

tests/e2e/test_sources_midstream_bug.py

@@ -119,9 +119,28 @@ def _run_in_thread(coro: Any) -> Any:
 
 
 def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
-    """Truncate the KB (and query log), then optionally re-import fixtures."""
+    """Truncate the KB (and query log), then optionally re-import fixtures.
+
+    The truncate covers EVERY table that shapes the prompt, not just the
+    document rows: this suite's story asserts the EXACT long-answer text,
+    and the mock echoes a stored KB overview into the answer verbatim
+    (`` (kb: <first bullet>)``, phase 31) plus a stored steering note
+    (phase 15). Those rows survive a documents-only truncate — another
+    suite in the shared E2E Postgres ran a real sync and left its
+    overview behind — so the answer would arrive with a suffix this
+    suite never seeded and the byte-exact pin would fail on the OTHER
+    suite's corpus (``tests/e2e/test_kb_tree.py`` is the usual holder).
+    Same list as that suite's ``_truncate_all`` (the house isolation
+    pattern); no assertion changes — the corpus under test is exactly
+    the one this fixture seeds.
+    """
     with SessionLocal() as db:
-        db.execute(text("TRUNCATE chunks, documents, query_log"))
+        db.execute(
+            text(
+                "TRUNCATE chunks, documents, query_log, steering_notes, "
+                "kb_overview, git_sources, folder_summaries"
+            )
+        )
         db.commit()
     if not seed:
         return None

tests/e2e/test_sync_summary_visibility.py

@@ -593,9 +593,11 @@ def _wait_top_level(page: Page) -> None:
 
 
 def _drill(page: Page, *names: str) -> None:
-    """Drill one level at a time (client-side — no fetch, no URL
-    change): each name is the EXACT text of the source/folder link at
-    the current level (the row builders' link text: the source name,
+    """Drill one level at a time (no URL change; phase 167: the first
+    open of a level the depth-1 payload only NAMES costs one
+    /api/docs/tree/node fetch, a level already held costs nothing):
+    each name is the EXACT text of the source/folder link at the
+    current level (the row builders' link text: the source name,
     or the folder's last path segment — the phase-97 pattern)."""
     for name in names:
         page.click(f'#folders-tbody a.folder-link:text-is("{name}")')