phase: 167_api_payload_diet
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 14s

All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
2026-10-08 21:23:49 -04:00
parent ab9391d019
commit 2ac37455f0
73 changed files with 7313 additions and 228 deletions
+299 -51
View File
@@ -52,12 +52,14 @@
*
* Phase 97 (task 04) — the catalog becomes the DRILL-DOWN TREE the
* agent's `ls` sees (the phase-94 concept, one end to end). The load
* is now `loadTree()`: ONE fetch of GET /api/docs/tree returns the
* FULL recursive tree (sources → folders → files, each folder carrying
* is `loadTree()` (sources → folders → files, each folder carrying
* its STORED description — the rows the agent reads), and the view
* drills CLIENT-SIDE from that single fetch — zero per-level fetches,
* no URL change (the navigation is a re-render of the already-fetched
* tree).
* drills level by level with no URL change (the navigation is a
* re-render). PHASE 167 SUPERSEDED the single-payload model here
* (TODO L5 — the 2 MB first load, LOCKED A3): ONE
* `GET /api/docs/tree?depth=1` plus ONE node fetch per not-yet-loaded
* level — see the Phase 167 (task 04) section; the drill UX itself
* (breadcrumb, scroll panel, editing) is unchanged.
*
* • the drill state is module-scoped: `current` —
* { source: null, folder: null } = the top level (the rows ARE the
@@ -76,8 +78,11 @@
* the same-page modal; the no-JS href escape hatch intact), hidden
* when the level has none (at the top level it is ALWAYS hidden —
* files are seen per source, as with ls(source)).
* • the stat cards walk the WHOLE tree (document count, chunks sum,
* max indexed_at) — the values identical to the former flat walk.
* • the stat cards show the KB-wide numbers (document count, chunks
* sum, max indexed_at) — identical to the former flat walk. Since
* phase 167 they read the server's `totals` (a depth-limited tree
* cannot be summed client-side); the whole-tree walk stays only as
* the defensive no-`totals` fallback.
* • the empty state (#sources-empty) is now the "zero SOURCES"
* semantic (nothing registered, nothing indexed) — the deliberate
* phase-97 change: a registered 0-document source renders its row
@@ -423,6 +428,35 @@
* control's immediate disable keeps the never-stale contract). The
* running label, the button title, and the button aria-label are
* byte-identical to pre-phase; no copy changed.
*
* Phase 167 (task 04, D4/A3 — TODO L5) — the LAZY DRILL: the 2 MB
* first payload is gone. The catalog load is ONE
* `fetch("/api/docs/tree?depth=1")` (the sources plus one level plus
* the server's `totals`), and a folder the payload only NAMES
* (`node.truncated === true`, not yet in `levelCache`) fetches
* `GET /api/docs/tree/node?source=…&path=…` the FIRST time it opens —
* the merged children ride the node (Object.assign, so
* `currentLevelNode()`/`renderLevel()` keep their pre-phase semantics
* for every loaded level), the level is cached (key
* `source + "/" + folder`), and re-opening it is free (zero network).
* A NEW tree load clears the cache — the sync/upload/re-show refresh
* call sites keep working unchanged — and rehydrates the user's OWN
* level when the fresh payload truncates it (a refresh never paints a
* half-empty level). §7.4 never-stale, in full: while the node fetch
* runs the level the user is LEAVING stays rendered, the clicked link
* carries aria-busy + aria-disabled (drillPending — a second click
* cannot start a parallel fetch; adopted popstate navigations
* rehydrate their level the same way, via ensureCurrentLevel) and
* ONE role="status"
* "Loading <folder>…" row sits in the folders panel; a failure shows
* a role="alert" line with a real Retry <button> and keeps the
* current level rendered; the phase-79 loadSeq token EXTENDS to the
* drill (a response superseded by a newer load — or by a navigation —
* never paints). The node route's 404 IS the vanished-folder signal:
* the honest reset to the top level, nothing cached. The existence
* rules stop walking a full tree — folderKnownIn() only claims what
* the HELD levels prove — and treeStats() prefers kbTree.totals (the
* walk is the pre-phase-167 fallback only).
*/
import { fetchIsAdmin } from "./header.js";
@@ -1622,11 +1656,30 @@ export async function mount(root) {
let loadSeq = 0;
let levelEditor = null; // phase 97 (task 05): the level editor handle
/* Phase 167 (task 04, D4/A3 — the TODO L5 2 MB fix): the client no
longer holds the whole tree. `levelCache` marks the folder levels
the drill has already fetched (key: source + "/" + the
source-relative folder path); the merged children live ON THE
NODES (Object.assign in fetchLevel), so currentLevelNode() and
renderLevel() keep their pre-phase semantics for every loaded
level. `drillPending` is the in-flight drill { source, folder } —
a second drill click is a no-op while a node fetch runs (§7.4:
every settle path re-enables the controls). `navSeq` bumps on
every applyTarget so a navigation invalidates an in-flight drill
exactly like a newer load does. A NEW tree load clears the cache
(the sync/refresh call sites keep working unchanged) and any
in-flight drill with it. */
const levelCache = new Map();
let drillPending = null;
let drillRowEl = null; // the ONE in-flight row (loading OR alert)
let navSeq = 0;
async function loadTree() {
const my = ++loadSeq;
drillPending = null; // any in-flight drill is invalidated by this load
let r;
try {
r = await fetch("/api/docs/tree");
r = await fetch("/api/docs/tree?depth=1");
} catch {
if (my === loadSeq) renderEmpty();
return;
@@ -1638,14 +1691,25 @@ export async function mount(root) {
const tree = await r.json();
if (my !== loadSeq) return; // a newer load owns the DOM now
kbTree = tree && Array.isArray(tree.sources) ? tree : { sources: [] };
levelCache.clear(); // a fresh tree invalidates every drilled level
resetVanishedLocation();
// Phase 167: the user's own level may be one the fresh depth-1
// payload only NAMES (truncated, cache just cleared) — rehydrate
// it through the node route BEFORE it paints, so a refresh never
// shows a half-empty level.
if (needsLevelFetch(currentLevelNode(), current.source)) {
enterFolder({ source: current.source, folder: current.folder }, null, false);
return;
}
renderLevel();
}
/* Never-stale (PLAN §7.4, phase 97): after a re-fetch, the drilled
* location may no longer exist in the NEW tree (the source was
* unregistered/pruned, the folder vanished) — reset to the top level
* BEFORE rendering: no stale breadcrumb, no stale block. */
* BEFORE rendering: no stale breadcrumb, no stale block. Since
* phase 167 the check is folderKnownIn (what the HELD levels prove)
* — the fetch path's own vanished-signal is the node route's 404. */
function resetVanishedLocation() {
if (current.source === null) return;
const src = kbTree.sources.find((s) => s.name === current.source);
@@ -1653,39 +1717,48 @@ export async function mount(root) {
current = { source: null, folder: null };
return;
}
if (current.folder !== null && !folderExistsIn(src, current.folder)) {
if (current.folder !== null && !folderKnownIn(src, current.folder)) {
current = { source: null, folder: null };
}
}
/* The phase-94 existence rule, mirrored client-side over the tree's
* file paths: a folder exists under the source iff some of the
* source's indexed paths starts with `folder + "/"` (the source root
* — `""` — always exists). */
function folderExistsIn(src, folderPath) {
if (folderPath === "") return true;
const paths = [];
const collect = (node) => {
for (const child of node.children || []) {
if (child.kind === "file") paths.push(child.path);
else collect(child);
}
};
collect(src);
return paths.some((p) => p.startsWith(folderPath + "/"));
}
/* The node for `current`: the source node at the root, or the folder
* node found by walking the tree with the folder's cumulative
* source-relative path (the builder's folder paths are
* source-relative, so `one/two` resolves one level at a time). */
function currentLevelNode() {
const src = kbTree.sources.find((s) => s.name === current.source);
if (!src) return null;
if (current.folder === null || current.folder === "") return src;
/* Phase 167 (task 04, D4): the client no longer holds the whole
* tree, so the phase-97 full-tree existence walk becomes KNOWN-ness:
* a folder is known iff every node of its ancestor chain exists in
* what the client holds (the depth-1 tree + the drilled levels —
* merged children ride the nodes, so one walk over the held children
* answers). A broken chain means the folder is gone OR simply not
* loaded that deep — for the never-stale RESET both are honestly
* "back to the top level". The drill path does not guess: the node
* route's 404 IS the "the folder vanished" signal (fetchLevel). */
function folderKnownIn(src, folderPath) {
if (folderPath === "") return true; // the source root always exists
let node = src;
let acc = "";
for (const part of current.folder.split("/")) {
for (const part of folderPath.split("/")) {
acc = acc ? acc + "/" + part : part;
const next = (node.children || []).find(
(c) => c.kind === "folder" && c.path === acc
);
if (!next) return false;
node = next;
}
return true;
}
/* The node for a drill target (phase 97, phase 167-generalized):
* the source node at the root, or the folder node found by walking
* the HELD levels with the folder's cumulative source-relative path
* (the builder's folder paths are source-relative, so `one/two`
* resolves one level at a time; a truncated node simply has no
* children to walk into). */
function nodeForTarget(target) {
const src = kbTree.sources.find((s) => s.name === target.source);
if (!src) return null;
if (target.folder === null || target.folder === "") return src;
let node = src;
let acc = "";
for (const part of target.folder.split("/")) {
acc = acc ? acc + "/" + part : part;
const next = (node.children || []).find(
(c) => c.kind === "folder" && c.path === acc
@@ -1696,11 +1769,27 @@ export async function mount(root) {
return node;
}
/* The KB-wide stat cards (phase 97): walk the WHOLE tree — document
* count, chunks sum, max indexed_at — the values identical to the
* former flat /api/docs walk (the same documents, one level deeper).
* fmtDate reuse for the max. */
function currentLevelNode() {
return nodeForTarget(current);
}
/* The KB-wide stat cards. Phase 167 (task 04, D2/D4): the server's
* `totals` (computed from the FULL catalogue rows beside any depth
* of tree) are the source of truth — a depth-limited tree CANNOT be
* summed client-side. The phase-97 recursive walk (document count,
* chunks sum, max indexed_at — the values identical to the former
* flat /api/docs walk) stays ONLY as the defensive fallback for a
* tree served without `totals` (a pre-phase-167 process) — the
* no-crash path. fmtDate reuse for the max (renderLevel). */
function treeStats() {
const totals = kbTree.totals;
if (totals && typeof totals === "object") {
return {
docs: totals.documents,
totalChunks: totals.chunks,
last: totals.last_indexed || "",
};
}
let docs = 0;
let totalChunks = 0;
let last = "";
@@ -1722,8 +1811,10 @@ export async function mount(root) {
/* The breadcrumb (phase 97): hidden at the top level; when drilled
* in, one link per ancestor — the top level (back to the sources
* list), the source, then the folder chain — the LAST segment a
* span with aria-current="page". Client-side only: no fetch, no URL
* change (the navigation is a re-render of the fetched tree). */
* span with aria-current="page". No URL change (the navigation is a
* re-render); phase 167: an ancestor is already loaded (you cannot
* drill past it without holding it), so a breadcrumb step costs no
* fetch in practice — goTo stays the ONE drill code either way. */
function renderCrumb() {
if (!crumbEl) return;
crumbEl.replaceChildren();
@@ -1766,7 +1857,7 @@ export async function mount(root) {
a.textContent = label; // document-derived text — never innerHTML
a.addEventListener("click", (e) => {
e.preventDefault();
goTo(target);
goTo(target, a);
});
return a;
}
@@ -1779,9 +1870,9 @@ export async function mount(root) {
return span;
}
/* The drill navigation (phase 99, task 02, D2 — client-side, no
* fetch; the drill state IS the history state). `applyTarget(target,
* push)`: when `push` is true the target DIFFERS from the current
/* The drill navigation (phase 99, task 02, D2 — the drill state IS
* the history state; since phase 167 a user drill may FIRST fetch
* the level — see enterFolder). `applyTarget(target, push)`: when `push` is true the target DIFFERS from the current
* level (a re-click of the current row/segment pushes NO duplicate
* entry), it FIRST records a STATE-ONLY entry — history.pushState(
* { view: "rag", kb: target }, "") — the second arg "" + no third:
@@ -1795,14 +1886,166 @@ export async function mount(root) {
history.pushState({ view: "rag", kb: target }, "");
}
current = { source: target.source, folder: target.folder };
navSeq++; // phase 167: any navigation invalidates an in-flight drill
renderLevel();
}
/* Every user-initiated drill (source row, folder link, breadcrumb
* segment) — push a history entry, then drill (D2, phase 99 task
* 02). */
function goTo(target) {
applyTarget(target, true);
* 02). Since phase 167 (task 04, A3) the drill may FIRST fetch the
* level (enterFolder); the push lands when the level is in hand, so
* a failed drill records nothing. `link` is the clicked link — it
* carries the in-flight aria-busy/aria-disabled state. */
function goTo(target, link) {
enterFolder(target, link, true);
}
/* Phase 167 (task 04, D4/A3) — the lazy drill. The depth-1 tree
* NAMES deeper folders (truncated: true, children not built); the
* FIRST open of one fetches its level through the node route, merges
* the children into the node, and caches the level — a re-open is
* free (zero network). A level the client already holds (or a node
* the payload fully built) drills with NO fetch, exactly as before:
* applyTarget + render, nothing else. One drill at a time —
* drillPending is the §7.4 double-click guard, and every settle
* path (success, failure, 404, superseded) releases it. */
function needsLevelFetch(node, source) {
return (
!!node &&
node.truncated === true &&
!levelCache.has(source + "/" + node.path)
);
}
async function enterFolder(target, link, push) {
if (drillPending) return; // one drill at a time (§7.4 double-click guard)
const node = nodeForTarget(target);
if (needsLevelFetch(node, target.source)) {
drillPending = { source: target.source, folder: target.folder };
const loaded = await fetchLevel(target, node, link);
if (!loaded) return; // failed / vanished / superseded — handled inside
}
applyTarget(target, push);
}
/* ONE node fetch + its full §7.4 lifecycle. The fetch URL is
* /api/docs/tree/node?source=…&path=… (path source-relative; the
* route is documented in app/api/docs.py — one catalogue query per
* drill, no N+1). The race convention EXTENDS from loadTree: the
* response may only paint while NO newer load (loadSeq, phase 79)
* and NO navigation (navSeq) have happened — a stale response never
* touches the DOM. Success: Object.assign merges the node (children,
* counts, dates, summary, pending flags — the wire shape the full
* tree produces) and levelCache marks the level; only a 200 caches.
* A 404 IS the vanished-folder signal — honest reset to the top
* level, nothing cached. Any other failure: the alert + Retry row,
* the current level staying rendered. */
async function fetchLevel(target, targetNode, link) {
const my = loadSeq;
const nav = navSeq;
if (link) {
link.setAttribute("aria-busy", "true");
link.setAttribute("aria-disabled", "true"); // a double click cannot refetch
}
showDrillLoading(target.folder.split("/").pop());
let response = null;
let vanished = false;
try {
response = await fetch("/api/docs/tree/node?source=" + encodeURIComponent(target.source) + "&path=" + encodeURIComponent(target.folder));
if (response.status === 404) vanished = true;
} catch {
response = null; // network failure — the alert path below
}
// Every settle path re-enables the controls (§7.4) — the link is
// only ours to clear while this drill is still THE pending one (a
// newer load owns the flags the moment it starts).
if (drillPending && drillPending.source === target.source && drillPending.folder === target.folder) {
drillPending = null;
}
if (link) {
link.removeAttribute("aria-busy");
link.removeAttribute("aria-disabled");
}
if (my !== loadSeq || nav !== navSeq) return false; // superseded — never paints
clearDrillRow();
if (vanished) {
// The folder vanished under us (pruned/unregistered since the
// level was listed) — the phase-97 never-stale contract: reset
// to the top level, no stale breadcrumb, nothing cached.
current = { source: null, folder: null };
renderLevel();
return false;
}
if (response === null || !response.ok) {
showDrillFailure(target, link); // the current level stays rendered
return false;
}
const merged = await response.json();
if (my !== loadSeq || nav !== navSeq) return false; // stale — never paints
Object.assign(targetNode, merged);
levelCache.set(target.source + "/" + target.folder, true);
return true;
}
/* The ONE in-flight row of the folders panel — appended to the
* level the user is LEAVING (renderLevel's tbody clear owns it: a
* re-render always drops the row). Tokens-only .kb-level-loading
* (styles.css); textContent only, the house rule. */
function clearDrillRow() {
if (drillRowEl) {
drillRowEl.remove();
drillRowEl = null;
}
}
function showDrillLoading(name) {
clearDrillRow();
const row = document.createElement("tr");
row.className = "kb-level-loading";
const td = document.createElement("td");
td.colSpan = 4;
const note = document.createElement("span");
note.setAttribute("role", "status");
note.textContent = "Loading " + name + "…"; // never innerHTML
td.appendChild(note);
row.appendChild(td);
drillRowEl = row;
if (foldersWrap) foldersWrap.hidden = false; // the row needs its panel
if (foldersTbody) foldersTbody.appendChild(row);
}
/* Phase 167: rehydrate the CURRENT level when the held tree only
* names it — used by the ADOPTED navigations (popstate: the popped
* entry can point at a level the client has not fetched since the
* last load; the bor:view-refresh alignment is covered by loadTree's
* own rehydration). A no-op for held levels and at the top. */
function ensureCurrentLevel() {
if (drillPending) return;
if (needsLevelFetch(currentLevelNode(), current.source)) {
enterFolder({ source: current.source, folder: current.folder }, null, false);
}
}
function showDrillFailure(target, link) {
clearDrillRow();
const row = document.createElement("tr");
row.className = "kb-level-loading";
const td = document.createElement("td");
td.colSpan = 4;
const note = document.createElement("span");
note.setAttribute("role", "alert");
note.textContent =
"Couldn't load " + target.folder.split("/").pop() + " — "; // never innerHTML
const retry = document.createElement("button");
retry.type = "button"; // a REAL button (the phase-49 .retry-btn pill)
retry.className = "retry-btn";
retry.textContent = "Retry";
retry.addEventListener("click", () => enterFolder(target, link, true));
td.append(note, retry);
row.appendChild(td);
drillRowEl = row;
if (foldersWrap) foldersWrap.hidden = false;
if (foldersTbody) foldersTbody.appendChild(row);
}
/* ONE table for every level (phase 97): at the TOP level the rows
@@ -1818,7 +2061,7 @@ export async function mount(root) {
link.textContent = s.name; // document-derived text — never innerHTML
link.addEventListener("click", (e) => {
e.preventDefault();
goTo({ source: s.name, folder: "" });
goTo({ source: s.name, folder: "" }, link);
});
nameTd.appendChild(link);
tr.appendChild(nameTd);
@@ -1853,7 +2096,7 @@ export async function mount(root) {
link.textContent = f.path.split("/").pop(); // never innerHTML
link.addEventListener("click", (e) => {
e.preventDefault();
goTo({ source: current.source, folder: f.path });
goTo({ source: current.source, folder: f.path }, link);
});
nameTd.appendChild(link);
tr.appendChild(nameTd);
@@ -2209,6 +2452,11 @@ export async function mount(root) {
const kb = event.state && event.state.kb;
if (kb) applyTarget(kb, false);
else applyTarget({ source: null, folder: null }, false);
// Phase 167 (task 04): a forward/back can adopt an entry whose
// level the client has not fetched since — rehydrate it through
// the node route (a no-op for held levels; a truly vanished one
// takes the 404-reset / alert path inside).
ensureCurrentLevel();
});
loadTree();
}