phase: 167_api_payload_diet
All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
@@ -11,8 +11,18 @@ file metadata, and the ``group_folder_listing`` cross-check property
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app.api.docs import TreeDocRow, build_kb_tree
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
|
||||
from app.api.docs import (
|
||||
TreeDocRow,
|
||||
_folder_counts,
|
||||
_kb_totals,
|
||||
_subtree_max,
|
||||
build_kb_tree,
|
||||
)
|
||||
from app.rag.agent import group_folder_listing
|
||||
from app.schemas import KbTree, KbTreeFile, KbTreeTotals
|
||||
|
||||
T0 = "2026-09-01T08:00:00+00:00"
|
||||
T1 = "2026-09-02T08:00:00+00:00"
|
||||
@@ -633,3 +643,284 @@ def test_image_affordance_absent_without_the_map() -> None:
|
||||
assert set(file.model_dump()) == {
|
||||
"kind", "path", "title", "chunks", "created_at", "indexed_at"
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 167 (task 02, D2/A2) — the depth budget: a node whose children were
|
||||
# NOT built says so (``truncated``) while keeping every field the Sources
|
||||
# view renders (recursive count, recursive updated_at — the D9 max via the
|
||||
# _folder_counts max-date map, summary, summary_pending), the untruncated
|
||||
# wire shape stays byte-identical, and the catalogue totals ride the
|
||||
# response (``_kb_totals`` — the stat cards cannot be summed client-side
|
||||
# off a truncated tree).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The depth fixture: one chain four levels deep, distinct created dates
|
||||
#: (C0 < C1 < C2 < C3 — the DEEPEST file holds the max, the D9 trap) and
|
||||
#: distinct indexed stamps.
|
||||
DEPTH_ROWS: list[TreeDocRow] = [
|
||||
("S", "root.md", "Root", 1, T0, C0),
|
||||
("S", "a/top.md", "Top", 2, T1, C1),
|
||||
("S", "a/b/mid.md", "Mid", 3, T1, C2),
|
||||
("S", "a/b/c/deep.md", "Deep", 4, T2, C3),
|
||||
]
|
||||
|
||||
DEPTH_SUMMARIES = {("S", "a"): "A desc."} # a/b and a/b/c stay un-stored
|
||||
|
||||
|
||||
def _all_nodes(node):
|
||||
"""The node plus every descendant node (folders and files)."""
|
||||
yield node
|
||||
for child in getattr(node, "children", []):
|
||||
if child.kind == "folder":
|
||||
yield from _all_nodes(child)
|
||||
else:
|
||||
yield child
|
||||
|
||||
|
||||
def _all_folder_nodes(node):
|
||||
"""Every FOLDER descendant (the SOURCE node itself is excluded — it
|
||||
carries ``name``, not ``path``, and has no ``kind``)."""
|
||||
return [n for n in _all_nodes(node) if getattr(n, "kind", None) == "folder"]
|
||||
|
||||
|
||||
def _paths_under(node) -> set[str]:
|
||||
"""Paths of every descendant (folders and files); the source node
|
||||
has no path of its own."""
|
||||
out: set[str] = set()
|
||||
for child in getattr(node, "children", []):
|
||||
out.add(child.path)
|
||||
if child.kind == "folder":
|
||||
out |= _paths_under(child)
|
||||
return out
|
||||
|
||||
|
||||
def test_depth_none_is_byte_identical_to_the_unlimited_build() -> None:
|
||||
"""``depth=None`` (the builder default) reproduces today's tree
|
||||
byte-for-byte: the default call, the explicit None, and an
|
||||
over-deep budget all serialize identically, and NO node carries the
|
||||
truncated key (the omission rule)."""
|
||||
default = [s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)]
|
||||
explicit = [
|
||||
s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=None)
|
||||
]
|
||||
over_deep = [
|
||||
s.model_dump() for s in build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=8)
|
||||
]
|
||||
assert default == explicit == over_deep
|
||||
(source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
|
||||
assert all("truncated" not in n.model_dump() for n in _all_nodes(source))
|
||||
# The full chain IS built: every level of the fixture is present.
|
||||
assert _paths_under(source) == {
|
||||
"root.md",
|
||||
"a",
|
||||
"a/top.md",
|
||||
"a/b",
|
||||
"a/b/mid.md",
|
||||
"a/b/c",
|
||||
"a/b/c/deep.md",
|
||||
}
|
||||
|
||||
|
||||
def test_depth_one_ships_sources_plus_one_level_with_truncated_flags() -> None:
|
||||
"""``depth=1``: the source's direct children are built (direct file
|
||||
``root.md`` included) and the subfolder with unloaded contents
|
||||
carries the FULL truncation field set — ``truncated is True``,
|
||||
``children == []``, the RECURSIVE count, the RECURSIVE max date
|
||||
(the deep C3 — D9 survives without the children), the stored
|
||||
``summary`` and the ``summary_pending`` flag (rule unchanged)."""
|
||||
(source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=1)
|
||||
assert [f.path for f in _file_nodes(source)] == ["root.md"]
|
||||
(a,) = _folder_nodes(source)
|
||||
assert a.path == "a"
|
||||
assert a.truncated is True
|
||||
assert a.model_dump()["truncated"] is True
|
||||
assert a.children == []
|
||||
assert a.documents == 3 # recursive — the count rule is depth-proof
|
||||
assert a.updated_at == C3 # the DEEP max, no built children to derive from
|
||||
assert a.summary == "A desc." # the level row shows a subfolder's description
|
||||
assert a.summary_pending is False # the stored row covers it (phase 98)
|
||||
# Nothing deeper is built anywhere.
|
||||
assert _paths_under(source) == {"root.md", "a"}
|
||||
|
||||
|
||||
def test_depth_two_builds_two_levels_and_truncates_the_deeper_one() -> None:
|
||||
"""``depth=2``: level 1 (``a``) is expanded UNTRUNCATED (its direct
|
||||
file ``a/top.md`` ships), level 2 (``a/b``) is the truncated node —
|
||||
the pending rule applies to it (2 docs, no stored row →
|
||||
``summary_pending`` true with ``summary None``), and neither its
|
||||
own file (``a/b/mid.md``) nor the third level appear: the budget
|
||||
removes the LEVELS below it, level by level."""
|
||||
(source,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=2)
|
||||
(a,) = _folder_nodes(source)
|
||||
assert "truncated" not in a.model_dump() # untruncated: key absent
|
||||
assert [f.path for f in _file_nodes(a)] == ["a/top.md"]
|
||||
(a_b,) = _folder_nodes(a)
|
||||
assert a_b.truncated is True
|
||||
assert a_b.children == []
|
||||
assert (a_b.documents, a_b.updated_at) == (2, C3) # mid + deep, max C3
|
||||
assert a_b.summary is None
|
||||
assert a_b.summary_pending is True # the phase-98 rule rides truncated nodes
|
||||
assert _paths_under(source) == {
|
||||
"root.md",
|
||||
"a",
|
||||
"a/top.md",
|
||||
"a/b",
|
||||
}
|
||||
|
||||
|
||||
def test_source_node_is_never_truncated_at_any_depth() -> None:
|
||||
"""The view's top level is source → level 1, so the budget never
|
||||
marks the SOURCE node truncated (at any depth, empty sources
|
||||
included) — its wire shape never carries the key."""
|
||||
(ref,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
|
||||
for depth in (1, 2, 3, 8):
|
||||
full, empty = build_kb_tree(
|
||||
["S", "Empty"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=depth
|
||||
)
|
||||
assert full.truncated is False
|
||||
assert "truncated" not in full.model_dump()
|
||||
assert empty.truncated is False
|
||||
assert "truncated" not in empty.model_dump()
|
||||
assert empty.children == []
|
||||
# Direct children always built: the source's level is complete.
|
||||
assert [c.path for c in full.children] == [c.path for c in ref.children]
|
||||
|
||||
|
||||
def test_truncated_key_never_touches_the_file_node_shape() -> None:
|
||||
"""A FILE node has no children to truncate: the field does not even
|
||||
exist on the class, and no dump ever carries it (the phase-122 text
|
||||
wire shape stays byte-identical at every depth)."""
|
||||
assert "truncated" not in KbTreeFile.model_fields
|
||||
for depth in (1, 2, 8):
|
||||
(source,) = build_kb_tree(["S"], DEPTH_ROWS, {}, depth=depth)
|
||||
file_nodes = [n for n in _all_nodes(source) if getattr(n, "kind", None) == "file"]
|
||||
assert file_nodes # the fixture has files at every shipped level
|
||||
for node in file_nodes:
|
||||
assert "truncated" not in node.model_dump()
|
||||
|
||||
|
||||
def test_max_dates_map_agrees_with_subtree_max_at_every_depth() -> None:
|
||||
"""One concept, two derivations, cross-pinned: the
|
||||
``_folder_counts`` max-date map equals ``_subtree_max`` over the
|
||||
FULLY expanded children for every folder (the expanded builder path
|
||||
stays ``_subtree_max``, so a divergence fails loudly) — and every
|
||||
node present in a depth-limited tree carries the SAME
|
||||
``updated_at`` as the full build, truncated or not (the phase-106
|
||||
D9 column survives the payload diet)."""
|
||||
file_rows = [
|
||||
(path, title, chunks, indexed, created)
|
||||
for _source, path, title, chunks, indexed, created in DEPTH_ROWS
|
||||
]
|
||||
_folders, _counts, max_dates = _folder_counts(file_rows)
|
||||
(full,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES)
|
||||
full_dates = {n.path: n.updated_at for n in _all_folder_nodes(full)}
|
||||
assert full_dates == max_dates # the cross-pin at full expansion
|
||||
for node in _all_folder_nodes(full):
|
||||
# The expanded node's derivation equals the aggregate map.
|
||||
assert _subtree_max(node.children) == max_dates[node.path]
|
||||
for depth in (1, 2, 3):
|
||||
(limited,) = build_kb_tree(["S"], DEPTH_ROWS, DEPTH_SUMMARIES, depth=depth)
|
||||
for node in _all_folder_nodes(limited):
|
||||
assert node.updated_at == max_dates[node.path] == full_dates[node.path]
|
||||
# The source max also spans the UNSENT levels.
|
||||
assert limited.updated_at == C3
|
||||
|
||||
|
||||
def test_truncation_keeps_the_phase_94_and_98_invariants_at_every_depth() -> None:
|
||||
"""Existence rule, ordering, counts, summaries, pending markers and
|
||||
dates: every node a limited tree DOES send is field-for-field the
|
||||
node the full tree sends (the depth budget removes depth only — it
|
||||
never changes what a shipped level says), and the top level is
|
||||
identical at every depth."""
|
||||
(full,) = build_kb_tree(["S"], CROSS_ROWS, CROSS_SUMMARIES)
|
||||
full_map = {
|
||||
n.path: (n.documents, n.summary_pending, n.updated_at, n.summary)
|
||||
for n in _all_folder_nodes(full)
|
||||
}
|
||||
for depth in (1, 2, 3):
|
||||
(limited,) = build_kb_tree(["S"], CROSS_ROWS, CROSS_SUMMARIES, depth=depth)
|
||||
# Same level-1 rows, same order, at every depth.
|
||||
assert [c.path for c in limited.children] == [c.path for c in full.children]
|
||||
for node in _all_folder_nodes(limited):
|
||||
assert (
|
||||
node.documents,
|
||||
node.summary_pending,
|
||||
node.updated_at,
|
||||
node.summary,
|
||||
) == full_map[node.path]
|
||||
# Existence rule: a node is present ⟺ a catalogue prefix.
|
||||
assert node.path in full_map
|
||||
# Every truncated node is exactly the truncation contract.
|
||||
for node in _all_folder_nodes(limited):
|
||||
if node.truncated:
|
||||
assert node.children == []
|
||||
else:
|
||||
assert node.model_dump().get("truncated") is None
|
||||
|
||||
|
||||
def test_truncated_nodes_keep_the_image_affordance_at_their_level() -> None:
|
||||
"""Phase 122's affordance is row-driven, not depth-driven: a DIRECT
|
||||
image file of the source root still carries the thumbnail keys at
|
||||
``depth=1``; the truncated folder below simply does not list its
|
||||
own rows (they arrive with the drill)."""
|
||||
rows = [
|
||||
("S", "pic.png", "pic", 1, T0, C0),
|
||||
("S", "d/x.png", "x", 1, T0, C0),
|
||||
]
|
||||
images = {
|
||||
("S", "pic.png"): (DOC_ID, "A red square."),
|
||||
("S", "d/x.png"): (DOC_ID, "A blue circle."),
|
||||
}
|
||||
(source,) = build_kb_tree(["S"], rows, {}, images, depth=1)
|
||||
(pic,) = _file_nodes(source)
|
||||
assert pic.is_image is True
|
||||
assert pic.model_dump()["image_url"] == f"/api/documents/{DOC_ID}/image"
|
||||
assert pic.model_dump()["summary"] == "A red square."
|
||||
(d,) = _folder_nodes(source)
|
||||
assert d.truncated is True
|
||||
assert d.children == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _kb_totals (phase 167, task 02) — the stat-card totals the endpoint ships
|
||||
# beside any depth of tree (no extra query, empty-KB honest).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_kb_totals_math_over_the_catalogue_rows() -> None:
|
||||
"""documents = the row count, chunks = the sum, last_indexed = the
|
||||
MAX indexed stamp (lexicographic over one ISO shape = chronological
|
||||
— the same rule as the D9 max)."""
|
||||
totals = _kb_totals(DEPTH_ROWS)
|
||||
assert totals.documents == 4
|
||||
assert totals.chunks == 10 # 1 + 2 + 3 + 4
|
||||
assert totals.last_indexed == T2 # the newest indexed_at
|
||||
|
||||
|
||||
def test_kb_totals_empty_kb_is_zero_and_none() -> None:
|
||||
"""The empty KB: 0 documents, 0 chunks, ``last_indexed is None`` —
|
||||
the stat cards render the designed empty state, never a crash."""
|
||||
totals = _kb_totals([])
|
||||
assert totals == KbTreeTotals(documents=0, chunks=0, last_indexed=None)
|
||||
assert totals.model_dump() == {
|
||||
"documents": 0,
|
||||
"chunks": 0,
|
||||
"last_indexed": None,
|
||||
}
|
||||
|
||||
|
||||
def test_kb_tree_totals_default_and_floor_validation() -> None:
|
||||
"""``KbTree.totals`` is an additive top-level key with an honest
|
||||
empty default (an older call shape still serializes), and the
|
||||
ge=0 bounds refuse negative totals loudly (the endpoint computes
|
||||
them — a negative is a bug, never data)."""
|
||||
assert KbTree(sources=[]).model_dump()["totals"] == {
|
||||
"documents": 0,
|
||||
"chunks": 0,
|
||||
"last_indexed": None,
|
||||
}
|
||||
with pytest.raises(ValidationError):
|
||||
KbTreeTotals(documents=-1, chunks=0)
|
||||
with pytest.raises(ValidationError):
|
||||
KbTreeTotals(documents=0, chunks=-1)
|
||||
|
||||
Reference in New Issue
Block a user