phase: 167_api_payload_diet
All gates green. Final report: **Verified (phase 167 final pass — all 6 tasks already in `complete/`)** - Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. - Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. **Gates (all 100% green)** - `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). **Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): - Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. - AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). - Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. **Notable decisions / deviations** - `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. - A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. - Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. **Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
This commit is contained in:
@@ -0,0 +1,236 @@
|
||||
# Phase 167 · Task 05 — Measured API payload audit (the byte budget)
|
||||
|
||||
Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any
|
||||
other endpoints** that could load a ton of data." This file is the measured answer —
|
||||
every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` /
|
||||
server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a
|
||||
report, never a 413 — nothing was rejected, nothing was capped.
|
||||
|
||||
## How it was run
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-10-08 |
|
||||
| Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) |
|
||||
| App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) |
|
||||
| BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had |
|
||||
| AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) |
|
||||
| Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) |
|
||||
| Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below |
|
||||
|
||||
The script is **read-only**: it signs in through `POST /api/login` (the session
|
||||
cookie — the single-admin login is password-only, PLAN A10, so there is no
|
||||
`--username`) and GETs. It never mutates, so it is safe against a production target
|
||||
(`--base-url https://…`).
|
||||
|
||||
### Corpus caveat (read this before comparing with the 2 MB report)
|
||||
|
||||
The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents**
|
||||
(verified: `select count(*) from folder_summaries` → `0`,
|
||||
`select count(*) from documents where is_image` → `0`). The production 2 MB payload
|
||||
was 1,000 documents **plus** the hundreds of inlined AI folder descriptions
|
||||
(phase 127/155) and the image documents' full vision descriptions (phase 122/124) —
|
||||
none of which exist in the snapshot. **So the identity bytes below are a lower bound
|
||||
on the production payload**, and what transfers from this measurement is the
|
||||
*structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the
|
||||
ONE level the RAG view renders instead of the whole recursion below it — and every
|
||||
level of a summary-carrying folder is where the production payload's fat lives),
|
||||
gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1
|
||||
tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict
|
||||
table below names the live re-run as the follow-up.
|
||||
|
||||
## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available
|
||||
|
||||
> Read the BEFORE table's `gzip` column as "no compressor existed": ratio
|
||||
> `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts
|
||||
> are NOT a pre-phase property — the conditional re-requests hit the validators
|
||||
> tasks 01-04 added, which this run left in place because only the transport was
|
||||
> being switched off (the honest way to measure "the same bytes, uncompressed").
|
||||
> Pre-phase there were no validators at all, so every repeat load cost the
|
||||
> identity column again.
|
||||
|
||||
```
|
||||
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||
endpoint identity gzip ratio ms verdict
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
tree depth=1 (default) 8,474 8,474 1.000 12 304
|
||||
tree depth=8 (full) 265,147 265,147 1.000 37 304
|
||||
tree node homelab/active 20,844 20,844 1.000 11 304
|
||||
docs catalogue 246,366 246,366 1.000 14 304
|
||||
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 892,124 1.000 8 LOOK (> 262144)
|
||||
note: content field 772573 chars
|
||||
saved chats list 12 12 1.000 1 ok
|
||||
saved chat (one) — — — — n/a
|
||||
note: no saved chats
|
||||
git sources 188 188 1.000 1 ok
|
||||
suggestions 74 74 1.000 1 ok
|
||||
sync status (polled) 272 272 1.000 1 ok
|
||||
tokens 13 13 1.000 2 ok
|
||||
config 188 188 1.000 1 ok
|
||||
whoami 37 37 1.000 1 ok
|
||||
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||
```
|
||||
|
||||
What the pre-phase browser actually paid for the **Sources page first paint** is the
|
||||
sum of the rows it fetches on load, and the pre-phase tree route answered the FULL
|
||||
recursive tree by default (`tree depth=8 (full)` is that shape):
|
||||
`265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app
|
||||
published **no validators at all**, so every repeat load re-paid the same 512 KB.
|
||||
|
||||
## AFTER — shipped defaults (depth-1 default + gzip + ETag)
|
||||
|
||||
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||
endpoint identity gzip ratio ms verdict
|
||||
----------------------------------------------------------------------------------------------------------------
|
||||
tree depth=1 (default) 8,474 1,129 0.133 13 304
|
||||
tree depth=8 (full) 265,147 23,520 0.089 41 304
|
||||
tree node homelab/active 20,844 1,064 0.051 11 304
|
||||
docs catalogue 246,366 45,107 0.183 23 304
|
||||
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 60,776 0.068 14 LOOK (> 262144)
|
||||
note: content field 772573 chars
|
||||
saved chats list 12 12 1.000 2 ok
|
||||
saved chat (one) — — — — n/a
|
||||
note: no saved chats
|
||||
git sources 188 188 1.000 1 ok
|
||||
suggestions 74 74 1.000 2 ok
|
||||
sync status (polled) 272 272 1.000 1 ok
|
||||
tokens 13 13 1.000 2 ok
|
||||
config 188 188 1.000 1 ok
|
||||
whoami 37 37 1.000 1 ok
|
||||
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||
```
|
||||
|
||||
Sources-page first paint, same sum against the AFTER gzip column:
|
||||
`1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity,
|
||||
and every one of those five rows revalidates to **304 / 0 bytes** on the next load).
|
||||
The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×**
|
||||
(31× from the depth limit × 7.5× from gzip), and the second visit is free.
|
||||
|
||||
## `curl` proof on the audited corpus
|
||||
|
||||
```
|
||||
$ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree
|
||||
etag: W/"53c86d5d554efd8487ad52ca6cd6f913"
|
||||
cache-control: private, must-revalidate
|
||||
content-length: 8474
|
||||
vary: Cookie, Accept-Encoding
|
||||
8474 bytes # wc -c b1.json
|
||||
|
||||
$ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree
|
||||
etag: W/"53c86d5d554efd8487ad52ca6cd6f913" # the validator survives the encoding
|
||||
content-encoding: gzip
|
||||
1129 bytes # wc -c b2.gz
|
||||
|
||||
$ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \
|
||||
-w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree
|
||||
status=304 size=0
|
||||
```
|
||||
|
||||
## The byte-budget log line (the answer to "which endpoint is heavy, right now")
|
||||
|
||||
Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO);
|
||||
the shipped default is `65536`, so at rest only the big ones reach the log.
|
||||
`grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what
|
||||
went on the wire — the two together are the "how fat, and how fat did the client see
|
||||
it" pair.
|
||||
|
||||
```
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40
|
||||
INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10
|
||||
INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22
|
||||
INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12
|
||||
INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0
|
||||
INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0
|
||||
```
|
||||
|
||||
`POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded
|
||||
payload to budget — and it is skipped by the gzip layer by exact path (which is why
|
||||
its line reads `enc=identity`). One real turn against this corpus, measured:
|
||||
|
||||
```
|
||||
INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1
|
||||
```
|
||||
|
||||
39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture
|
||||
ended because the measuring `curl --max-time 45` hung up — which is itself the
|
||||
other half of the contract: a stream the client walks away from still writes
|
||||
exactly ONE line, with the bytes that actually got out, never one line per frame
|
||||
and never a missing line). Pinned by
|
||||
`tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out`
|
||||
and by the integration pin
|
||||
`tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`.
|
||||
|
||||
## Verdict per candidate endpoint
|
||||
|
||||
| Endpoint | Measured (identity / wire) | Verdict |
|
||||
|---|---|---|
|
||||
| `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. |
|
||||
| `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). |
|
||||
| `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. |
|
||||
| `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). |
|
||||
| `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). |
|
||||
| `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. |
|
||||
| `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. |
|
||||
| `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). |
|
||||
| `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). |
|
||||
| `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. |
|
||||
| `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). |
|
||||
| `GET /api/config` | 188 | **ok** — display strings only. |
|
||||
| `GET /api/whoami` | 37 | **ok** — two fields. |
|
||||
| `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. |
|
||||
|
||||
## What the audit changed, and what it proved
|
||||
|
||||
* **No new remedy was needed beyond tasks 01-04.** The only over-budget row
|
||||
(`/api/documents/content`) is the documented accepted case, and
|
||||
`/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag
|
||||
was **not** applied — a validator on a sub-860-byte response costs a hash
|
||||
and saves nothing (it would not even compress).
|
||||
* **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06)
|
||||
drives the real Sources view in Chromium and pins what this file measured: the
|
||||
boot is ONE `?depth=1` request whose body carries `totals` and no deep children,
|
||||
the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary`
|
||||
(Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the
|
||||
wire answers `304` for a fraction of the first transfer while the table still
|
||||
renders from the cache, and a level the payload only names costs exactly one
|
||||
`/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands
|
||||
`fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP
|
||||
Network events — the DevTools-panel truth, not Playwright's post-cache view.
|
||||
* **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on
|
||||
the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the
|
||||
second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat
|
||||
load free. Had the phase shipped gzip alone, the Sources page would still
|
||||
have paid 23 KB/request forever, and on the summary-inflated production
|
||||
corpus proportionally more.
|
||||
* **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the
|
||||
LOOK bar, already validated + compressed). The byte-budget line at its
|
||||
default `65536` threshold logs it at INFO on every uncached load, so the
|
||||
corpus growth that pushes it over shows up in the log before it shows up
|
||||
for the owner.
|
||||
* **Two limits of this measurement, stated instead of hidden:** (1) the
|
||||
snapshot has no folder summaries and no image docs, so the identity
|
||||
numbers are a lower bound on the production payload (see the caveat
|
||||
above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather
|
||||
than measured — the script is read-only by design (it must stay safe
|
||||
against the deployed app), so it does not invent a chat to size. Both are
|
||||
re-measurable with one command against the live corpus.
|
||||
|
||||
## Re-running this audit
|
||||
|
||||
```bash
|
||||
# 1. the corpus (destructive to the local DB — see the skill's own backup step)
|
||||
# restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese
|
||||
uv run alembic upgrade head # the snapshot carries an old alembic version
|
||||
# 2. the app
|
||||
uv run uvicorn app.main:app --port 8000
|
||||
# 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override)
|
||||
uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
|
||||
# the BEFORE transport, for a before/after pair like the tables above:
|
||||
BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit
|
||||
```
|
||||
@@ -0,0 +1,22 @@
|
||||
All gates green. Final report:
|
||||
|
||||
**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
|
||||
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
|
||||
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.
|
||||
|
||||
**Gates (all 100% green)**
|
||||
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
|
||||
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).
|
||||
|
||||
**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
|
||||
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
|
||||
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
|
||||
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.
|
||||
|
||||
**Notable decisions / deviations**
|
||||
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
|
||||
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
|
||||
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.
|
||||
|
||||
**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,130 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 234 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 280 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 73 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6102 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -0,0 +1,19 @@
|
||||
**Task 01 complete — all green.**
|
||||
|
||||
**Implemented**
|
||||
- `app/core/compression.py`: `SelectiveGZipMiddleware` (pure ASGI, composes Starlette `GZipMiddleware`, `DEFAULT_SKIP_PATHS=("/api/chat",)` EXACT match — `/api/chat-images` still compresses) + `_RebufferSmallResponses` framing seam
|
||||
- `app/config.py`: `compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`) with `agent_max_rounds`-pattern validator (`0` = off, `-1` refuses startup naming the field)
|
||||
- `app/main.py`: wired between `configure_caching` and `SecurityHeadersMiddleware` (A20 stays outermost); `0` adds nothing at all; `.env.example` entry with the three-value comment
|
||||
- `tests/unit/test_compression.py` (19 tests: gzip/identity/floor/pre-encoded skip, SSE frame-by-frame raw-ASGI interleaving proof + `list(stream)` idiom, exact-match pin, lifespan/websocket pass-through, wiring order, kill switch); `tests/integration/test_docs_api.py` + 4 pins (tree gzip→decompress-equal, A20 headers on gzipped body, small `/api/health` identity, A19 page contract through gzip)
|
||||
|
||||
**Gates**
|
||||
- `uv run pytest` → **3903 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **99%** overall, `app/core/compression.py` **100%**
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
|
||||
- E2E isolated (exit 0): `test_smoke`, `test_chat_rag`, `test_cache_busting`, `test_asset_cache_revalidation`, `test_kb_tree`
|
||||
- Measured: 50-doc tree 9,708 → 571 bytes (**17×**), `Vary: … Accept-Encoding` present
|
||||
|
||||
**Docs**: README.md — gzip bullet in "Caching & deploys" + `BOR_COMPRESS_MIN_BYTES` config table row; AGENTS.md: no doc impact (no env/middleware content there).
|
||||
|
||||
**Decisions**: (1) The `_RebufferSmallResponses` seam was required: starlette 1.x's `CachingMiddleware` (BaseHTTPMiddleware, below gzip per LOCKED order) re-frames every pass-through body (`more_body=True`), voiding Starlette's minimum-size exemption — `/api/health` came back gzip-inflated (33→71 B). The seam only re-delivers the buffered message shape Starlette's own exemption checks; no compression math forked. (2) Frame-by-frame proof is a deterministic raw-ASGI interleaving test (TestClient buffers bodies and cannot prove incremental arrival).
|
||||
|
||||
**Next pending task**: `167_api_payload_diet/02_tree_depth_and_totals.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 273 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 71 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 487 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5952 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
**Task 02 complete — depth-limited KB tree + server totals (phase 167 D2, LOCKED A2).**
|
||||
|
||||
Implemented:
|
||||
- `app/schemas.py`: new `KbTreeTotals` (documents/chunks/last_indexed, ge=0-bounded); `KbTree.totals` (additive top-level key); `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `_drop_truncated_when_false` wrap serializer (the `_drop_image_fields` idiom) — untruncated nodes stay wire byte-identical; `KbTreeFile` untouched.
|
||||
- `app/api/docs.py`: `_folder_counts` → `(folders, counts, max_dates)` (recursive max `created_at` map, same pass); `_level_children(..., depth)` — budget-exhausted subfolders ship `truncated=true`, `children=[]`, recursive count + `max_dates`-derived `updated_at` (D9-proof), `summary`, `summary_pending`; `build_kb_tree(..., depth=None)` (None = byte-identical unlimited); source never truncated at depth≥1; endpoint gains `depth: Query(1, ge=1, le=8)` + `_kb_totals()` computed from the same rows (no extra query). `GET /api/docs` / agent `ls` untouched.
|
||||
- Tests: 11 new unit pins (depth matrix, byte-identity at None, truncated field set, `_subtree_max`↔`max_dates` cross-check at every depth, phase-94/98 invariants per depth, totals math/empty/validation, image affordance at depth 1); 4 new integration tests (default depth=1 + totals vs `/api/docs`, depth=3, 422 matrix, pre-phase wire-shape pin); migrated deep-walking pins to explicit `?depth=8` (`test_docs_api.py` ×6, `test_docs_api_dates.py` + new truncated-date test, `test_git_source_dates.py`, `test_document_dates.py`).
|
||||
- Docs: README RAG section documents depth/truncated/totals; AGENTS.md: no command/env changes → untouched.
|
||||
|
||||
Gates: `uv run pytest --cov=app` → **3919 passed, coverage 99%** (`app/api/docs.py` 99% — only the pre-existing defensive `continue` uncovered; `app/schemas.py` 100%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
|
||||
|
||||
Notable decisions:
|
||||
- **Deviation (plan gap):** `tests/e2e/test_document_dates.py` also fetched the tree with no depth (its `_find_file` recurses) — same repair class as item 9's two suites; migrated with `?depth=8` too.
|
||||
- **Expected interim E2E state:** client `sources.js` (task 04's scope) still walks the tree for stat cards/drill, so `test_kb_tree.py` stat/drill assertions now fail against the intentional depth=1 default (verified the failure is exactly `#stat-docs` under-counting, not a server defect). This is the owner-confirmed A3 supersede; task 04 (client lazy drill + `totals` cards) and task 06 (E2E gate) repair it. Unit/integration suites (the project's `testpaths`) are 100% green; sources.js pins stay green untouched.
|
||||
|
||||
Next pending task: `167_api_payload_diet/03_tree_node_endpoint_etag.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 207 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 273 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 71 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5981 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
## Report — phase 167, task 03 (`167_api_payload_diet/03_tree_node_endpoint_etag`)
|
||||
|
||||
**Implemented**
|
||||
- `app/core/http_cache.py` (new, 100% covered): `weak_etag` (`W/"sha256[:32]"`, str≡bytes), tolerant `etag_matches` (`*`, comma-lists, weak↔strong both directions, whitespace, `None`/empty→False), `revalidate_headers` (one place for `ETag` + `Cache-Control: private, must-revalidate`; docstrings name the A4/A19 scope).
|
||||
- `app/api/docs.py`: extracted `_kb_catalogue` (one catalogue query + the two bounded selects, shared by both tree routes — no N+1) and `_revalidated_json` (serialize once → tag → 304/200); `GET /api/docs/tree/node?source&path&depth` (admin gate, `path=""`→`KbTreeSource` via `_source_node`, else `KbTreeFolder` with `counts`/`max_dates`/summary/`summary_pending`; 404 `{"detail":"unknown node"}` incl. file-paths and traversal — row-only, no filesystem surface); ETag/304 wired on `/api/docs`, `/api/docs/tree`, `/api/docs/tree/node` (`response_model` kept for OpenAPI; node uses `response_model=None` + union annotation).
|
||||
- Tests: `tests/unit/test_http_cache.py` (12 pins) + 10 new integration tests in `tests/integration/test_docs_api.py` (node shape/depth/404-matrix/403/422s, node≡`depth=8` tree level equality, full ETag matrix on all 3 routes, A4 bust after folder-summary + document-date PATCHes with `sources_meta.version` provably unmoved, gzip coexistence).
|
||||
|
||||
**Gates** — `uv run pytest` → **3941 passed**; `uv run pytest tests/integration/test_docs_api.py tests/unit/test_http_cache.py -v` → **71 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/core/http_cache.py` 100%, `app/api/docs.py` 99% (only the pre-existing defensive `continue`); `uv run ruff check .` clean; `uv run pyright` 0 errors; caching suites stay green (49 passed).
|
||||
|
||||
**Docs** — README updated (Sources section: node route + revalidation; Caching section: the read-only-API validator arm of A19/A4). AGENTS.md: no doc impact. No new env vars.
|
||||
|
||||
**Notable decisions**
|
||||
- The node-route node itself is never `truncated` (its direct children are always built — same rule as `KbTreeSource` at depth ≥1); deeper levels ride the children's flags.
|
||||
- `tests/e2e/test_kb_tree.py` has 2 failures (stat cards, 2nd-level drill) — **not introduced here**: they stem from task 02's `depth=1` default and are explicitly owned by task 04 (`sources.js` lazy drill + `totals`) and task 06 (E2E pin migration) per the phase plan; unit+integration (the project suite per `testpaths`) is 100% green.
|
||||
- ASSUMPTION kept per task text: `/api/documents/content` gets no validator this phase (task 05's audit decides).
|
||||
|
||||
**Next pending task:** `167_api_payload_diet/04_client_lazy_drill.md`
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
..................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 234 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 273 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 71 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6030 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -0,0 +1,15 @@
|
||||
**Task 04 complete — client lazy drill (phase 167 D4/A3).**
|
||||
|
||||
- `frontend/assets/sources.js`: load is now ONE `fetch("/api/docs/tree?depth=1")`; `levelCache` (Map, `source + "/" + folder`) cleared on every fresh tree; `needsLevelFetch` + `enterFolder`/`fetchLevel` fetch `GET /api/docs/tree/node?…` only for uncached `truncated` nodes, `Object.assign`-merge + cache only after a 200; `drillPending` double-click guard with `aria-busy`/`aria-disabled` on the row link; `role="status"` "Loading <folder>…" row (departing level stays rendered) and `role="alert"` "Couldn't load … — Retry" with a real `<button type="button">`; `loadSeq` extended to the drill (+ `navSeq` so navigation invalidates it, never a stale paint); node 404 → honest top-level reset; `folderExistsIn` → `folderKnownIn` (ancestor-chain over held levels); `treeStats()` prefers server `totals` (walk kept as fallback); popstate adopts rehydrate via `ensureCurrentLevel` (fixed the forward-nav E2E regression); module docstring rewritten for the depth-1 + per-level model.
|
||||
- `frontend/assets/styles.css`: one tokens-only `.kb-level-loading` rule (Retry reuses the existing `.retry-btn` pill).
|
||||
- Tests: migrated `tests/unit/test_kb_tree_ui.py` (depth=1/node fetch counts, levelCache lifecycle, race token, aria states, status/alert rows, 404 reset, totals-first stats, folderKnownIn, docstring pins — all phase-97/99 pins preserved) and `tests/unit/test_frontend_router.py:591`; migrated `tests/e2e/test_navbar_refresh.py` request-pin to `?depth=1`.
|
||||
|
||||
Results:
|
||||
- `uv run pytest` → **3948 passed**; `--cov=app` → **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
|
||||
- E2E in isolation, all green: `test_kb_tree.py` (8), `test_kb_tree_nav.py` (4, incl. the back/forward walk), `test_navbar_refresh.py`, `test_document_dates.py`, `test_git_source_dates.py`, `test_sync_summary_visibility.py`, `test_table_ux_polish.py` (26), `test_sources_midstream_bug.py` + `test_ls_tree_drilldown.py` + `test_smoke.py` (12).
|
||||
|
||||
Docs: README RAG-view paragraph extended (per-level cache, Loading/Retry states, 404 reset); no AGENTS.md impact.
|
||||
|
||||
Notable: added `navSeq` + `ensureCurrentLevel` beyond the task sketch — popstate/forward adopts of a not-yet-refetched level would otherwise render a half-empty level (`test_kb_tree_nav.py::test_back_button_walks_the_breadcrumb` caught it; now green).
|
||||
|
||||
Next pending task: `167_api_payload_diet/05_response_budget_audit.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
............................................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 234 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 273 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 71 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6030 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
All green. Final state verified.
|
||||
|
||||
**Implemented (task 05)**
|
||||
- `app/core/response_budget.py` — pure-ASGI byte-budget middleware: one `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per `/api/*` response (`stream=1` for `text/event-stream`), INFO at/over budget, DEBUG below, `304`→`bytes=0`, non-`/api/`+non-HTTP silent, exceptions logged and re-raised unchanged; `bytes=` = route bytes, `enc=` = wire encoding (reads the shared start-message the gzip layer decorates in place — documented, integration-pinned).
|
||||
- `Settings.log_large_response_bytes=65536` + `agent_max_rounds`-style validator (`0`=all INFO, negative refuses startup); `.env.example` entry; `app/main.py` wires it **before** the gzip call → SecurityHeaders → gzip → response-budget → caching → session.
|
||||
- `scripts/api_payload_audit.py` (stdlib+httpx, read-only, exit 0 report / 2 unreachable-or-bad-login) + `AUDIT.md` with verbatim BEFORE (`BOR_COMPRESS_MIN_BYTES=0`) and AFTER tables, curl proof, real log lines incl. `POST /api/chat … stream=1`, and a verdict per endpoint.
|
||||
- Tests: new `tests/unit/test_response_budget.py` (24), new `tests/integration/test_response_budget_api.py` (7), one SSE `stream=1` pin in `tests/integration/test_chat_api.py`; migrated the two `test_compression.py` middleware-stack pins.
|
||||
|
||||
**Live-infra used (local dev only)**: restored the 1,000-doc live replica via the `restore-test-db` skill (safety backup `/tmp/pre_restore_20261008_073453.dump`; fingerprint `docs=1000 chunks=8866`; alembic 0011→0027), ran the dev server for the audit, then stopped it. Key measured numbers: tree depth=1 **8,474 B → 1,129 B gzip → 304/0** (full depth-8 tree 265,147 B); `/api/docs` 246,366→45,107; `/api/documents/content` 892,124→60,776 (LOOK, accepted with why). AUDIT.md states the snapshot has **0 folder summaries / 0 image docs**, so identity bytes are a lower bound on the 2 MB production report.
|
||||
|
||||
**Docs**: README "Operational notes: the response byte budget" (log format, `grep ' :: api '`, `BOR_LOG_LARGE_RESPONSE_BYTES`, audit run recipe, self-gzip/nginx deploy note) + config-reference row; AGENTS.md rule 10 + quick-reference line.
|
||||
|
||||
**Gates (exact)**: `uv run pytest --cov=app --cov-report=term-missing` → **3980 passed**, TOTAL 99%, `app/core/response_budget.py` **100%**; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors; `uv run pytest tests/e2e/test_smoke.py -q --no-cov` → green.
|
||||
|
||||
**Deviations**: no `--username` flag (the single-admin `/api/login` is password-only, A10 — documented in the script); no new ETag remedies applied — `/api/git-sources` measures 188 B and `/api/documents/content` is the task's documented accepted case with a named follow-up.
|
||||
|
||||
**Next pending task**: `167_api_payload_diet/06_e2e_payload_diet.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 234 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 280 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 73 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6102 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -0,0 +1,14 @@
|
||||
Final verification pass complete. Everything is green.
|
||||
|
||||
**Phase 167 task 06 — report**
|
||||
|
||||
1. **New suite** `tests/e2e/test_kb_tree_lazy_load.py` — **11 passed in isolation**: one `?depth=1` boot request (zero node fetches, parsed body: `totals` 8 docs/12 chunks, recursive source counts, no node deeper than level 1, every level-1 folder `truncated: true`, D9 recursive date on the truncated `alpha`); gzip + `Vary: Accept-Encoding` + `W/"…"` ETag on the real Chromium negotiation; one node fetch per uncached level (source click = zero); two levels deep then back = zero new fetches, cache never re-fetched; D4 pending marker + image thumbnail + phase-156 sticky header/scroll panel/44 px Retry + 72rem container in a lazily rendered level; `role="status"`/"Loading alpha…" in-flight state where a second click issues no second request; 500 → alert + Retry with the level preserved, Retry recovers; 404 → honest reset to the top level.
|
||||
2. **Revalidation**: Chromium hands `fetch` the cached 200, so reload and History→Sources revalidation are pinned on the CDP Network events (DevTools-panel truth): `If-None-Match` sent, wire `304`, transfer < ½ the first load, table still renders; plus the never-stale half — a folder-description PATCH busts the tag → `200` + the new text renders.
|
||||
3. **Migrated, never deleted**: the superseded "ONE fetch / drill is client-side" claims re-pointed in `test_kb_tree.py` (module contract + `_wait_top_level` + `_drill` + the breadcrumb-back pin) and in 8 sibling suites' drill helpers (`test_kb_tree_nav`, `test_admin_auth`, `test_folder_summary_voice`, `test_sync_summary_visibility`, `test_import_documents`, `test_extensionless_import`, `test_import_extensions_env`, `test_quadlet_jinja_import`). All re-ran green in isolation.
|
||||
4. **Regression set, isolation**: kb_tree 8, kb_tree_nav 4, navbar_refresh 7, document_dates 6, git_source_dates 4, sync_summary_visibility 3, table_ux_polish 6, sources_midstream_bug 6, ls_tree_drilldown 3 (agent `ls` unchanged), smoke 3 — all pass.
|
||||
5. **Full gates**: `uv run pytest` **3980 passed**; `--cov=app` **TOTAL 99 %** (`compression.py`, `http_cache.py`, `response_budget.py` 100 %, `api/docs.py` 99 %); `ruff check .` and `pyright` clean.
|
||||
6. **Retrieval battery**: no `app/rag/**` file is touched by this phase (`git diff --name-only`), so retrieval is unchanged by construction; the battery itself cannot measure here — the shared E2E Postgres holds no corpus (the suites truncate it), so every question deflects at `best_cosine=0.0000`.
|
||||
7. **Defect found and fixed** (pre-existing, not phase 167): `test_sources_midstream_bug.py::_reset_db` truncated only `chunks, documents, query_log`, so a `kb_overview` row left by another suite's sync leaked into the prompt and the mock echoed `(kb: …)` into the answer, breaking its exact-text pin. It now truncates the same house list as `test_kb_tree._truncate_all`; verified failing before, 6/6 after, including directly after `test_kb_tree`. 27 other suites still truncate narrowly — same latent order coupling, left alone as out of scope.
|
||||
8. **Deviation flagged**: task 06's vanished-folder bullet says a `404` shows alert + Retry, but `00_phase.md` D4/D6 and the shipped client make the node route's 404 the vanished signal (reset to the top level). Both paths are pinned honestly — 404 → honest reset, 500 → alert + Retry with the level preserved — no assertion weakened.
|
||||
9. **Docs**: `AUDIT.md` gained the browser-verification paragraph (gzip, 304, one fetch per level); README's tree/node endpoint table, gzip + ETag deploy note, env var list and `AGENTS.md` rule 10 / quick-reference were reviewed and are already phase-167 consistent; `.env.example` carries both new vars.
|
||||
10. No git index/history commands run, no `todo/` phase file moved, renamed or edited.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 277 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 234 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 280 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 73 0 100%
|
||||
app/models.py 136 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 454 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 841 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 156 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 505 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6102 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
Reference in New Issue
Block a user