diff --git a/.agents/phases/todo/167_api_payload_diet/00_phase.md b/.agents/phases/complete/167_api_payload_diet/00_phase.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/00_phase.md rename to .agents/phases/complete/167_api_payload_diet/00_phase.md diff --git a/.agents/phases/todo/167_api_payload_diet/01_gzip_middleware.md b/.agents/phases/complete/167_api_payload_diet/01_gzip_middleware.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/01_gzip_middleware.md rename to .agents/phases/complete/167_api_payload_diet/01_gzip_middleware.md diff --git a/.agents/phases/todo/167_api_payload_diet/02_tree_depth_and_totals.md b/.agents/phases/complete/167_api_payload_diet/02_tree_depth_and_totals.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/02_tree_depth_and_totals.md rename to .agents/phases/complete/167_api_payload_diet/02_tree_depth_and_totals.md diff --git a/.agents/phases/todo/167_api_payload_diet/03_tree_node_endpoint_etag.md b/.agents/phases/complete/167_api_payload_diet/03_tree_node_endpoint_etag.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/03_tree_node_endpoint_etag.md rename to .agents/phases/complete/167_api_payload_diet/03_tree_node_endpoint_etag.md diff --git a/.agents/phases/todo/167_api_payload_diet/04_client_lazy_drill.md b/.agents/phases/complete/167_api_payload_diet/04_client_lazy_drill.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/04_client_lazy_drill.md rename to .agents/phases/complete/167_api_payload_diet/04_client_lazy_drill.md diff --git a/.agents/phases/todo/167_api_payload_diet/05_response_budget_audit.md b/.agents/phases/complete/167_api_payload_diet/05_response_budget_audit.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/05_response_budget_audit.md rename to .agents/phases/complete/167_api_payload_diet/05_response_budget_audit.md diff --git a/.agents/phases/todo/167_api_payload_diet/06_e2e_payload_diet.md b/.agents/phases/complete/167_api_payload_diet/06_e2e_payload_diet.md similarity index 100% rename from .agents/phases/todo/167_api_payload_diet/06_e2e_payload_diet.md rename to .agents/phases/complete/167_api_payload_diet/06_e2e_payload_diet.md diff --git a/.agents/phases/todo/167_api_payload_diet/AUDIT.md b/.agents/phases/todo/167_api_payload_diet/AUDIT.md new file mode 100644 index 0000000..849a0d9 --- /dev/null +++ b/.agents/phases/todo/167_api_payload_diet/AUDIT.md @@ -0,0 +1,236 @@ +# Phase 167 · Task 05 — Measured API payload audit (the byte budget) + +Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any +other endpoints** that could load a ton of data." This file is the measured answer — +every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` / +server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a +report, never a 413 — nothing was rejected, nothing was capped. + +## How it was run + +| | | +|---|---| +| Date | 2026-10-08 | +| Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) | +| App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) | +| BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had | +| AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) | +| Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) | +| Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below | + +The script is **read-only**: it signs in through `POST /api/login` (the session +cookie — the single-admin login is password-only, PLAN A10, so there is no +`--username`) and GETs. It never mutates, so it is safe against a production target +(`--base-url https://…`). + +### Corpus caveat (read this before comparing with the 2 MB report) + +The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents** +(verified: `select count(*) from folder_summaries` → `0`, +`select count(*) from documents where is_image` → `0`). The production 2 MB payload +was 1,000 documents **plus** the hundreds of inlined AI folder descriptions +(phase 127/155) and the image documents' full vision descriptions (phase 122/124) — +none of which exist in the snapshot. **So the identity bytes below are a lower bound +on the production payload**, and what transfers from this measurement is the +*structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the +ONE level the RAG view renders instead of the whole recursion below it — and every +level of a summary-carrying folder is where the production payload's fat lives), +gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1 +tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict +table below names the live re-run as the follow-up. + +## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available + +> Read the BEFORE table's `gzip` column as "no compressor existed": ratio +> `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts +> are NOT a pre-phase property — the conditional re-requests hit the validators +> tasks 01-04 added, which this run left in place because only the transport was +> being switched off (the honest way to measure "the same bytes, uncompressed"). +> Pre-phase there were no validators at all, so every repeat load cost the +> identity column again. + +``` +api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold) +api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json + endpoint identity gzip ratio ms verdict + ---------------------------------------------------------------------------------------------------------------- + tree depth=1 (default) 8,474 8,474 1.000 12 304 + tree depth=8 (full) 265,147 265,147 1.000 37 304 + tree node homelab/active 20,844 20,844 1.000 11 304 + docs catalogue 246,366 246,366 1.000 14 304 + document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 892,124 1.000 8 LOOK (> 262144) + note: content field 772573 chars + saved chats list 12 12 1.000 1 ok + saved chat (one) — — — — n/a + note: no saved chats + git sources 188 188 1.000 1 ok + suggestions 74 74 1.000 1 ok + sync status (polled) 272 272 1.000 1 ok + tokens 13 13 1.000 2 ok + config 188 188 1.000 1 ok + whoami 37 37 1.000 1 ok +api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports. +``` + +What the pre-phase browser actually paid for the **Sources page first paint** is the +sum of the rows it fetches on load, and the pre-phase tree route answered the FULL +recursive tree by default (`tree depth=8 (full)` is that shape): +`265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app +published **no validators at all**, so every repeat load re-paid the same 512 KB. + +## AFTER — shipped defaults (depth-1 default + gzip + ETag) + +api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold) +api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json + endpoint identity gzip ratio ms verdict + ---------------------------------------------------------------------------------------------------------------- + tree depth=1 (default) 8,474 1,129 0.133 13 304 + tree depth=8 (full) 265,147 23,520 0.089 41 304 + tree node homelab/active 20,844 1,064 0.051 11 304 + docs catalogue 246,366 45,107 0.183 23 304 + document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 60,776 0.068 14 LOOK (> 262144) + note: content field 772573 chars + saved chats list 12 12 1.000 2 ok + saved chat (one) — — — — n/a + note: no saved chats + git sources 188 188 1.000 1 ok + suggestions 74 74 1.000 2 ok + sync status (polled) 272 272 1.000 1 ok + tokens 13 13 1.000 2 ok + config 188 188 1.000 1 ok + whoami 37 37 1.000 1 ok +api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports. +``` + +Sources-page first paint, same sum against the AFTER gzip column: +`1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity, +and every one of those five rows revalidates to **304 / 0 bytes** on the next load). +The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×** +(31× from the depth limit × 7.5× from gzip), and the second visit is free. + +## `curl` proof on the audited corpus + +``` +$ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree + etag: W/"53c86d5d554efd8487ad52ca6cd6f913" + cache-control: private, must-revalidate + content-length: 8474 + vary: Cookie, Accept-Encoding + 8474 bytes # wc -c b1.json + +$ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree + etag: W/"53c86d5d554efd8487ad52ca6cd6f913" # the validator survives the encoding + content-encoding: gzip + 1129 bytes # wc -c b2.gz + +$ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \ + -w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree +status=304 size=0 +``` + +## The byte-budget log line (the answer to "which endpoint is heavy, right now") + +Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO); +the shipped default is `65536`, so at rest only the big ones reach the log. +`grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what +went on the wire — the two together are the "how fat, and how fat did the client see +it" pair. + +``` +INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14 +INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12 +INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11 +INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34 +INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40 +INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10 +INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22 +INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12 +INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0 +INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0 +``` + +`POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded +payload to budget — and it is skipped by the gzip layer by exact path (which is why +its line reads `enc=identity`). One real turn against this corpus, measured: + +``` +INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1 +``` + +39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture +ended because the measuring `curl --max-time 45` hung up — which is itself the +other half of the contract: a stream the client walks away from still writes +exactly ONE line, with the bytes that actually got out, never one line per frame +and never a missing line). Pinned by +`tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out` +and by the integration pin +`tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`. + +## Verdict per candidate endpoint + +| Endpoint | Measured (identity / wire) | Verdict | +|---|---|---| +| `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. | +| `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). | +| `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. | +| `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). | +| `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). | +| `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. | +| `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. | +| `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). | +| `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). | +| `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. | +| `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). | +| `GET /api/config` | 188 | **ok** — display strings only. | +| `GET /api/whoami` | 37 | **ok** — two fields. | +| `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. | + +## What the audit changed, and what it proved + +* **No new remedy was needed beyond tasks 01-04.** The only over-budget row + (`/api/documents/content`) is the documented accepted case, and + `/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag + was **not** applied — a validator on a sub-860-byte response costs a hash + and saves nothing (it would not even compress). +* **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06) + drives the real Sources view in Chromium and pins what this file measured: the + boot is ONE `?depth=1` request whose body carries `totals` and no deep children, + the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary` + (Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the + wire answers `304` for a fraction of the first transfer while the table still + renders from the cache, and a level the payload only names costs exactly one + `/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands + `fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP + Network events — the DevTools-panel truth, not Playwright's post-cache view. +* **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on + the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the + second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat + load free. Had the phase shipped gzip alone, the Sources page would still + have paid 23 KB/request forever, and on the summary-inflated production + corpus proportionally more. +* **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the + LOOK bar, already validated + compressed). The byte-budget line at its + default `65536` threshold logs it at INFO on every uncached load, so the + corpus growth that pushes it over shows up in the log before it shows up + for the owner. +* **Two limits of this measurement, stated instead of hidden:** (1) the + snapshot has no folder summaries and no image docs, so the identity + numbers are a lower bound on the production payload (see the caveat + above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather + than measured — the script is read-only by design (it must stay safe + against the deployed app), so it does not invent a chat to size. Both are + re-measurable with one command against the live corpus. + +## Re-running this audit + +```bash +# 1. the corpus (destructive to the local DB — see the skill's own backup step) +# restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese +uv run alembic upgrade head # the snapshot carries an old alembic version +# 2. the app +uv run uvicorn app.main:app --port 8000 +# 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override) +uv run python -m scripts.api_payload_audit --base-url http://localhost:8000 +# the BEFORE transport, for a before/after pair like the tables above: +BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit +``` diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.err b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md new file mode 100644 index 0000000..2fd7c37 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md @@ -0,0 +1,22 @@ +All gates green. Final report: + +**Verified (phase 167 final pass — all 6 tasks already in `complete/`)** +- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte. +- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`. + +**Gates (all 100% green)** +- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line). +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. +- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line). + +**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines): +- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed. +- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree). +- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere. + +**Notable decisions / deviations** +- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's. +- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened. +- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run. + +**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits. diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md.status b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.validate b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.validate new file mode 100644 index 0000000..dd2ba67 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__00_phase.a1.validate @@ -0,0 +1,130 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 41%] +........................................................................ [ 43%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +.................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 234 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 280 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 73 0 100% +app/models.py 136 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 841 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 505 0 100% +-------------------------------------------------- +TOTAL 6102 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.err b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md new file mode 100644 index 0000000..8a47dda --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md @@ -0,0 +1,19 @@ +**Task 01 complete — all green.** + +**Implemented** +- `app/core/compression.py`: `SelectiveGZipMiddleware` (pure ASGI, composes Starlette `GZipMiddleware`, `DEFAULT_SKIP_PATHS=("/api/chat",)` EXACT match — `/api/chat-images` still compresses) + `_RebufferSmallResponses` framing seam +- `app/config.py`: `compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`) with `agent_max_rounds`-pattern validator (`0` = off, `-1` refuses startup naming the field) +- `app/main.py`: wired between `configure_caching` and `SecurityHeadersMiddleware` (A20 stays outermost); `0` adds nothing at all; `.env.example` entry with the three-value comment +- `tests/unit/test_compression.py` (19 tests: gzip/identity/floor/pre-encoded skip, SSE frame-by-frame raw-ASGI interleaving proof + `list(stream)` idiom, exact-match pin, lifespan/websocket pass-through, wiring order, kill switch); `tests/integration/test_docs_api.py` + 4 pins (tree gzip→decompress-equal, A20 headers on gzipped body, small `/api/health` identity, A19 page contract through gzip) + +**Gates** +- `uv run pytest` → **3903 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **99%** overall, `app/core/compression.py` **100%** +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors +- E2E isolated (exit 0): `test_smoke`, `test_chat_rag`, `test_cache_busting`, `test_asset_cache_revalidation`, `test_kb_tree` +- Measured: 50-doc tree 9,708 → 571 bytes (**17×**), `Vary: … Accept-Encoding` present + +**Docs**: README.md — gzip bullet in "Caching & deploys" + `BOR_COMPRESS_MIN_BYTES` config table row; AGENTS.md: no doc impact (no env/middleware content there). + +**Decisions**: (1) The `_RebufferSmallResponses` seam was required: starlette 1.x's `CachingMiddleware` (BaseHTTPMiddleware, below gzip per LOCKED order) re-frames every pass-through body (`more_body=True`), voiding Starlette's minimum-size exemption — `/api/health` came back gzip-inflated (33→71 B). The seam only re-delivers the buffered message shape Starlette's own exemption checks; no compression math forked. (2) Frame-by-frame proof is a deterministic raw-ASGI interleaving test (TestClient buffers bodies and cannot prove incremental arrival). + +**Next pending task**: `167_api_payload_diet/02_tree_depth_and_totals.md`. diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md.status b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.validate b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.validate new file mode 100644 index 0000000..577f0c3 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__01_gzip_middleware.a1.validate @@ -0,0 +1,127 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 60%] +........................................................................ [ 62%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 70%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 273 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 71 0 100% +app/models.py 136 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 841 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 487 0 100% +-------------------------------------------------- +TOTAL 5952 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.err b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md new file mode 100644 index 0000000..5e250bf --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md @@ -0,0 +1,15 @@ +**Task 02 complete — depth-limited KB tree + server totals (phase 167 D2, LOCKED A2).** + +Implemented: +- `app/schemas.py`: new `KbTreeTotals` (documents/chunks/last_indexed, ge=0-bounded); `KbTree.totals` (additive top-level key); `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `_drop_truncated_when_false` wrap serializer (the `_drop_image_fields` idiom) — untruncated nodes stay wire byte-identical; `KbTreeFile` untouched. +- `app/api/docs.py`: `_folder_counts` → `(folders, counts, max_dates)` (recursive max `created_at` map, same pass); `_level_children(..., depth)` — budget-exhausted subfolders ship `truncated=true`, `children=[]`, recursive count + `max_dates`-derived `updated_at` (D9-proof), `summary`, `summary_pending`; `build_kb_tree(..., depth=None)` (None = byte-identical unlimited); source never truncated at depth≥1; endpoint gains `depth: Query(1, ge=1, le=8)` + `_kb_totals()` computed from the same rows (no extra query). `GET /api/docs` / agent `ls` untouched. +- Tests: 11 new unit pins (depth matrix, byte-identity at None, truncated field set, `_subtree_max`↔`max_dates` cross-check at every depth, phase-94/98 invariants per depth, totals math/empty/validation, image affordance at depth 1); 4 new integration tests (default depth=1 + totals vs `/api/docs`, depth=3, 422 matrix, pre-phase wire-shape pin); migrated deep-walking pins to explicit `?depth=8` (`test_docs_api.py` ×6, `test_docs_api_dates.py` + new truncated-date test, `test_git_source_dates.py`, `test_document_dates.py`). +- Docs: README RAG section documents depth/truncated/totals; AGENTS.md: no command/env changes → untouched. + +Gates: `uv run pytest --cov=app` → **3919 passed, coverage 99%** (`app/api/docs.py` 99% — only the pre-existing defensive `continue` uncovered; `app/schemas.py` 100%); `uv run ruff check .` clean; `uv run pyright` 0 errors. + +Notable decisions: +- **Deviation (plan gap):** `tests/e2e/test_document_dates.py` also fetched the tree with no depth (its `_find_file` recurses) — same repair class as item 9's two suites; migrated with `?depth=8` too. +- **Expected interim E2E state:** client `sources.js` (task 04's scope) still walks the tree for stat cards/drill, so `test_kb_tree.py` stat/drill assertions now fail against the intentional depth=1 default (verified the failure is exactly `#stat-docs` under-counting, not a server defect). This is the owner-confirmed A3 supersede; task 04 (client lazy drill + `totals` cards) and task 06 (E2E gate) repair it. Unit/integration suites (the project's `testpaths`) are 100% green; sources.js pins stay green untouched. + +Next pending task: `167_api_payload_diet/03_tree_node_endpoint_etag.md`. diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md.status b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.validate b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.validate new file mode 100644 index 0000000..827795d --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__02_tree_depth_and_totals.a1.validate @@ -0,0 +1,127 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 60%] +........................................................................ [ 62%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 207 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 273 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 71 0 100% +app/models.py 136 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 841 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 505 0 100% +-------------------------------------------------- +TOTAL 5981 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.err b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md new file mode 100644 index 0000000..7f91e12 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md @@ -0,0 +1,17 @@ +## Report — phase 167, task 03 (`167_api_payload_diet/03_tree_node_endpoint_etag`) + +**Implemented** +- `app/core/http_cache.py` (new, 100% covered): `weak_etag` (`W/"sha256[:32]"`, str≡bytes), tolerant `etag_matches` (`*`, comma-lists, weak↔strong both directions, whitespace, `None`/empty→False), `revalidate_headers` (one place for `ETag` + `Cache-Control: private, must-revalidate`; docstrings name the A4/A19 scope). +- `app/api/docs.py`: extracted `_kb_catalogue` (one catalogue query + the two bounded selects, shared by both tree routes — no N+1) and `_revalidated_json` (serialize once → tag → 304/200); `GET /api/docs/tree/node?source&path&depth` (admin gate, `path=""`→`KbTreeSource` via `_source_node`, else `KbTreeFolder` with `counts`/`max_dates`/summary/`summary_pending`; 404 `{"detail":"unknown node"}` incl. file-paths and traversal — row-only, no filesystem surface); ETag/304 wired on `/api/docs`, `/api/docs/tree`, `/api/docs/tree/node` (`response_model` kept for OpenAPI; node uses `response_model=None` + union annotation). +- Tests: `tests/unit/test_http_cache.py` (12 pins) + 10 new integration tests in `tests/integration/test_docs_api.py` (node shape/depth/404-matrix/403/422s, node≡`depth=8` tree level equality, full ETag matrix on all 3 routes, A4 bust after folder-summary + document-date PATCHes with `sources_meta.version` provably unmoved, gzip coexistence). + +**Gates** — `uv run pytest` → **3941 passed**; `uv run pytest tests/integration/test_docs_api.py tests/unit/test_http_cache.py -v` → **71 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/core/http_cache.py` 100%, `app/api/docs.py` 99% (only the pre-existing defensive `continue`); `uv run ruff check .` clean; `uv run pyright` 0 errors; caching suites stay green (49 passed). + +**Docs** — README updated (Sources section: node route + revalidation; Caching section: the read-only-API validator arm of A19/A4). AGENTS.md: no doc impact. No new env vars. + +**Notable decisions** +- The node-route node itself is never `truncated` (its direct children are always built — same rule as `KbTreeSource` at depth ≥1); deeper levels ride the children's flags. +- `tests/e2e/test_kb_tree.py` has 2 failures (stat cards, 2nd-level drill) — **not introduced here**: they stem from task 02's `depth=1` default and are explicitly owned by task 04 (`sources.js` lazy drill + `totals`) and task 06 (E2E pin migration) per the phase plan; unit+integration (the project suite per `testpaths`) is 100% green. +- ASSUMPTION kept per task text: `/api/documents/content` gets no validator this phase (task 05's audit decides). + +**Next pending task:** `167_api_payload_diet/04_client_lazy_drill.md` diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md.status b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.validate b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.validate new file mode 100644 index 0000000..7b66d46 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__03_tree_node_endpoint_etag.a1.validate @@ -0,0 +1,128 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 43%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 60%] +........................................................................ [ 62%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 96%] +........................................................................ [ 98%] +..................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 234 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 273 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 71 0 100% +app/models.py 136 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 841 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 505 0 100% +-------------------------------------------------- +TOTAL 6030 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__04_client_lazy_drill.a1.err b/.agents/reports/167_api_payload_diet/167_api_payload_diet__04_client_lazy_drill.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/167_api_payload_diet/167_api_payload_diet__04_client_lazy_drill.a1.md b/.agents/reports/167_api_payload_diet/167_api_payload_diet__04_client_lazy_drill.a1.md new file mode 100644 index 0000000..c097e88 --- /dev/null +++ b/.agents/reports/167_api_payload_diet/167_api_payload_diet__04_client_lazy_drill.a1.md @@ -0,0 +1,15 @@ +**Task 04 complete — client lazy drill (phase 167 D4/A3).** + +- `frontend/assets/sources.js`: load is now ONE `fetch("/api/docs/tree?depth=1")`; `levelCache` (Map, `source + "/" + folder`) cleared on every fresh tree; `needsLevelFetch` + `enterFolder`/`fetchLevel` fetch `GET /api/docs/tree/node?…` only for uncached `truncated` nodes, `Object.assign`-merge + cache only after a 200; `drillPending` double-click guard with `aria-busy`/`aria-disabled` on the row link; `role="status"` "Loading …" row (departing level stays rendered) and `role="alert"` "Couldn't load … — Retry" with a real ` whose click re-runs the SAME + enterFolder fetch. Neither builder calls renderLevel (the current + level never blanks); both clear the previous drill row first (ONE + row at a time) and textContent only (the house rule).""" + js = _js() + loading = _fn(js, "showDrillLoading") + assert "clearDrillRow();" in loading + assert 'note.setAttribute("role", "status");' in loading + assert 'note.textContent = "Loading " + name + "…"' in loading + assert 'row.className = "kb-level-loading"' in loading + assert "foldersTbody.appendChild(row)" in loading, "the row lives in the folders panel" + assert "renderLevel" not in _code(loading), "the leaving level stays rendered" + fail = _fn(js, "showDrillFailure") + assert "clearDrillRow();" in fail + assert 'note.setAttribute("role", "alert");' in fail + assert '"Couldn\'t load " + target.folder.split("/").pop() + " — "' in fail + assert 'retry.type = "button";' in fail, "Retry is a REAL button" + assert 'retry.textContent = "Retry";' in fail + assert ( + "retry.addEventListener(\"click\", () => enterFolder(target, link, true));" in fail + ), "Retry re-runs the SAME drill fetch" + assert "renderLevel" not in _code(fail), "the failure keeps the current level rendered" + clear = _fn(js, "clearDrillRow") + assert "drillRowEl.remove();" in clear and "drillRowEl = null;" in clear + + +def test_vanished_node_resets_to_the_top_level() -> None: + """The node route's 404 IS the "the folder vanished" signal (task + 04, D4): the drill resets `current` to the TOP level and re-renders + (no stale breadcrumb), caches NOTHING for the failed level, and + never applies the vanished target (enterFolder returns on false). + A network/non-OK failure takes the alert arm instead — the current + level stays rendered (pinned above).""" + js = _js() + fetch = _fn(js, "fetchLevel") + assert "response.status === 404" in fetch, "the 404 is detected" + vanish = fetch[fetch.find("if (vanished) {") : fetch.find( + "if (response === null || !response.ok) {" + )] + assert "current = { source: null, folder: null };" in vanish, ( + "the 404 resets to the top level (the phase-97 never-stale contract)" + ) + assert "renderLevel();" in vanish, "the reset renders — no stale breadcrumb" + assert "return false;" in vanish, "the vanished target is never applied" + assert "levelCache" not in vanish, "the failing level is NOT cached" + enter = _fn(js, "enterFolder") + assert "if (!loaded) return;" in enter, "enterFolder applies only on a real merge" + + +def test_module_docstring_documents_the_lazy_drill() -> None: + """The house per-phase module-note convention: the Phase 167 + (task 04) section records the depth-1 load, the per-level node + fetch + cache, the §7.4 drill states, the extended race token, the + 404 reset and the totals-first stat cards — citing TODO L5.""" + js = _js() + header = js[: js.find("import { fetchIsAdmin }")] + assert "Phase 167 (task 04, D4/A3" in header + assert "TODO L5" in header, "the 2 MB payload defect is documented" + assert "LAZY DRILL" in header + assert "/api/docs/tree?depth=1" in header + assert "levelCache" in header + assert "loadSeq token EXTENDS" in header, "the race-token narrative stays accurate" + assert "folderKnownIn" in header, "the existence-rule change is documented" # ---------- the level rendering ---------- @@ -507,12 +707,39 @@ def test_render_clears_both_row_containers_before_filling() -> None: assert clear_docs < body.find(append), f"{append} must come after the clears" -# ---------- the stat cards: the whole-tree walk ---------- +# ---------- the stat cards: server totals, walk as the fallback ---------- + + +def test_stat_cards_prefer_the_server_totals() -> None: + """Phase 167 (task 04, D2/D4): a depth-limited tree CANNOT be + summed client-side, so treeStats() reads the server's `totals` + (documents / chunks / last_indexed) FIRST; the phase-97 recursive + walk stays only as the defensive no-`totals` fallback (a tree + served by a pre-phase process — the no-crash path). The stat-card + rendering (String(st.docs)/String(st.totalChunks)/fmtDate(st.last) + with the "–" null idiom) is UNCHANGED.""" + js = _js() + body = _fn(js, "treeStats") + totals_i = body.find("const totals = kbTree.totals;") + guard_i = body.find('if (totals && typeof totals === "object") {') + docs_i = body.find("docs: totals.documents,") + chunks_i = body.find("totalChunks: totals.chunks,") + last_i = body.find('last: totals.last_indexed || "",') + walk_i = body.find("const walk = (node) => {") + assert -1 < totals_i < guard_i < docs_i < chunks_i < last_i < walk_i, ( + "totals first, the recursive walk ONLY as the no-totals fallback" + ) + render = js[js.find("function renderLevel(") : js.find("function renderEmpty(")] + assert "const st = treeStats();" in render + assert 'st.last ? fmtDate(st.last) : "–"' in render, ( + "the exact fmtDate formatting call is unchanged at every level" + ) def test_stat_cards_walk_the_whole_tree() -> None: - """The KB-wide stat cards walk the WHOLE tree (every file node at - any depth): document count, chunks sum, max indexed_at (fmtDate + """The no-`totals` fallback walk (phase 97, kept defensive since + phase 167): walk the WHOLE held tree (every file node at any + depth): document count, chunks sum, max indexed_at (fmtDate reuse) — the values identical to the former flat walk. A 0-document registered source contributes nothing (and still lists its row — the ls invariant).""" @@ -577,21 +804,30 @@ def test_module_docstring_documents_the_semantic_change() -> None: def test_reset_to_top_when_the_location_vanished() -> None: """PLAN §7.4: after a re-fetch, if the current source is no longer - in the tree (unregistered/pruned) OR the current folder no longer - exists under it, `current` RESETS to the top level BEFORE rendering - — no stale breadcrumb, no stale block. The folder existence check - mirrors the phase-94 rule: some indexed path starts with - `folder + "/"` (the source root — \"\" — always exists).""" + in the tree (unregistered/pruned) OR the current folder is no + longer KNOWN under it, `current` RESETS to the top level BEFORE + rendering — no stale breadcrumb, no stale block. Phase 167 (task + 04, D4): the phase-94 full-tree path-prefix walk becomes + folderKnownIn — a folder is known iff its WHOLE ancestor chain + exists in the levels the client HOLDS (the depth-1 tree + the + drilled-and-merged levels; a broken chain — gone OR simply not + loaded that deep — resets honestly; the node route's 404 is the + drill path's own vanished-signal, pinned separately).""" js = _js() - body = js[js.find("function resetVanishedLocation()") : js.find("function folderExistsIn(")] + body = _fn(js, "resetVanishedLocation") assert "if (current.source === null) return;" in body, "the top level never vanishes" assert "kbTree.sources.find((s) => s.name === current.source)" in body assert body.count("current = { source: null, folder: null };") == 2, ( "both vanishing arms reset to the top level" ) - exists = js[js.find("function folderExistsIn(") : js.find("function currentLevelNode(")] - assert 'if (folderPath === "") return true;' in exists, "the source root always exists" - assert "p.startsWith(folderPath + \"/\")" in exists, "the phase-94 existence rule" + assert "folderKnownIn(src, current.folder)" in body, ( + "the reset consults KNOWN-ness (the held levels), never a full-tree walk" + ) + known = _fn(js, "folderKnownIn") + assert 'if (folderPath === "") return true;' in known, "the source root always exists" + assert "if (!next) return false;" in known, "a broken ancestor chain is not known" + assert "collect" not in _code(known), "the recursive path-collection walk is gone" + assert "folderExistsIn" not in js, "the old full-tree existence walker is gone" # The reset runs on the success path of the load, before the render # (pinned in the load-order test — here: it is defined and called). assert "resetVanishedLocation();" in js[js.find("async function loadTree()"):] @@ -736,6 +972,7 @@ def test_styles_carry_the_tree_classes_with_no_new_hue() -> None: ".kb-crumb-current", ".kb-crumb-sep", ".kb-level", + ".kb-level-loading", ".kb-folders-table", ".folder-link", ): diff --git a/tests/unit/test_response_budget.py b/tests/unit/test_response_budget.py new file mode 100644 index 0000000..7c9475d --- /dev/null +++ b/tests/unit/test_response_budget.py @@ -0,0 +1,702 @@ +"""Unit tests: ``app.core.response_budget`` — the byte-budget log line +(phase 167, task 05, D5, LOCKED A5). + +Pins the contract of the line, the level cut, and the pass-through rules +on a tiny ASGI app (the raw-ASGI interleaving harness of +``test_compression.py`` where ordering matters, ``TestClient`` where the +composed stack matters): + +* the EXACT shape ``api method=… path=… status=… bytes=… enc=… + duration_ms=…`` for a large 200 at INFO and a small 200 at DEBUG; +* a 304 logs ``bytes=0`` (the revalidation win stays visible); +* a non-``/api/`` path and a non-HTTP scope log NOTHING; +* a stream logs ONE line with ``stream=1``, ``bytes`` = the sum of the + chunks, emitted when the stream COMPLETES (never mid-flight) — and the + frames are never buffered; +* ``enc`` reads the WIRE encoding (``gzip`` through the real compression + layer, ``identity`` for a client that does not accept it) while + ``bytes`` stays the size the ROUTE produced; +* an app exception is logged (finally-style, bytes as counted) and + re-raised UNCHANGED; +* the config knob: ``0`` logs every ``/api/*`` response at INFO, a + negative value refuses startup naming the field, and the middleware sits + INSIDE ``SelectiveGZipMiddleware`` so ``bytes=`` is the route's size. +""" +from __future__ import annotations + +import asyncio +import gzip +import json +import logging +import re +from typing import Any + +import pytest +from fastapi.testclient import TestClient +from pydantic import ValidationError +from starlette.applications import Starlette +from starlette.responses import Response +from starlette.routing import Route +from starlette.types import Message, Receive, Scope, Send + +import app.core.response_budget as budget_mod +import app.main as main_mod +from app.config import Settings +from app.core.compression import SelectiveGZipMiddleware +from app.core.response_budget import ( + API_PATH_PREFIX, + UNKNOWN_STATUS, + ResponseBudgetMiddleware, +) + +#: One line, fully anchored — the shape README documents and the +#: ``grep ' :: api '`` workflow parses. +LINE_RE = re.compile( + r"^api method=(?P[A-Z]+) path=(?P\S+) status=(?P\d+) " + r"bytes=(?P\d+) enc=(?P\S+) duration_ms=(?P\d+)$" +) + +#: ~28 KB of tree-shaped JSON. +_BODY = json.dumps( + { + "sources": [ + { + "name": "src", + "documents": i, + "children": [{"path": f"d/{i}/{n}.md"} for n in range(20)], + } + for i in range(200) + ] + } +).encode() + +_SMALL_BODY = b'{"ok":true}' + +#: Four SSE-class frames, each comfortably above the gzip floor. +_FRAMES = [f"data: {'x' * 500}\n\n".encode() for _ in range(4)] + +_LOGGER_NAME = "app.response_budget" + + +def _settings(**kwargs: Any) -> Settings: + """Build Settings without reading a .env file (the test_config pattern).""" + kwargs.setdefault("_env_file", None) + return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime) + + +def _api_lines(caplog: pytest.LogCaptureFixture) -> list[tuple[int, str]]: + """``(levelno, message)`` for this module's logger only, in order.""" + return [ + (record.levelno, record.getMessage()) + for record in caplog.records + if record.name == _LOGGER_NAME + ] + + +class _Counter(logging.Handler): + """Counts emitted lines live, so an app can observe "has the line been + written yet?" from INSIDE the response — the ordering proof.""" + + def __init__(self) -> None: + super().__init__() + self.count = 0 + + def emit(self, record: logging.LogRecord) -> None: + self.count += 1 + + +def _http_scope(path: str, method: str = "GET") -> Scope: + return {"type": "http", "method": method, "path": path, "headers": []} + + +async def _receive() -> Message: + return {"type": "http.disconnect"} + + +async def _send(message: Message) -> None: + return None + + +def _raw_json_app(body: bytes, *, status: int = 200, media_type: str = "application/json") -> Any: + """The smallest ASGI app that answers one buffered body — the raw-ASGI + level is where the WIRE bytes (post-compression) are observable.""" + + async def app(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": status, + "headers": [(b"content-type", media_type.encode())], + } + ) + await send({"type": "http.response.body", "body": body, "more_body": False}) + + return app + + +def _raw_harness(middleware_app: Any, *, path: str, accept: str) -> list[Message]: + """Drive a middleware stack at the raw ASGI level; return the exact + messages that reached the client's ``send``.""" + sent: list[Message] = [] + + async def recording_send(message: Message) -> None: + sent.append(message) + + asyncio.run( + middleware_app( + { + "type": "http", + "method": "GET", + "path": path, + "headers": [(b"accept-encoding", accept.encode())], + }, + _receive, + recording_send, + ) + ) + return sent + + +def _inner_app( + *, + body: bytes = _BODY, + status: int = 200, + path: str = "/api/docs/tree", + media_type: str = "application/json", +) -> Starlette: + async def respond(request: Any) -> Response: + return Response(body, status_code=status, media_type=media_type) + + async def boom(request: Any) -> Response: + raise RuntimeError("kaboom") + + return Starlette( + routes=[ + Route(path, respond, methods=["GET", "POST"]), + Route("/api/boom", boom, methods=["GET"]), + Route("/sources.html", respond, methods=["GET"]), + ] + ) + + +def _client(app: Any, *, threshold: int = 1024) -> TestClient: + return TestClient(ResponseBudgetMiddleware(app, threshold=threshold)) + + +# --------------------------------------------------------------------------- +# The line's exact shape, and the level the budget decides +# --------------------------------------------------------------------------- + + +def test_large_response_logs_one_info_line_in_the_exact_shape( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + client = _client(_inner_app(), threshold=1024) + assert client.get("/api/docs/tree").status_code == 200 + + lines = _api_lines(caplog) + assert len(lines) == 1 + level, message = lines[0] + assert level == logging.INFO + match = LINE_RE.match(message) + assert match is not None, message + assert match.group("method") == "GET" + assert match.group("path") == "/api/docs/tree" + assert match.group("status") == "200" + assert int(match.group("bytes")) == len(_BODY) + assert match.group("enc") == "identity" + assert int(match.group("ms")) >= 0 + + +def test_small_response_logs_the_same_line_at_debug( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + client = _client(_inner_app(body=_SMALL_BODY), threshold=1024) + assert client.get("/api/docs/tree").status_code == 200 + + lines = _api_lines(caplog) + assert len(lines) == 1 + level, message = lines[0] + assert level == logging.DEBUG + assert f"bytes={len(_SMALL_BODY)}" in message + + +def test_budget_is_the_inclusive_cut(caplog: pytest.LogCaptureFixture) -> None: + """``bytes >= budget`` is INFO, one byte less is DEBUG — the boundary the + README and ``.env.example`` state.""" + + async def exactly(request: Any) -> Response: + return Response(b"x" * 100) + + async def one_less(request: Any) -> Response: + return Response(b"x" * 99) + + app = Starlette( + routes=[ + Route("/api/a", exactly, methods=["GET"]), + Route("/api/b", one_less, methods=["GET"]), + ] + ) + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + client = _client(app, threshold=100) + client.get("/api/a") + client.get("/api/b") + + assert [level for level, _ in _api_lines(caplog)] == [logging.INFO, logging.DEBUG] + + +def test_threshold_defaults_to_the_settings_value( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + """No ``threshold`` argument — the way ``create_app`` wires it — so the + live ``log_large_response_bytes`` decides per response.""" + monkeypatch.setattr( + budget_mod, "get_settings", lambda: _settings(log_large_response_bytes=5) + ) + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + TestClient(ResponseBudgetMiddleware(_inner_app(body=_SMALL_BODY))).get( + "/api/docs/tree" + ) + + assert [level for level, _ in _api_lines(caplog)] == [logging.INFO] + + +def test_zero_budget_logs_every_api_response_at_info( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + """``BOR_LOG_LARGE_RESPONSE_BYTES=0`` is the loud mode: even an 11-byte + body is INFO (the operator's one-off investigation setting).""" + monkeypatch.setattr( + budget_mod, "get_settings", lambda: _settings(log_large_response_bytes=0) + ) + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + TestClient(ResponseBudgetMiddleware(_inner_app(body=_SMALL_BODY))).get( + "/api/docs/tree" + ) + + # 11 bytes, budget 0 -> INFO (the same body is DEBUG at any positive + # budget: see test_small_response_logs_the_same_line_at_debug). + assert [level for level, _ in _api_lines(caplog)] == [logging.INFO] + + +# --------------------------------------------------------------------------- +# Statuses: a 304 logs 0 bytes, an error is counted like any other response +# --------------------------------------------------------------------------- + + +def test_304_logs_zero_bytes(caplog: pytest.LogCaptureFixture) -> None: + """The revalidation win stays in the log — a bodiless 304 is a line with + ``bytes=0``, not a missing line.""" + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + client = _client(_inner_app(body=b"", status=304), threshold=1024) + assert client.get("/api/docs/tree").status_code == 304 + + lines = _api_lines(caplog) + assert len(lines) == 1 + match = LINE_RE.match(lines[0][1]) + assert match is not None + assert match.group("status") == "304" + assert match.group("bytes") == "0" + assert lines[0][0] == logging.DEBUG + + +def test_error_status_is_logged_like_any_other_response( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + client = _client(_inner_app(body=b'{"detail":"admin only"}', status=403)) + assert client.get("/api/docs/tree").status_code == 403 + + match = LINE_RE.match(_api_lines(caplog)[0][1]) + assert match is not None and match.group("status") == "403" + + +# --------------------------------------------------------------------------- +# What is NOT measured +# --------------------------------------------------------------------------- + + +def test_non_api_path_logs_nothing(caplog: pytest.LogCaptureFixture) -> None: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + # budget 0: ANY /api line would show, so silence is the whole pin. + client = _client(_inner_app(), threshold=0) + assert client.get("/sources.html").status_code == 200 + + assert _api_lines(caplog) == [] + + +def test_non_http_scopes_pass_through_silently( + caplog: pytest.LogCaptureFixture, +) -> None: + """``lifespan`` / ``websocket`` scopes reach the app untouched and write + no line — there is no response to measure.""" + seen: list[str] = [] + + async def app(scope: Scope, receive: Receive, send: Send) -> None: + seen.append(scope["type"]) + + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + for scope_type in ("lifespan", "websocket"): + asyncio.run( + ResponseBudgetMiddleware(app, threshold=0)( + {"type": scope_type, "path": "/api/x"}, _receive, _send + ) + ) + + assert seen == ["lifespan", "websocket"] + assert _api_lines(caplog) == [] + + +# --------------------------------------------------------------------------- +# The stream: one line, stream=1, counted total, emitted at completion +# --------------------------------------------------------------------------- + + +def test_stream_logs_once_at_completion_with_stream_flag_and_total_bytes( + caplog: pytest.LogCaptureFixture, +) -> None: + """The SSE contract of the line: nothing while frames are in flight, one + ``stream=1`` line when the final chunk completes, ``bytes`` = the sum of + the frames (NOT one frame).""" + counter = _Counter() + logger = logging.getLogger(_LOGGER_NAME) + logger.addHandler(counter) + in_flight: list[int] = [] + + async def sse_app(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"text/event-stream")], + } + ) + for frame in _FRAMES: + await send( + {"type": "http.response.body", "body": frame, "more_body": True} + ) + in_flight.append(counter.count) # after each frame is delivered + await send({"type": "http.response.body", "body": b"", "more_body": False}) + in_flight.append(counter.count) + + try: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + asyncio.run( + ResponseBudgetMiddleware(sse_app, threshold=1)( + _http_scope("/api/chat", "POST"), _receive, _send + ) + ) + finally: + logger.removeHandler(counter) + + # Every in-flight probe except the last saw zero lines; the completion + # is what emitted. + assert in_flight == [0] * len(_FRAMES) + [1] + lines = _api_lines(caplog) + assert len(lines) == 1 + message = lines[0][1] + assert message.endswith(" stream=1") + match = LINE_RE.match(message.removesuffix(" stream=1")) + assert match is not None + assert match.group("method") == "POST" + assert match.group("path") == "/api/chat" + assert int(match.group("bytes")) == sum(len(frame) for frame in _FRAMES) + + +def test_stream_frames_are_never_buffered() -> None: + """Zero-buffering proof (the interleaving idiom of + ``test_compression.py``): every frame reaches the client's ``send`` + before the app produces the next one — a ``BaseHTTPMiddleware`` here + would emit all ``app:*`` events first.""" + events: list[str] = [] + + async def sse_app(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"text/event-stream")], + } + ) + for i, frame in enumerate(_FRAMES): + events.append(f"app:frame-{i}") + await send( + {"type": "http.response.body", "body": frame, "more_body": True} + ) + await send({"type": "http.response.body", "body": b"", "more_body": False}) + + async def recording_send(message: Message) -> None: + events.append(f"client:{message['type']}") + + asyncio.run( + ResponseBudgetMiddleware(sse_app, threshold=0)( + _http_scope("/api/chat", "POST"), _receive, recording_send + ) + ) + + expected = ["client:http.response.start"] + for i in range(len(_FRAMES)): + expected += [f"app:frame-{i}", "client:http.response.body"] + expected += ["client:http.response.body"] + assert events == expected + + +# --------------------------------------------------------------------------- +# enc= reads the wire, bytes= stays the route's size +# --------------------------------------------------------------------------- + + +def _stack(*, threshold: int = 1024) -> TestClient: + """response-budget INSIDE the selective-gzip layer — the order + ``create_app`` builds.""" + return TestClient( + SelectiveGZipMiddleware( + ResponseBudgetMiddleware(_inner_app(), threshold=threshold), + minimum_size=860, + ) + ) + + +def test_enc_gzip_is_reported_through_the_compression_layer( + caplog: pytest.LogCaptureFixture, +) -> None: + """The two halves of the line, measured against the wire: the client + receives a gzipped body, the log says ``enc=gzip`` and still counts the + ROUTE's uncompressed bytes (the position inside the compressor is what + buys that).""" + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + sent = _raw_harness( + SelectiveGZipMiddleware( + ResponseBudgetMiddleware(_raw_json_app(_BODY), threshold=1024), + minimum_size=860, + ), + path="/api/docs/tree", + accept="gzip", + ) + + wire = b"".join(m.get("body", b"") for m in sent if m["type"] == "http.response.body") + assert gzip.decompress(wire) == _BODY # the transfer really was gzip + match = LINE_RE.match(_api_lines(caplog)[-1][1]) + assert match is not None + assert match.group("enc") == "gzip" + assert int(match.group("bytes")) == len(_BODY) > len(wire) + + +def test_enc_identity_when_the_client_does_not_accept_gzip( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + response = _stack().get( + "/api/docs/tree", headers={"Accept-Encoding": "identity"} + ) + + assert "content-encoding" not in response.headers + match = LINE_RE.match(_api_lines(caplog)[-1][1]) + assert match is not None and match.group("enc") == "identity" + assert int(match.group("bytes")) == len(_BODY) + + +def test_enc_reports_a_pre_encoded_response(caplog: pytest.LogCaptureFixture) -> None: + """A route that encodes its own body (the fronting-proxy case) is + reported as such — the header on the wire is the answer.""" + + async def encoded(request: Any) -> Response: + return Response( + gzip.compress(_BODY), + media_type="application/json", + headers={"Content-Encoding": "gzip"}, + ) + + app = Starlette(routes=[Route("/api/encoded", encoded, methods=["GET"])]) + with caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME): + TestClient(ResponseBudgetMiddleware(app, threshold=1024)).get("/api/encoded") + + match = LINE_RE.match(_api_lines(caplog)[-1][1]) + assert match is not None and match.group("enc") == "gzip" + + +# --------------------------------------------------------------------------- +# The app raising: logged, and re-raised unchanged +# --------------------------------------------------------------------------- + + +def test_app_exception_logs_and_reraises_unchanged( + caplog: pytest.LogCaptureFixture, +) -> None: + """An app that dies before it can answer still writes one line (nothing + was counted, status 500 — what the outer error layer will answer) and the + exception propagates UNCHANGED.""" + + async def dying_app(scope: Scope, receive: Receive, send: Send) -> None: + raise RuntimeError("kaboom") + + with ( + caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME), + pytest.raises(RuntimeError, match="kaboom"), + ): + asyncio.run( + ResponseBudgetMiddleware(dying_app, threshold=0)( + _http_scope("/api/docs/tree"), _receive, _send + ) + ) + + lines = _api_lines(caplog) + assert len(lines) == 1 + match = LINE_RE.match(lines[0][1]) + assert match is not None + assert match.group("status") == str(UNKNOWN_STATUS) + assert match.group("bytes") == "0" + + +def test_a_500_produced_by_the_framework_is_counted_like_a_response( + caplog: pytest.LogCaptureFixture, +) -> None: + """Through a real app the unhandled exception becomes a 500 WITH a body + (Starlette's ``Internal Server Error`` text): the line then reports that + status and those bytes — one line either way, never zero.""" + with ( + caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME), + pytest.raises(RuntimeError, match="kaboom"), + ): + _client(_inner_app(), threshold=0).get("/api/boom") + + lines = _api_lines(caplog) + assert len(lines) == 1 + match = LINE_RE.match(lines[0][1]) + assert match is not None + assert match.group("status") == "500" + assert int(match.group("bytes")) == len(b"Internal Server Error") + + +def test_exception_mid_stream_logs_the_bytes_counted_so_far( + caplog: pytest.LogCaptureFixture, +) -> None: + """A stream that dies after two frames still writes ONE line with the + bytes it actually got out (finally-style) — and the exception is not + swallowed.""" + + async def dying_stream_app(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"text/event-stream")], + } + ) + await send({"type": "http.response.body", "body": _FRAMES[0], "more_body": True}) + await send({"type": "http.response.body", "body": _FRAMES[1], "more_body": True}) + raise RuntimeError("stream died") + + with ( + caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME), + pytest.raises(RuntimeError, match="stream died"), + ): + asyncio.run( + ResponseBudgetMiddleware(dying_stream_app, threshold=0)( + _http_scope("/api/chat", "POST"), _receive, _send + ) + ) + + lines = _api_lines(caplog) + assert len(lines) == 1 + message = lines[0][1] + assert message.endswith(" stream=1") + assert f"bytes={len(_FRAMES[0]) + len(_FRAMES[1])}" in message + + +def test_a_cancelled_stream_still_logs_one_line_with_what_got_out( + caplog: pytest.LogCaptureFixture, +) -> None: + """The client-walked-away path (the live audit hit it: an SSE turn cut + mid-answer): the app unwinds with ``CancelledError`` and no completing + body, the ``finally`` writes ONE line with the bytes that actually got + out, and the cancellation propagates (the middleware never swallows it).""" + + async def cancelled_stream_app(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"text/event-stream")], + } + ) + await send({"type": "http.response.body", "body": _FRAMES[0], "more_body": True}) + raise asyncio.CancelledError # what the server does on a client disconnect + + with ( + caplog.at_level(logging.DEBUG, logger=_LOGGER_NAME), + pytest.raises(asyncio.CancelledError), + ): + asyncio.run( + ResponseBudgetMiddleware(cancelled_stream_app, threshold=0)( + _http_scope("/api/chat", "POST"), _receive, _send + ) + ) + + lines = _api_lines(caplog) + assert len(lines) == 1 + assert lines[0][1].endswith(" stream=1") + assert f"bytes={len(_FRAMES[0])}" in lines[0][1] + + +# --------------------------------------------------------------------------- +# Config + wiring +# --------------------------------------------------------------------------- + + +def test_log_large_response_bytes_defaults_to_64_kb() -> None: + assert _settings().log_large_response_bytes == 65536 + + +def test_log_large_response_bytes_env_override( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("BOR_LOG_LARGE_RESPONSE_BYTES", "4096") + assert _settings().log_large_response_bytes == 4096 + monkeypatch.setenv("BOR_LOG_LARGE_RESPONSE_BYTES", "0") # loud mode is legal + assert _settings().log_large_response_bytes == 0 + + +def test_negative_budget_refuses_startup_naming_the_field() -> None: + with pytest.raises(ValidationError, match="log_large_response_bytes must be >= 0"): + _settings(log_large_response_bytes=-1) + + +def test_middleware_sits_inside_gzip_outside_caching( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The documented stack (README + ``app/main.py``): SecurityHeaders + outermost → gzip → response-budget → caching → session. That position is + what makes ``bytes=`` the ROUTE's size while ``enc=`` still names the + wire encoding.""" + monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 860) + names = [ + getattr(mw.cls, "__name__", "") for mw in main_mod.create_app().user_middleware + ] + assert names == [ + "SecurityHeadersMiddleware", + "SelectiveGZipMiddleware", + "ResponseBudgetMiddleware", + "CachingMiddleware", + "SessionMiddleware", + ] + + +def test_budget_is_wired_even_with_compression_off( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """``BOR_COMPRESS_MIN_BYTES=0`` removes gzip only — the budget line is + not optional (LOCKED A5: measure everything, reject nothing), so the + measurement survives the compression kill switch.""" + monkeypatch.setattr(main_mod.settings, "compress_min_bytes", 0) + names = [ + getattr(mw.cls, "__name__", "") for mw in main_mod.create_app().user_middleware + ] + assert "SelectiveGZipMiddleware" not in names + assert "ResponseBudgetMiddleware" in names + + +def test_api_path_prefix_constant() -> None: + assert API_PATH_PREFIX == "/api/"