Compare commits
52
Commits
eebc824636
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a45f744604 | ||
|
|
ae080ac7f3 | ||
|
|
fb17d15fbe | ||
|
|
4f20c256d7 | ||
|
|
c6ffa050a6 | ||
|
|
47456a44df | ||
|
|
140a343b92 | ||
|
|
f20d40e4fe | ||
|
|
c054b47df5 | ||
|
|
644f7de696 | ||
|
|
632fb88e43 | ||
|
|
065304b52f | ||
|
|
7b5ea7c2ee | ||
|
|
b34b8ca672 | ||
|
|
814a400e6d | ||
|
|
6880bd0866 | ||
|
|
3d82a72b6f | ||
|
|
2b57180998 | ||
|
|
76887639f3 | ||
|
|
50658580e9 | ||
|
|
dad54a2854 | ||
|
|
149e753952 | ||
|
|
c6223c93dc | ||
|
|
36eddce672 | ||
|
|
dd119d534a | ||
|
|
2a5df7cb31 | ||
|
|
bd271a6f52 | ||
|
|
375169bbfc | ||
|
|
938d6cf09d | ||
|
|
da441ea799 | ||
|
|
a86caec821 | ||
|
|
f27106a638 | ||
|
|
b21b05eaee | ||
|
|
e9ae6e2ba3 | ||
|
|
c4f098d0c0 | ||
|
|
1200437bf8 | ||
|
|
abe7388dda | ||
|
|
542e568d25 | ||
|
|
a88a5a3713 | ||
|
|
05f7d0645e | ||
|
|
8fac4e64d4 | ||
|
|
a12fdcdf0a | ||
|
|
a2b2f283fd | ||
|
|
2ac37455f0 | ||
|
|
ab9391d019 | ||
|
|
b0c7aed613 | ||
|
|
8d70aa946c | ||
|
|
dcb1062b83 | ||
|
|
34d60efbc1 | ||
|
|
6ba46e735a | ||
|
|
74eb2e265c | ||
|
|
855fa27a30 |
+35
-3
@@ -84,6 +84,25 @@ A15 — `thinking`, `tool`, unlimited tool rounds under
|
|||||||
never-truncated contract is unchanged; the `read`-tool cap is
|
never-truncated contract is unchanged; the `read`-tool cap is
|
||||||
the only exception, owner-permitted 2026-09-10.
|
the only exception, owner-permitted 2026-09-10.
|
||||||
|
|
||||||
|
Phase 182 (verbatim text display, owner chat 2026-10-14) adds
|
||||||
|
**`show_text(path, offset?, limit?)` as the fifth agent tool** — offered
|
||||||
|
only when `BOR_ANSWER_TEXT=1` (the default), gated exactly like
|
||||||
|
`show_image`, with its own `<tools>` clause and its own kill switch — and
|
||||||
|
the **additive `done.answer_text`** field above. The new settings are
|
||||||
|
`BOR_ANSWER_TEXT` (master switch, ON), `BOR_SHOW_TEXT_MAX_CHARS` (per-block
|
||||||
|
budget, 12 000) and `BOR_SHOW_TEXT_MAX_BLOCKS` (per-answer budget, 5), all
|
||||||
|
following the `agent_max_rounds` pattern (`0` inert, negative refuses
|
||||||
|
startup). The principle in one sentence: **the model points, the server
|
||||||
|
supplies the bytes** — a ` ```bor source/path 12-40 ` fence or a
|
||||||
|
`show_text` call names a location (canonical locator `source/path` or
|
||||||
|
`source/path#start-end`, 1-based and inclusive), and the displayed text is
|
||||||
|
sliced from `documents.content` server-side, so what the user copies is the
|
||||||
|
knowledge base, never a re-typed approximation. **A1–A6 and A15 are
|
||||||
|
otherwise
|
||||||
|
UNCHANGED:** no new SSE event type, no new table or column, no new model,
|
||||||
|
endpoint or transport, no change to retrieval, the honesty gate, the
|
||||||
|
persona text or the embedding dimension.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. High-Level Architecture
|
## 3. High-Level Architecture
|
||||||
@@ -129,7 +148,9 @@ POST /api/chat {message, history?} (auth: require_user)
|
|||||||
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
||||||
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
||||||
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
||||||
│ + <tools> → agent loop (ls/read/grep; round-capped
|
│ + <tools> → agent loop (ls/read/grep, plus show_image when
|
||||||
|
│ BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
|
||||||
|
│ "show the user an artifact" tools, gated independently; round-capped
|
||||||
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
|
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
|
||||||
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
|
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
|
||||||
byte-identical direct-stream path) → turbo streamed
|
byte-identical direct-stream path) → turbo streamed
|
||||||
@@ -139,7 +160,15 @@ POST /api/chat {message, history?} (auth: require_user)
|
|||||||
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
|
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
|
||||||
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
|
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
|
||||||
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
|
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
|
||||||
suggestions[], budget_exhausted?}` (terminal). Failure: `error` (terminal — no `done`, no
|
suggestions[], budget_exhausted?, answer_text[]?}` (terminal). **Phase 182
|
||||||
|
extends the `done` frame ADDITIVELY** — `answer_text` is the list of
|
||||||
|
server-resolved verbatim blocks (`locator`, `source`, `path`, `start`,
|
||||||
|
`end`, `total_lines`, `text`, `format`, `truncated`) the answer displayed;
|
||||||
|
there is **no eighth event type**, the key is OMITTED when no block was
|
||||||
|
displayed, and with `BOR_ANSWER_TEXT=0` it is absent on every turn (the
|
||||||
|
frame is then byte-identical to pre-phase). The same array persists on the
|
||||||
|
saved chat record, so restore and the anonymous shared page re-render the
|
||||||
|
panels from the server's bytes. Failure: `error` (terminal — no `done`, no
|
||||||
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
|
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
|
||||||
`tools=None` request never emits a `tool` frame; a round-capped turn
|
`tools=None` request never emits a `tool` frame; a round-capped turn
|
||||||
whose final reply carries no visible content answers with the budget
|
whose final reply carries no visible content answers with the budget
|
||||||
@@ -273,8 +302,11 @@ scripts.…`, and tests all honor it.
|
|||||||
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
|
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
|
||||||
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
|
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
|
||||||
thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
|
thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
|
||||||
retries=N scaffold_stripped=N
|
retries=N scaffold_stripped=N verbatim_blocks=N
|
||||||
```
|
```
|
||||||
|
(`verbatim_blocks=` — phase 182 — is the count of verbatim display blocks
|
||||||
|
actually SENT on the `done` frame, `0` on every turn without them,
|
||||||
|
deflected included.)
|
||||||
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
|
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
|
||||||
executed only; a cancelled turn writes neither line nor `query_log`
|
executed only; a cancelled turn writes neither line nor `query_log`
|
||||||
row.)
|
row.)
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# Phase 163 — PDF page strip: additive text+pages view, write-first atomic copies, deterministic fast-path re-gate (self-heal)
|
||||||
|
|
||||||
|
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, the document modal for `Letumdei_ Heirs to the Imperium/Letumdei_ Heirs to the Imperium.pdf` (doc id `b72f5515-dc40-4167-98ff-179a6503e6e5`). Every page rendered "Page N unavailable." and the document was unreadable in the viewer. **Pre-fix evidence (measured live, 2026-10-07):** the payload carries 36 `page_urls` (so `page_count = 36`); `GET /api/documents/<id>/pages/1…36` **all 404**; the content is genuine extracted text (114 389 chars, 36 `---PAGE-N---` markers, exactly ONE `[Page 36 illustration: …]` block — a map page under `pdf_pictorial_min_chars`); the same server serves the 10/3-indexed IKEA manual's pages 200 (the pages dir was NOT wiped — only this doc's copies are gone). Two stacked defects:
|
||||||
|
|
||||||
|
1. **Eligibility consequence (the "shouldn't be a pictorial pdf" complaint).** The phase-140 gate is per-page (`len(text.strip()) < pdf_pictorial_min_chars`, default 100); ONE sparse page in a text-heavy 36-page book set `pictorial_described = True` → the doc joined the strip class → the phase-148 viewer renders the strip **INSTEAD OF** the text content and the Summary panel. The owner's call (2026-10-07): **additive strip** — a usable-layer (pictorial-class) doc shows its TEXT (plus the Summary panel, parity with any text doc) with a **Text/Pages toggle** for the strip; the text is never hidden. The fallback class (unusable text layer — the phase-133 screenshots ARE the content) keeps today's strip-only view, DOM byte-identical.
|
||||||
|
2. **Lost copies, stale `page_count`, no self-heal.** `_persist_pdf_pages` **deletes the old copies from disk first**, and the filesystem work is NOT transactional with the DB session: any later rollback of the file's transaction (the per-file `EmbeddingError` handler, a killed sync) — or an external wipe of `pdf_page_dir` — leaves the row claiming `page_count = 36` over ZERO files. The phase-147 fast path ("the copies persist") never re-renders, so the strip 404s forever. The owner's call: **write-before-delete + fast-path re-gate** — the changed path writes the new copies (atomically, per file) BEFORE any deletion; the unchanged paths deterministically re-gate the strip (no LLM/vision call) — re-stamp the class, re-render missing copies, or clear a strip that no longer qualifies.
|
||||||
|
|
||||||
|
**Story:** n/a (owner-directed bug fix; extends `148_pdf_pages_in_sources`, `147_sync_changed_only`, `140_pictorial_pdf_pages`, `133_pdf_screenshot_fallback`).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pictorial-class PDF is a **readable text document with the pages one toggle away** (Text default; Pages shows the phase-148 strip); a fallback-class PDF keeps the strip-only view byte-identically; and a strip never dies silently — the next sync **re-gates and re-renders** missing copies (or clears a strip that no longer qualifies) with zero LLM calls. The live `Letumdei` doc is repaired by the next production Sync (no manual script): additive mode stamped, 36 copies re-rendered, the book's text back in the viewer.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `148_pdf_pages_in_sources` (complete) — `page_count`, `pdf_page_dir`, `<doc-id>-p{N}.png`, the `/pages/{n}` route, the viewer's `docPagesStrip`, and the persist/regate seams this phase changes.
|
||||||
|
- `147_sync_changed_only` (complete) — the `file_hash` fast path + shared `_unchanged_doc` (the re-gate hook site) whose "the copies persist" assumption this phase replaces.
|
||||||
|
- `140_pictorial_pdf_pages` (complete) — the deterministic `pictorial_pages` gate + the kill switch `settings.pdf_pictorial` (the re-gate mirrors this gate exactly).
|
||||||
|
- `133_pdf_screenshot_fallback` (complete) — `is_usable_pdf_text` (the fallback-class signal the re-gate mirrors).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — the strip-mode wire signal (task 01, LOCKED A2/A3).** New nullable `documents.page_strip_mode` VARCHAR(16) (migration **0027**, tested downgrade, A13): `"primary"` = fallback class (strip replaces the text — today's behavior), `"additive"` = pictorial class (text + toggle). Set exactly when `page_count` is set (cleared with it). `DocContent` gains `page_strip_mode: str | None = None` with the `image_url` omission rule: **ABSENT (never null) when `page_count` is NULL** — every non-qualifying doc's payload stays byte-identical to pre-phase. For a qualifying doc the key is present; a **NULL-mode pre-phase row serializes ABSENT** and the viewer treats absent as `"primary"` (existing rows keep today's DOM until the re-gate stamps them). The `/pages/{n}` route is untouched.
|
||||||
|
|
||||||
|
**D2 — write-first atomic persist (task 02, LOCKED A4).** `_persist_pdf_pages` reorders: **render → write every page (atomic per file: write to `<name>.png.tmp`, `os.replace`) → only then delete stale copies** outside the written set (the shrunk-PDF tail; the current blanket `_delete_page_copies` BEFORE the render is removed — a rollback or crash can no longer leave the row over zero files). The partial-write failure path unlinks BOTH the new final files and the `.tmp` files before re-raising (no orphans; the existing fail-soft contract — caller sets `page_count` NULL + warning — unchanged). The function also stamps `doc.page_strip_mode`: `"primary"` when `fallback_fired`, `"additive"` when `pictorial_described` (the caller passes the CLASS, not just the boolean — `eligible=fallback_fired or pictorial_described` is replaced by `strip_class="fallback"|"pictorial"|None`); the ineligible arm clears mode with `page_count`. **The eligibility RULE itself is unchanged** (the phase-148 lock `fallback_fired or pictorial_described` stands — only its VIEW consequence changes, via D1/D4).
|
||||||
|
|
||||||
|
**D3 — the deterministic fast-path re-gate (task 03, LOCKED A5).** A new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths — inside `_unchanged_doc` (an optional `full_path` kwarg, defaulting to `Path(doc.full_path)`), so it fires on the `file_hash` fast path AND the content-digest-hit path — guarded by `doc.page_count is not None` (only PDF rows ever carry it; every other file's fast path keeps its zero-extra-read contract). It mirrors the changed-path gate **deterministically from the current file bytes + current settings — ZERO vision/LLM calls** (`render_pdf_pages` is CPU-only, A21-safe):
|
||||||
|
- `not is_usable_pdf_text(extract_pdf_text(full_path))` → class `fallback` (mode `primary`) — same doc as today;
|
||||||
|
- elif `settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=settings.pdf_pictorial_min_chars)` is non-empty → class `pictorial` (mode `additive`) — **candidates are exactly the existing gate's** (text-carrying pages under `min_chars`; no-text pages are NOT candidates, mirroring `pictorial_pages` — the re-gate adds no new semantics);
|
||||||
|
- else → **repair arm**: the strip no longer qualifies (e.g. the owner lowered `pdf_pictorial_min_chars`) → `page_count = None`, `page_strip_mode = None`, `_delete_page_copies(doc, page_dir)`.
|
||||||
|
|
||||||
|
For a re-gated-eligible doc, **verify the copies**: any of `page_dir/<doc.id>-p1..p{page_count}.png` missing → re-render ALL pages (`render_pdf_pages`), atomic write-first (D2's writer), and set `page_count` to the fresh render's length (a drift names the doc in a warning; bytes unchanged makes drift pathological). A fresh render producing NO pages (unopenable despite the just-computed hash — pathological) keeps the row as-is with a warning (fail-soft, never clears on a transient). One greppable INFO line whenever re-gate ACTS (never per-unchanged-doc): `import: PDF strip re-gate source=… path=… — healed N pages | cleared stale strip | mode <old>-><new>`. The re-gate mutates the row INSIDE the unchanged path's transaction (the commit already owned by `_unchanged_doc`'s caller flow); a copy-verify/render exception is caught, warned, and never fails the sync.
|
||||||
|
|
||||||
|
**D4 — the additive viewer (task 04, LOCKED A1/A6).** `renderDocument` (the ONE shared core — modal AND `/document.html`): the strip branch now keys on the mode —
|
||||||
|
- `page_strip_mode` absent or `"primary"` (with `page_urls`): the EXACT current DOM (`.doc-pages` strip, no toggle, no Summary panel) — byte-identical, pinned;
|
||||||
|
- `"additive"` (with `page_urls`): the text doc renders **exactly as a non-qualifying text doc** (Summary panel + content, the pre-148 branch reused), PLUS a `Text`/`Pages` toggle above the content: `<div class="doc-view-toggle" role="group" aria-label="Document view">` with two `<button type="button" aria-pressed>`; **default `Text`** (LOCKED A6 — no sparse-ratio default flip); switching to `Pages` appends/reveals the `docPagesStrip` (imgs `loading="lazy"` — hidden-until-shown never fetches) and hides the text block; switching back reverses. State is per-render (no persistence). createElement + properties/textContent only (the never-build-HTML rule); styles.css gains `.doc-view-toggle` per the house button idiom (focus-visible ring, ≥4.5:1, pressed state is text + style, never color alone — B5).
|
||||||
|
- no `page_urls`: byte-identical today's text doc (untouched).
|
||||||
|
|
||||||
|
**D5 — the live repair (task 05, records; no code).** With D3 shipped, the next production Sync takes the `Letumdei` doc's unchanged fast path → re-gate: usable text layer + ≥1 sparse candidate (page 36) → mode `additive` stamped → all 36 copies missing → re-rendered → the viewer shows the book's text with the Pages toggle. The IKEA-class doc (bytes unchanged, copies present, mode NULL pre-phase) gets mode `additive` stamped with zero render work. The E2E fixture reproduces the exact live shape (one illustration block on the last page, copies deleted out from under a stamped `page_count`).
|
||||||
|
|
||||||
|
**Invariants (all tasks).** The changed-path eligibility rule (`fallback_fired or pictorial_described`), every vision pass, the `/pages/{n}` route, the fallback-class strip DOM, the non-qualifying payloads, all SSE frame shapes, the mock-LLM markers, and the prune (`_delete_page_copies` at prune time) are byte-identical. `PLAN.md` untouched (the §7 viewer line describing the strip is folded at the next redo; AGENTS.md's viewer bullets get the additive note in the final pass).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_schema_wire_strip_mode.md` — the `page_strip_mode` column + migration 0027 + the `DocContent` field + the payload builder + unit pins (D1).
|
||||||
|
2. `02_persist_write_first.md` — `_persist_pdf_pages` write-before-delete + atomic replace + the `strip_class` stamp (D2).
|
||||||
|
3. `03_fast_path_regate.md` — `_regate_pdf_strip` on the unchanged paths: mirror gate, heal, repair, log line (D3).
|
||||||
|
4. `04_viewer_additive_toggle.md` — `renderDocument` mode branch + Text/Pages toggle + styles + frontend unit pins (D4).
|
||||||
|
5. `05_e2e_selfheal_pins.md` — the dedicated Playwright suite (additive view, primary byte-identity, heal, repair arm), the final gates, the docs pass (D5 + gates).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_pdf_page_copies.py` (D2: write-first order, atomic replace, partial-write cleanup, the class stamp; D1: column defaults), `tests/unit/test_importer_unchanged_fast_path.py` (D3: every re-gate arm incl. kill-switch, heal, repair, fail-softs, the zero-extra-read guard for non-strip docs), `tests/unit/test_docs_api.py` or the phase-148 payload-pin file (D1: the key present/absent rule, the byte-identity of non-qualifying payloads), a frontend source-contract file per the house style (D4: the mode branch, the toggle semantics, the never-build-HTML rule, the primary-DOM byte-identity).
|
||||||
|
- E2E: `tests/e2e/test_pdf_strip_additive_selfheal.py`, **in isolation** (`uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov`); `test_pdf_pages_in_sources.py` / `test_pictorial_pdf.py` / `test_garbage_pdf.py` stay green (the re-cut pins included).
|
||||||
|
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A pictorial-class PDF opens in the viewer showing its text + Summary panel with a Text/Pages toggle (Text pressed by default); Pages shows the strip and every page image loads (200); a fallback-class PDF's strip-only DOM is byte-identical to pre-phase (no toggle).
|
||||||
|
- [ ] The changed path writes copies before deleting any (measured by the unit order pin); a mid-write failure leaves the prior copies intact and no orphans.
|
||||||
|
- [ ] Deleting a strip's copies on disk + a re-sync (unchanged bytes) restores them and stamps the mode (the E2E heal test); a stale `page_count` on a now-ineligible doc is cleared on re-sync with the copies removed (the E2E repair test); zero vision/LLM calls on the fast path (asserted via the mock).
|
||||||
|
- [ ] Non-qualifying-doc payloads and the primary-class DOM are byte-identical; the full `uv run pytest` suite is green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass (the viewer description gains the additive-mode sentence; `PLAN.md` untouched per the invariants).
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate. Post-deploy (owner runtime, recorded here): the next production Sync self-heals `Letumdei` per D5 — no manual script.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — additive strip**: usable-layer (pictorial-class) PDFs show text + Summary + a Text/Pages toggle; the text is NEVER hidden for them; fallback-class keeps the strip-only view byte-identical (owner-confirmed 2026-10-07 — supersedes phase-148's "strip instead of the text content" for the pictorial class only).
|
||||||
|
- **A2 — the wire signal**: `page_strip_mode: "primary"|"additive"` ABSENT (never null) when `page_count` is NULL; absent ⇒ viewer treats `"primary"`; non-qualifying payloads byte-identical (owner-accepted default, 2026-10-07).
|
||||||
|
- **A3 — the column**: `documents.page_strip_mode` VARCHAR(16) nullable, migration 0027 with a tested downgrade (A13); stamped/cleared exactly with `page_count`.
|
||||||
|
- **A4 — write-first atomic copies**: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete; a rollback can never leave the row over zero files; the fail-soft contract unchanged (owner-confirmed 2026-10-07).
|
||||||
|
- **A5 — the deterministic re-gate**: unchanged paths mirror the changed-path gate exactly (`is_usable_pdf_text` / `pictorial_pages` + the `pdf_pictorial` kill switch, current settings), re-render missing copies (CPU-only, no vision), or clear a stale strip; only rows with `page_count` pay the extra PDF parse; fail-soft on any re-gate exception (owner-confirmed 2026-10-07).
|
||||||
|
- **A6 — default view = Text** for additive docs (no sparse-ratio default flip; the illustration descriptions already live in the text via phase 140) (owner-accepted default, 2026-10-07).
|
||||||
|
- **A7 — the changed-path eligibility RULE is unchanged** (`fallback_fired or pictorial_described`, phase-148 lock intact); this phase changes only the persist ordering, the recovery, and the view consequence.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ alembic/ .agents/phases/ && git commit --no-gpg-sign -m "fix(pdf): additive text+pages viewer for pictorial PDFs, write-first atomic page copies, and a deterministic fast-path re-gate that self-heals lost strip images"
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 01 — The `page_strip_mode` column + wire signal (phase-163 D1, LOCKED A2/A3)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D1 — the live modal showed the strip replacing the text of a genuinely textual book; the viewer needs an explicit class signal (fallback class = strip primary; pictorial class = additive) that never disturbs non-qualifying payloads.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`documents.page_strip_mode` (VARCHAR(16), nullable, migration 0027 with a tested downgrade) exists and `DocContent` carries `page_strip_mode` with the `image_url` omission rule — ABSENT (never null) when `page_count` is NULL, present for qualifying docs — so the frontend (task 04) can branch on the class, and every non-qualifying doc's payload stays byte-identical to pre-phase.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — `Document.page_strip_mode: Mapped[str | None]` (VARCHAR(16), nullable, default NULL), docstring right after `page_count` (phase 148's column): the strip CLASS — `"primary"` = the phase-133 fallback class (the strip IS the view — the screenshots are the content), `"additive"` = the phase-140 pictorial class (the text view + the strip behind a toggle); NULL = no strip. Stamped/cleared exactly with `page_count` (tasks 02/03 own the writes; this task only adds the column).
|
||||||
|
2. `alembic/versions/0027_documents_page_strip_mode.py` — add the column; downgrade drops it. Follow `0026_documents_page_count.py` exactly (revision naming, `nullable=True`, no index). `uv run alembic upgrade head && uv run alembic downgrade -1 && uv run alembic upgrade head` round-trips.
|
||||||
|
3. `app/schemas.py` — `DocContent` gains `page_strip_mode: str | None = None` beside `page_urls`, docstring: the phase-163 strip class; ABSENT (never null — the `image_url` omission rule, `exclude_none`-style serialization already in place for `image_url` — verify how `DocContent` drops `image_url=None` and apply the SAME mechanism) when there is no strip; `"primary"`/`"additive"` otherwise. A pre-phase row (column NULL while `page_count` is set) serializes ABSENT — the viewer treats absent as `"primary"` (task 04).
|
||||||
|
4. `app/api/docs.py` — the content payload builder (the `page_urls=(…)` block, ~line 250): pass `page_strip_mode=doc.page_strip_mode` (the None-drops-the-key serialization handles the absent rule). The `/documents/{doc_id}/pages/{n}` route is NOT touched.
|
||||||
|
5. Unit tests — extend the phase-148 payload-pin file (find where `page_urls` serialization is pinned — `tests/unit/test_docs_api.py` or `test_pdf_page_copies.py`):
|
||||||
|
- a qualifying doc with mode set: the serialized payload carries `"page_strip_mode": "primary"` / `"additive"` next to `page_urls`;
|
||||||
|
- a doc with `page_count` set and mode NULL (pre-phase row): the key is ABSENT (not null) and `page_urls` still present — the exact live `Letumdei` state;
|
||||||
|
- a non-qualifying doc (`page_count` NULL): NO `page_strip_mode` and NO `page_urls` keys — pin the payload byte-identity against the current expected dict;
|
||||||
|
- `page_strip_mode` never appears alone: with `page_count` NULL it is absent even if the column somehow holds a value (the builder keys the whole strip block off `page_count` — if the current builder does NOT, make it so).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; the existing `page_urls` pins stay green unmodified except where a test asserts an exact payload dict (then re-cut in this task). Coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run alembic upgrade head` applies 0027; downgrade removes the column; upgrade again is clean.
|
||||||
|
- [ ] The three payload shapes (mode set / mode NULL with strip / no strip) serialize exactly per D1, pinned by unit tests.
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — Write-first atomic page-copy persist + the strip-class stamp (phase-163 D2, LOCKED A4/A7)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D2 — the live loss: the changed path deletes the old copies BEFORE rendering, outside the DB transaction, so any later per-file rollback (the `EmbeddingError` handler in the import loop) restores the stale `page_count` over ZERO files on disk (filesystem ops are not transactional). External wipes (`pdf_page_dir` cleaned) land the same state; the heal is task 03.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`_persist_pdf_pages` never removes a live copy before the replacement bytes are durably written: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete, and the function stamps `page_strip_mode` from a `strip_class` keyword instead of the bare `eligible` boolean — the eligibility RULE (`fallback_fired or pictorial_described`, phase-148 lock) is unchanged at the call site.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — `_persist_pdf_pages(doc, *, full_path, page_dir, strip_class, source, rel)` (replaces the `eligible: bool` keyword):
|
||||||
|
- `strip_class` is `"fallback" | "pictorial" | None` (the caller passes the CLASS — see 2).
|
||||||
|
- `strip_class is None`: clear BOTH `doc.page_count = None` and `doc.page_strip_mode = None`, THEN `_delete_page_copies(doc, page_dir)` (the deletion-first order is correct HERE — there is no replacement set to protect; note the helper reads `doc.page_count`, so call it BEFORE nulling or pass the old count — check the current order and keep the deletion actually deleting: today the helper runs FIRST while the count is still set; preserve that behavior).
|
||||||
|
- eligible: `render_pdf_pages(full_path)` → the no-pages guard unchanged (`page_count`/mode NULL, warning) → `page_dir.mkdir` → **write loop FIRST**: every non-empty page to `<doc-id>-p{n}.png.tmp` then `os.replace` onto the final name (atomic per file — a reader never sees a torn PNG); collect the written names → **then** delete the stale set: `for n in range(1, doc.page_count + 1)` beyond/present-but-not-written → unlink (the shrunk-PDF tail; reuse `_delete_page_copies` semantics by deleting the old extent files that are NOT in the written set — a file about to be overwritten is not deleted at all).
|
||||||
|
- the failure path: on any exception in the render/write, unlink this call's `.tmp` AND final-name files it wrote (track both), then re-raise (the caller's fail-soft handler — `page_count` NULL + warning — is extended to also null `page_strip_mode`). The OLD copies are untouched by a failed call (they were never deleted up-front — the crash/rollback orphan state is structurally impossible now).
|
||||||
|
- success tail: `doc.page_count = len(pages)` AND `doc.page_strip_mode = "primary"` if `strip_class == "fallback"` else `"additive"`.
|
||||||
|
- docstring: the write-first rationale (the 2026-10-07 live loss: delete-first + a later rollback = the row over zero files) + the per-file atomicity note.
|
||||||
|
2. `app/rag/importer.py` — the `_index_pdf_file` call site (~line 1671): `strip_class = "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)` passed as the keyword; the `eligible=fallback_fired or pictorial_described` expression is GONE (the class carries the same truth table — pin it in the docstring: class non-None ⇔ eligible, the phase-148 rule verbatim). The `except Exception` fail-soft handler nulls mode too.
|
||||||
|
3. `tests/unit/test_pdf_page_copies.py` — extend (the file's existing fixture pattern):
|
||||||
|
- **the order pin**: patch `_delete_page_copies`-class file ops / use a real tmp dir — after a successful re-render of a CHANGED doc, the new files exist and the stale tail (old count > new count) is gone; the overwritten pages keep the NEW bytes;
|
||||||
|
- **the failure pin**: a render that raises mid-write (the 2nd of 3 pages) → the FIRST render's old copies still on disk (the rollback-orphan state is unrepresentable), no `.tmp`/partial orphans from this call, the caller's fail-soft fires (`page_count`/mode NULL + the warning);
|
||||||
|
- **the atomic pin**: the writer uses tmp + `os.replace` (spy or a failing-replace test — a reader opening the final path never sees truncated bytes; assert the tmp name pattern is used and `os.replace` is called);
|
||||||
|
- **the class stamp**: fallback → `page_strip_mode == "primary"`; pictorial → `"additive"`; ineligible → both NULL and old copies deleted;
|
||||||
|
- the existing phase-148 pins (gap = per-page `b""` leaves no file and counts in `page_count`; the shrunk tail) stay green, re-cut only where the deletion order changes expectations.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A successful changed-path render: new copies durable BEFORE any old copy is removed (measured by the order pin); the shrunk tail still cannot leak.
|
||||||
|
- [ ] A mid-write failure leaves the prior copies intact and zero orphans (files on disk exactly the prior set), `page_count`/`page_strip_mode` NULL, sync continues (the fail-soft unchanged).
|
||||||
|
- [ ] `strip_class` carries the exact phase-148 eligibility truth table; no call site passes a bare `eligible` anymore.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 03 — The deterministic fast-path re-gate: heal & repair (phase-163 D3, LOCKED A5)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D3 — with the copies lost (external wipe, or the pre-phase-163 delete-first window), the phase-147 fast path assumes "the copies persist" and NEVER re-renders: the live `Letumdei` row (36 `page_urls`, zero files, unchanged bytes on every sync since) is the proof. The unchanged paths must re-gate the strip deterministically — no LLM/vision calls — heal missing copies, and clear strips that no longer qualify.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths (the `file_hash` fast path and the content-digest-hit path), guarded by `doc.page_count is not None`: it mirrors the changed-path gate from the current bytes + settings, re-stamps `page_strip_mode`, re-renders when any copy is missing, or clears a stale strip — one greppable INFO line only when it ACTS, fail-soft on every exception.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — new `_regate_pdf_strip(session, doc, *, full_path, llm, summary) -> None`:
|
||||||
|
- **mirror the changed-path gate exactly** (the truth table in the phase overview D3):
|
||||||
|
`text = extract_pdf_text(full_path)`; `is_usable = is_usable_pdf_text(text)`;
|
||||||
|
- `not is_usable` → class `fallback` (mode `"primary"`) — the phase-133 signal verbatim;
|
||||||
|
- elif `llm.settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=llm.settings.pdf_pictorial_min_chars)` non-empty → class `pictorial` (mode `"additive"`) — **the existing gate verbatim** (text-carrying pages under `min_chars` only; no-text pages are NOT candidates — `pictorial_pages`'s own contract; the re-gate adds no new eligibility semantics);
|
||||||
|
- else → **repair arm**: `_delete_page_copies(doc, page_dir)` (while `page_count` is still set — the helper reads it), then `doc.page_count = None`, `doc.page_strip_mode = None`; INFO `import: PDF strip re-gate source=… path=… — cleared stale strip`.
|
||||||
|
- **mode re-stamp** (eligible arms): a mode change (NULL→stamped, or the owner's kill-switch/class flip) writes `doc.page_strip_mode` and, when it changes an already-set mode, INFO `… mode <old>-><new>`; a NULL→stamp on existing rows is silent-beneficial (fold into the same line only when work follows).
|
||||||
|
- **copy verify + heal** (eligible arms): `page_dir = Path(llm.settings.pdf_page_dir).expanduser()`; missing = any of `page_dir/f"{doc.id}-p{n}.png"` for `n in 1..doc.page_count` absent. Any missing → `render_pdf_pages(full_path)`: the no-pages result (unopenable — pathological right after the fast path hashed the file) keeps the row untouched + WARNING `import: PDF strip re-gate source=… path=… — re-render produced no pages (state kept)`; else WRITE EVERY page via task-02's atomic writer (tmp + `os.replace`) — reuse it (extract the shared writer helper from task 02 rather than duplicating), then delete the stale tail beyond the new count; `doc.page_count = len(pages)` — a drift from the old count adds a WARNING naming both counts; INFO `… healed N pages`.
|
||||||
|
- **fail-soft**: the whole body in `try/except Exception` → `logger.warning` (the re-gate never fails a sync; the row keeps its pre-re-gate state — but note the repair arm's DB writes and the heal's row writes are in the caller's transaction, so an exception AFTER them rolls back naturally with the file anyway).
|
||||||
|
- docstring: the 2026-10-07 live state as the rationale; the zero-LLM-call property (render is CPU — A21-safe); the guard means every non-strip file's fast path keeps its zero-extra-read contract.
|
||||||
|
2. `app/rag/importer.py` — `_unchanged_doc(...)`: gains the optional keyword `full_path: Path | None = None` (defaults to `Path(doc.full_path)`); at its tail, `if doc.page_count is not None and str(doc.path).lower().endswith(".pdf"): _regate_pdf_strip(session, doc, full_path=…, llm=llm, summary=summary)`. Call sites: the `_index_file` fast path passes the live `full_path`; the branches' content-digest-hit calls pass their own `full_path` (every call site already has it). Docstring: the phase-147 "the copies persist" paragraph is replaced — "unchanged files re-gate the PDF strip deterministically (phase 163)".
|
||||||
|
3. `tests/unit/test_importer_unchanged_fast_path.py` — extend (the file's fast-path fixture pattern; the mock embedder asserts ZERO chat/vision calls across every case):
|
||||||
|
- **heal**: a stamped strip doc (`page_count=3`, mode NULL or set), copies deleted out from under it, unchanged bytes → sync → the 3 files exist (PNG bytes), mode stamped `"additive"` (usable layer + sparse candidate), INFO `healed 3 pages`, `page_count` unchanged (no drift);
|
||||||
|
- **fallback heal**: an unusable-layer doc → mode `"primary"`, copies restored;
|
||||||
|
- **no-op**: all copies present, mode set → zero file writes, zero INFO lines (pin "INFO never fires per unchanged doc" — assert the log records contain no re-gate line);
|
||||||
|
- **repair**: a row with `page_count` set (copies present) on a PDF whose gate now says ineligible (clean text, zero sparse pages) → cleared (both columns NULL, copies deleted), INFO `cleared stale strip`;
|
||||||
|
- **kill switch**: `pdf_pictorial=False` + pictorial-class doc → repair arm clears the strip (the changed-path semantics mirrored);
|
||||||
|
- **fail-soft**: `render_pdf_pages` patched to raise → sync completes, row keeps its prior state, WARNING, no copies touched;
|
||||||
|
- **guard**: a non-PDF unchanged file and a PDF with `page_count` NULL → `extract_pdf_text` NEVER called (spy) — the zero-extra-read contract intact.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Unchanged re-sync restores deleted copies (files + row + route-serveable) and stamps the mode — the live-loss state cannot persist past one sync (unit-proven; the E2E pins it in task 05).
|
||||||
|
- [ ] A stale strip on a now-ineligible doc (incl. via the kill switch) is cleared with its copies; the re-gate fires ZERO LLM/vision calls and no INFO when there is nothing to do.
|
||||||
|
- [ ] Non-strip fast paths perform zero PDF parsing (spy-pinned); every re-gate exception is fail-soft.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 04 — The additive viewer: text + Summary + the Text/Pages toggle (phase-163 D4, LOCKED A1/A6)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D4 — the live complaint: "Notice how all the pages are unavailable… it shouldn't be a pictorial pdf." One short map page flipped a 114 KB-text book into the phase-148 strip-only view (the strip renders INSTEAD of the text content AND the Summary panel). The owner's call: additive for the pictorial class — the text is never hidden; the strip is a toggle away.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`renderDocument` (the ONE shared core — the Sources modal AND `/document.html`) branches on `doc.page_strip_mode`: absent/`"primary"` keeps the current strip-only DOM byte-identically; `"additive"` renders the doc exactly as a text doc (Summary panel + content) plus a `Text`/`Pages` toggle (default `Text`) revealing the existing `docPagesStrip`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document.js` — the `renderDocument` strip branch (~line 374, `const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;`):
|
||||||
|
- `pageUrls` truthy AND `doc.page_strip_mode === "additive"`: fall through to the EXISTING text-doc rendering (the pre-148 branch — Summary panel + content, reused unchanged), THEN append the view toggle + the strip container:
|
||||||
|
- `<div class="doc-view-toggle" role="group" aria-label="Document view">` + two `<button type="button" class="doc-view-btn" aria-pressed>` — labels `Text` / `Pages`; initial state `Text` pressed (`"true"`), the strip NOT in the content flow;
|
||||||
|
- `Pages` click: `aria-pressed` flips; the text block + Summary panel hide (a class toggle, e.g. a `.doc-views[data-view="pages"]` wrapper — CSS-driven, never re-render), the `docPagesStrip(doc, pageUrls)` (the EXISTING builder, unmodified — lazy imgs never fetch while hidden) is appended once and shown;
|
||||||
|
- `Text` click reverses (the strip node stays mounted — no re-fetch on flip-flop);
|
||||||
|
- structure: wrap the text view + the strip in one `.doc-views` container appended to `contentEl` (both surfaces get it via the shared core; the wrapper exists ONLY on additive docs — every other doc's DOM is untouched).
|
||||||
|
- `pageUrls` truthy AND mode absent or `"primary"`: the CURRENT branch verbatim (strip only, no Summary panel, no toggle) — byte-identical DOM, pinned;
|
||||||
|
- falsy `pageUrls`: untouched.
|
||||||
|
- the never-build-HTML rule: createElement + properties/textContent only (the labels are fixed strings — no doc-derived text inside the toggle); the `strip_mode` read is `doc.page_strip_mode === "additive"` (anything else ⇒ primary — the tolerant-reading house rule).
|
||||||
|
- the module docstring: the phase-163 paragraph (the additive class contract, LOCKED A1/A6) replacing the phase-148 "INSTEAD of the text content AND the .doc-summary panel" sentence's scope (it now reads: primary class only).
|
||||||
|
2. `frontend/assets/styles.css` — `.doc-view-toggle` (house button idiom beside the existing segmented/pill controls — check `.format-badge`/the modal action buttons for the closest idiom): the two buttons adjacent, pressed state via `aria-pressed="true"` styling (weight/background pair — **text + style, never color alone**, B5), `:focus-visible` 3px ring, ≥4.5:1 text contrast on both states, ≥44px touch targets on mobile (§7.1); `.doc-views[data-view="pages"]` hides the text block + Summary panel and shows the strip; default (`data-view="text"`) hides the strip container entirely.
|
||||||
|
3. Unit (frontend source-contract — the house pattern; extend the file that already pins the phase-148 strip DOM — locate it via `docPagesStrip` in `tests/unit/`, likely `test_pdf_page_copies.py` / a viewer-contract file):
|
||||||
|
- the additive branch builds the `.doc-views` wrapper + the toggle with the exact labels/roles/`aria-pressed` initial state (source-regex or the file's established DOM-assertion style);
|
||||||
|
- the primary/absent branch contains NO `doc-view-toggle` string and the strip-only structure is byte-identical to the pre-phase pins;
|
||||||
|
- the toggle handler mutates only `aria-pressed` + the `data-view` attribute (no innerHTML anywhere in the new code path — the never-build-HTML pin);
|
||||||
|
- `loading = "lazy"` remains on the strip imgs (the existing pin re-runs green).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the contract pins above; coverage **>90%** on `app/` (no `app/` change in this task — the gate must simply stay green).
|
||||||
|
- The phase's E2E (task 05) exercises the real DOM; `test_pdf_pages_in_sources.py` (the phase-148 E2E) is re-cut ONLY where a modal-DOM assertion meets the additive fixture (the fixture's mode stamp decides; prefer fixture docs that keep primary semantics for the existing pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An additive doc shows Summary + text with `Text` pressed by default; one click shows the strip (every page img loads); clicking back shows the text with zero refetch (the strip node persists — E2E-pinned in task 05).
|
||||||
|
- [ ] A primary/absent-mode doc's viewer DOM is byte-identical to the pre-phase structure (unit pin green); a text doc and every non-PDF are untouched.
|
||||||
|
- [ ] The toggle passes the WCAG basics: labeled group, `aria-pressed` state as text + style, focus-visible ring (checked against §7.2; the E2E asserts the attributes).
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 05 — The dedicated E2E: additive view, primary byte-identity, heal & repair; the final gates (phase-163 D5)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D5 + the AGENTS.md per-phase E2E rule (A16).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
**Pre-fix evidence (measured on the live server, 2026-10-07 — recorded here per house practice):** `Letumdei_ Heirs to the Imperium.pdf` — payload carries 36 `page_urls`; `GET /api/documents/b72f5515-dc40-4167-98ff-179a6503e6e5/pages/1,2,18,36` → **404/404/404/404** (page 37 → 404, the out-of-range arm); the content is 114 389 chars of genuine extracted text with 36 `---PAGE-N---` markers and exactly ONE `[Page 36 illustration: …]` block; the same server serves `besta-frame-white__AA-1272068-11-100.pdf` page 1 → **200** (the dir is intact; only the 10/1-indexed doc's copies were lost). The viewer showed "Page N unavailable." on every page with the text hidden.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One Playwright suite proves the whole phase end to end — the additive view with the toggle, the fallback class's byte-identical strip-only DOM, the disk-level heal, and the repair arm — and the phase's final pass runs every gate plus the full documentation consistency review.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_pdf_strip_additive_selfheal.py` — new suite (run in isolation; read `tests/e2e/test_pdf_pages_in_sources.py` and `tests/e2e/test_pictorial_pdf.py` first and reuse their synthetic-PDF builder + fixture-KB pattern; deterministic mock LLM, the default):
|
||||||
|
- **fixture**: source `strip_fixture` with (a) `manual.pdf` — usable text layer, 3 text pages + 1 sparse `<100-char` map-style page (the pictorial class — the Letumdei shape in miniature); (b) `scanned.pdf` — the garbage/unusable-layer fixture (the phase-133 pattern) → fallback class; (c) `plain.pdf` — clean text, zero sparse pages → no strip.
|
||||||
|
- **assert 1 (additive view)**: index → the modal for `manual.pdf`: mode stamped `additive` (via the content API), the text + Summary panel visible by default, the `Text`/`Pages` toggle present with `Text` `aria-pressed="true"`; click `Pages` → the strip renders, EVERY page `<img>` has `naturalWidth > 0` (routes serve — no "unavailable" note anywhere); click `Text` → text visible, strip hidden, the strip node still mounted (no refetch: assert the img count does not reset).
|
||||||
|
- **assert 2 (primary byte-identity)**: the modal for `scanned.pdf`: the strip-only DOM (no `.doc-view-toggle`, no Summary panel) and the "Page N" figures match the phase-148 structure (the existing suite's DOM assertions adapted); `plain.pdf` renders the plain text doc with no strip artifacts.
|
||||||
|
- **assert 3 (heal)**: for `manual.pdf`, delete ALL `pdf_page_dir/<id>-p*.png` (the fixture's page dir) + confirm a page route 404s → run a sync (unchanged bytes → the fast path) → the files exist, the page route 200s, the payload still carries `page_urls` + `additive`, the viewer's Pages view loads every img (the live-loss state, healed by the next sync — the exact D5 promise).
|
||||||
|
- **assert 4 (repair)**: direct-DB seed — set `page_count` (and stale copies on disk) on `plain.pdf`'s row (the stale-eligible state no current index produces) → sync → `page_count`/`page_strip_mode` NULL (read through the fixture session), the stale copies deleted, the modal shows the plain text (the strip gone).
|
||||||
|
- **assert 5 (no vision)**: the whole suite's syncs ran against the deterministic mock — no chat/vision endpoint hits beyond the initial index's own passes (the heal burns zero LLM calls; assert via the mock's call counter or the suite's request log, the house pattern).
|
||||||
|
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
|
||||||
|
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` green **in isolation**; `tests/e2e/test_pdf_pages_in_sources.py`, `test_pictorial_pdf.py`, `test_garbage_pdf.py`, `test_document_viewer*.py`, `test_image_documents.py` green;
|
||||||
|
- `uv run ruff check . && uv run pyright` clean; the alembic 0027 up/down/up round-trip clean.
|
||||||
|
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — the viewer description gains the additive-mode sentence (text + Text/Pages toggle for pictorial-class PDFs; the fallback class keeps the strip), the importer notes (write-first copies, the fast-path re-gate + its INFO line) — and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` is NOT touched (its §7 strip line folds at the next redo — recorded in the phase overview invariants).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the E2E + gates task; its own new-code coverage rides the suite (page route + fixture helpers are test-side; `app/` coverage stays **>90%**).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All five E2E asserts green in isolation; the four existing viewer/PDF suites green unmodified (except explicitly re-cut pins, each named in the commit body).
|
||||||
|
- [ ] The heal assert proves: copies deleted externally → one unchanged-byte sync → files restored, routes serve, viewer complete (the live `Letumdei` failure mode is impossible after this phase).
|
||||||
|
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; migration round-trip clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full, every inconsistency fixed (the additive-viewer sentence included); the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (see the overview) is made by the pipeline gate.
|
||||||
|
- [ ] Owner follow-up recorded (not code): after deploy, the next production Sync self-heals `Letumdei` (D5) — verify once via the Sources modal (text visible, Pages loads all 36) and `grep` the server log for the one `import: PDF strip re-gate … healed 36 pages` line.
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# Phase 164 — Chat failure honesty: the shared error-detail serializer, a distinct pre-stream failure line, and caps that accept a full-length answer
|
||||||
|
|
||||||
|
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, right after the live answer cap was raised to `BOR_LLM_MAX_OUTPUT_TOKENS=131072`: when an answer hit the length limit, the next turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."* The failure chain (traced in code, 2026-10-07): a long answer (> 32 000 chars) lives in `bor.chat.v1` and rides back to the server as `history` on the next `POST /api/chat`; `HistoryTurn.text` (`max_length=32_000`) rejects it with a **422 whose `detail` is a LIST OF OBJECTS** (`[{loc, msg, type}, …]`); the chat's pre-stream error path (`frontend/assets/app.js` ~3956: `if (body.detail) detail = body.detail`) has **no type guard** — `new Error(<array>)` stringifies to `"[object Object]"`, which the zero-frame settle shows as the fixed `FAILED_TURN_TEXT` ("the connection dropped" — untrue: the server WAS reachable and answered 422) plus the garbage detail. This is the SECOND occurrence of the same class: `HistoryTurn`'s own docstring records the 4 000 → 32 000 raise for an identical 422 (phase-42 E2E); the 32 000 ceiling broke again the moment outputs could exceed it — **at the DEFAULT `max_output_tokens=32_768` (~130k chars), not only at 131072**.
|
||||||
|
|
||||||
|
**Story:** n/a (owner-directed bug fix; extends the phase-120 failed-turn contract, the phase-114 hint channel, the phase-83 SEC-05 boundary caps, the phase-74 history budgets).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A failed turn states **what actually happened**: pre-stream HTTP rejections get their own honest line (not "the connection dropped") with the validation message serialized verbatim — never `[object Object]` — and a full-length answer (up to a 131 072-token cap ≈ 524k chars) flows through the whole chain: accepted as `history` on the next turn, and saveable/restorable as part of a saved chat. The anonymous saved-chat surface keeps a bounded worst case (the phase-83 SEC-05 property survives the raise).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `83_chat_save_payload_limits` (complete) — the SEC-05 caps this phase raises TOGETHER (`ChatMessage.text`/`thinking` mirror `HistoryTurn`; the mirror invariant must stay true).
|
||||||
|
- `74`-line history budgets (complete; `history_to_messages`, `history_max_turns=40` / `history_max_chars=24_000`) — the LOCKED budget trim (owner 2026-09-08: overflow turns dropped WHOLE, never cut) — **untouched**; it is the real prompt-side gate and stays so.
|
||||||
|
- `120_failed_turn_persistence` (complete; the `FAILED_TURN_TEXT` / `finalizeFailedTurn` / `.failed-note` contract) and `114` hint channel (`err.hint` → banner) — the bubble/note mechanics this phase extends.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — the caps chain up, together (task 01, LOCKED A1).** `app/schemas.py`:
|
||||||
|
- `HistoryTurn.text` / `HistoryTurn.thinking`: `max_length=600_000` (rationale IN the docstring: a full-cap answer at the owner's `BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token — 600 000 is that with margin; the budget trim (locked A3, 2026-09-08) remains the real prompt gate — a turn over `history_max_chars` is dropped WHOLE, so a long answer trims itself out of later prompts; this phase fixes the 422, NOT the trim (its locked semantics are a separate owner decision, recorded in the overview));
|
||||||
|
- `ChatRequest.history` list: `max_length=100` → `500` (the same "DoS sanity ceiling only" docstring, now also noting the client sends the whole `bor.chat.v1` minus the current question);
|
||||||
|
- `ChatMessage.text` / `ChatMessage.thinking`: `max_length=600_000` (MIRRORS `HistoryTurn` — the phase-83 invariant "a saved chat can never legitimately carry more than the chat path accepts" must stay TRUE: raise both or neither);
|
||||||
|
- `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → `500` (symmetry with the history list) PLUS a new `model_validator` on both: **total chars across all messages (`len(text) + len(thinking or "")`) ≤ 8_000_000** — this bounds the ANONYMOUS SEC-05 surface at the same order as the audited pre-phase worst case (200 × 64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the validator error surfaces as the standard 422 (`ValueError` message names the budget);
|
||||||
|
- `ChatRequest.message`: `max_length=4000` UNCHANGED (the composer textarea carries the cap client-side — pin it).
|
||||||
|
|
||||||
|
**D2 — the shared `errorDetailText` serializer (task 02, LOCKED A2).** `frontend/assets/app.js` exports `errorDetailText(value, fallback)` (exported const/function, the `TURN_TIMEOUT_MS` unit-pin idiom): a non-empty **string** → verbatim; an **array** → per-entry `entry.msg` when the entry is an object else `String(entry)`, empties dropped, `"; "`-joined, all-empty → `fallback`; an **object** → `String(entry.msg)` when present, else `fallback`; nullish / empty / anything else → `fallback`. NEVER `JSON.stringify` a raw detail (the FastAPI 422 entries carry the FULL `input` — the whole conversation — which must never reach the DOM; only `msg` and plain strings serialize). Applied at the TWO unguarded chat paths: the pre-stream `!res.ok` handler (~3956 — `detail = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`)`) and the SSE `error`-frame throw (~4353 — `new Error(errorDetailText(ev.detail, "Something went wrong on my side."))`, the `err.hint` line untouched). The seven EXISTING guarded copies (`app.js` tuning form, `tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`) are behaviorally correct and stay byte-identical (no opportunistic refactor — the new helper matches their semantics for the shared shapes, generalized to arrays-of-strings and multi-entry joins).
|
||||||
|
|
||||||
|
**D3 — the distinct pre-stream failure line (task 03, LOCKED A3/A4).** `app.js`:
|
||||||
|
- `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` (sibling of `FAILED_TURN_TEXT`, its docstring: the PRE-STREAM HTTP-rejection settle — the server answered, the request did not pass; the real detail rides the `.failed-note` + persisted `error` key exactly as `FAILED_TURN_TEXT` does);
|
||||||
|
- the pre-stream `!res.ok` throw carries `err.bubbleText = FAILED_REQUEST_TEXT` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send."` (the phase-114 banner-hint channel);
|
||||||
|
- `runTurn`'s catch passes `bubbleText: err?.bubbleText` into `finalizeFailedTurn` (new options key); the zero-frame settle uses `bubbleText || FAILED_TURN_TEXT` — the persisted record text stays "what the user saw is what is stored";
|
||||||
|
- UNCHANGED settles: a transport-level failure (the `fetch` itself rejects — `TypeError`) has no `bubbleText` → keeps `FAILED_TURN_TEXT` (that one IS the connection); the SSE `error`-frame zero-frame path (a model-side failure) keeps `FAILED_TURN_TEXT`; the stop / abort paths are untouched. A partial-stream settle (`wrap` + `acc`) never shows a bubble text at all — unchanged.
|
||||||
|
|
||||||
|
**D4 — verification (task 04).** Dedicated Playwright suite `tests/e2e/test_chat_error_honesty.py` (isolation; deterministic mock LLM) with (a) a `page.route` stub of `POST /api/chat` returning the EXACT FastAPI 422 shape for a long-history turn → assert the bubble reads `FAILED_REQUEST_TEXT`, the `.failed-note` detail contains the pydantic `msg` verbatim ("String should have at most 600000 characters" — never `[object Object]`, never `input` text), the banner hint shows, Retry + Edit land, composer re-enables (§7.4); (b) a 500 stub with a non-JSON body → the `HTTP 500` default string line (the serializer's fallback path); (c) a real-server acceptance: send a turn whose history carries a ~525k-char prior brain answer → the request is ACCEPTED (no 422) and the turn settles normally with the mock (the budget trim may drop the giant turn WHOLE from the prompt — the locked A3 behavior; the test asserts acceptance + settle, not prompt content); (d) a stubbed SSE `error` frame carrying an ARRAY detail → the note serializes (client tolerance). Server-side acceptance/422 matrix for every raised cap + the total-chars validator rides integration pins in task 01 (the `test_chats_api.py` pattern).
|
||||||
|
|
||||||
|
**Invariants (all tasks).** The budget trim (`history_to_messages`, budgets, whole-drop, contiguous window) is byte-identical (locked A3 2026-09-08). `ChatRequest.message` 4000. The `FAILED_TURN_TEXT` wording, the phase-120 persistence keys (`failed`, `error` ≤500), Retry/Edit pairing, the SSE frame shapes, and the seven guarded frontend detail-readers are byte-identical. `PLAN.md` untouched.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_caps_long_answer_chain.md` — the `schemas.py` raise + the total-chars validator + docstrings + the unit/integration acceptance-rejection matrix (D1).
|
||||||
|
2. `02_error_detail_serializer.md` — the exported `errorDetailText` + the two chat call sites + unit pins (D2).
|
||||||
|
3. `03_prestream_failure_copy.md` — `FAILED_REQUEST_TEXT` + the `bubbleText` threading + the banner hint + the failed-turn pins (D3).
|
||||||
|
4. `04_e2e_error_honesty.md` — the dedicated E2E (D4) + the final gates + the docs consistency pass (incl. the proxy body-size deploy note).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_schemas.py` (or the file pinning `HistoryTurn`/`ChatMessage` bounds — locate via `32_000` in `tests/`): every cap's boundary (600_000 accept / +1 reject; 500 vs 501 turns; the total-8M validator: accept ~15 full-cap messages, reject beyond), the 422 `detail` shapes; `tests/unit/test_frontend_error_detail.py` (new; the `test_frontend_*` house pattern): all serializer shapes incl. the exact live FastAPI 422 body; `tests/unit/test_frontend_failed_turn.py` extended: the `bubbleText` threading (pre-stream vs transport vs SSE-frame settles), the never-`innerHTML` rule.
|
||||||
|
- Integration: `tests/integration/test_chats_api.py` extended — a saved chat carrying ONE 525k-char answer saves 201 (the phase-83 surface stays bounded by the total validator), over-total rejects 422.
|
||||||
|
- E2E: `tests/e2e/test_chat_error_honesty.py` green **in isolation**; the phase-120 failed-turn E2E stays green (the `FAILED_TURN_TEXT` path unchanged for transport failures).
|
||||||
|
- Coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A pre-stream 422 shows the distinct honest line + the validation `msg` verbatim in the note — `[object Object]` is structurally impossible on the chat path (unit + E2E pinned; no raw `input` reaches the DOM).
|
||||||
|
- [ ] A history turn with a 525k-char brain answer is ACCEPTED (200 turn settles); >600k rejects 422; the saved-chat chain accepts the same answer inside a 201 and the anonymous surface stays bounded (total-8M validator pinned).
|
||||||
|
- [ ] Transport failures and SSE error frames keep `FAILED_TURN_TEXT` verbatim; stop/abort paths byte-identical; §7.4 failure feedback re-enables every control.
|
||||||
|
- [ ] The budget trim, `message` 4000, the seven guarded detail-readers, and all SSE shapes are byte-identical; full `uv run pytest` green; coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass — includes the deploy note: the proxy `client_max_body_size` must accommodate the now-legal large chat POSTs (recommend ≥ 16 MB) — and every other inconsistency fixed.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — the caps chain together**: `HistoryTurn.text/thinking` AND `ChatMessage.text/thinking` → 600_000 (≈ 131 072 × 4 chars with margin); history list → 500; saved-messages list → 500 with a total-chars ≤ 8_000_000 model validator (the SEC-05 bounded-anonymous property preserved at the audited order); `message` 4000 unchanged; the phase-83 mirror invariant explicitly honored (owner-confirmed 2026-10-07 — "static generous caps", with the SEC-05 coupling added by this phase's audit of the chain — raise ONE side would break saves).
|
||||||
|
- **A2 — `errorDetailText`**: the shared serializer's exact semantics (string verbatim; array entries `msg`-else-String joined `"; "`; object `msg`-else-fallback; never `JSON.stringify`; never the `input`), exported from `app.js`, applied at the two unguarded chat paths ONLY; the seven guarded copies untouched (owner-confirmed 2026-10-07).
|
||||||
|
- **A3 — `FAILED_REQUEST_TEXT`**: distinct pre-stream HTTP-rejection bubble text verbatim; the persisted record is what the user saw (the phase-120 contract extended, not changed); transport failures keep `FAILED_TURN_TEXT` (owner-confirmed 2026-10-07 — "distinct pre-stream line").
|
||||||
|
- **A4 — the budget trim is NOT touched**: the locked 2026-09-08 whole-drop contiguous-window semantics stand; the consequence — a >24k-char answer is dropped whole from later prompts (follow-ups WORK with a trimmed window, without that answer's context) — is recorded here; making long answers useful in later prompts (bigger `BOR_HISTORY_MAX_CHARS` is pure env, or a smarter cut) is an owner call outside this phase.
|
||||||
|
- **A5 — PLAN.md untouched** (the caps rationale lives in the schema docstrings; folded at the next redo).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): honest pre-stream failure line, a shared error-detail serializer, and history caps that accept a full-length answer"
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 01 — The caps chain up: 600k text, 500 turns, a bounded anonymous total (phase-164 D1, LOCKED A1/A4)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D1 — the live 422: a >32k-char answer rode back as `history` and rejected the NEXT turn. `HistoryTurn`'s docstring already records this failure class happening once (4 000 → 32 000, phase-42 E2E); the ceiling broke again the moment outputs could exceed 32k — at the DEFAULT `max_output_tokens=32_768` too.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every schema cap in the long-answer chain (`HistoryTurn` → `ChatRequest.history` → `ChatMessage` → saved-chat lists) accepts a full-cap answer (~524k chars at `max_output_tokens=131072`), the phase-83 mirror invariant between the chat path and the saved-chat path stays TRUE, and the anonymous saved-chat worst case stays bounded by a new total-chars validator.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py` — `HistoryTurn`: `text`/`thinking` `max_length=32_000`/`32000` → **600_000** both; docstring: the cap accepts a full-length answer (`BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token, plus margin — the 2026-10-07 live 422: the 32 000 ceiling rejected the FOLLOW-UP turn of a long answer, the same class as the recorded 4 000 → 32 000 raise); the real gate stays the locked budget trim (`history_to_messages` — overflow turns dropped WHOLE; a long answer trims itself out of later prompts, never a 422).
|
||||||
|
2. `app/schemas.py` — `ChatRequest.history`: `max_length=100` → **500**; the docstring sentence updates: still "a DoS sanity ceiling only — the config budgets do the real trimming (A3)", plus: the client sends the whole `bor.chat.v1` minus the current question, so the ceiling must clear long conversations.
|
||||||
|
3. `app/schemas.py` — `ChatMessage.text`/`thinking`: → **600_000** (docstring: the phase-83 SEC-05 mirror RAISED with `HistoryTurn` — the invariant "a saved chat can never legitimately carry more than the chat path accepts" survives the raise; raise ONE side and every done-time auto-save of a long answer 422s into the phase-113 quiet-failure).
|
||||||
|
4. `app/schemas.py` — `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → **500** AND a `model_validator(mode="after")` on each: total chars `sum(len(m.text) + len(m.thinking or "") for m in messages) <= 8_000_000` else `raise ValueError("saved chat exceeds the 8,000,000-character budget")` — docstring: bounds the ANONYMOUS SEC-05 worst case at the audited order (the pre-phase 200×64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the 422 is the framework's standard validation response (the phase-83 A2 boundary-only style — NO route changes).
|
||||||
|
5. `ChatRequest.message`: **unchanged** `max_length=4000` (the composer textarea caps client-side — `app.js` ~1122 comment; pin it stays).
|
||||||
|
6. Unit — extend the file pinning these bounds (locate via `32_000` under `tests/unit/`):
|
||||||
|
- `HistoryTurn.text` 600_000 accept / 600_001 reject; same for `thinking`;
|
||||||
|
- `history` 500 accept / 501 reject; `message` 4000 boundary re-pinned unchanged;
|
||||||
|
- `ChatMessage.text/thinking` 600_000 accept / +1 reject;
|
||||||
|
- messages 500 accept; the total validator: ~15 × 524k messages REJECT with the budget-naming `ValueError` surfaced in the 422 `detail`, a saved chat with ONE 525k answer + normal turns ACCEPTS;
|
||||||
|
- the 422 `detail` SHAPE of a cap rejection is the pydantic list-of-objects (the serializer's task-02 input — pinned here as the server-side half of the contract).
|
||||||
|
7. Integration — `tests/integration/test_chats_api.py`: `POST /api/chats` with a one-525k-answer conversation → 201 (round-trips through JSONB — the DB is JSONB, no column changes needed — verify); over-total → 422; the existing phase-83 boundary matrix green (re-cut only the pins that assert the literal `32_000`/`200` values, naming each in the task's completion).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The full-cap-answer conversation flows: accepted as `/api/chat` history (integration-level) AND saveable as a saved chat (201), with the 8M total cap the only new anonymous bound.
|
||||||
|
- [ ] `ChatMessage` mirrors `HistoryTurn` exactly (the phase-83 invariant re-stated in both docstrings, verified by identical-boundary tests).
|
||||||
|
- [ ] The budget-trim suites (`history_to_messages`) are untouched and green — D1 changes what ARRIVES, never what the trim keeps.
|
||||||
|
- [ ] `uv run pytest tests/unit tests/integration -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 02 — The shared `errorDetailText` serializer for the two chat paths (phase-164 D2, LOCKED A2)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D2 — the live "[object Object]": `frontend/assets/app.js` ~3956 does `if (body.detail) detail = body.detail` on a FastAPI 422 whose `detail` is a LIST OF OBJECTS; `new Error(<array>)` → message `"[object Object]"`. The SSE `error`-frame path (~4353 `new Error(ev.detail || …)`) is the second unguarded reader. The guarded idiom ALREADY exists in seven copies (`tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`, `app.js`'s tuning form) — this task generalizes it ONCE and applies it ONLY where it is missing.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`app.js` exports `errorDetailText(value, fallback)` with exact, total semantics (every input shape yields a human string or the fallback — `[object Object]` and raw `input` leakage are structurally impossible), wired at the pre-stream HTTP-error handler and the SSE `error`-frame throw; the seven existing guarded copies stay byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — new exported function (the `THINKING_NEAR_BOTTOM_PX` / `TURN_TIMEOUT_MS` export-for-unit-pin idiom; place beside the other error constants):
|
||||||
|
```
|
||||||
|
errorDetailText(value, fallback) → string
|
||||||
|
```
|
||||||
|
- non-empty **string** → the string verbatim;
|
||||||
|
- **array** → per entry: `String(entry.msg)` when the entry is an object with a non-empty `msg`, else (non-object entry) `String(entry)`; entries with neither drop; join with `"; "`; empty result → `fallback`;
|
||||||
|
- **object** → `String(value.msg)` when a non-empty `msg` exists, else `fallback`;
|
||||||
|
- nullish / empty string / any other type → `fallback`;
|
||||||
|
- NEVER `JSON.stringify` anything, NEVER touch `input`/`loc`/`ctx` (the FastAPI entries carry the full rejected input — the whole conversation — which must never reach the DOM). Docstring names the live 2026-10-07 `[object Object]` failure and the `input`-leak rule.
|
||||||
|
2. `frontend/assets/app.js` — the pre-stream handler (~3956, the `!res.ok || !res.body` block): replace the unguarded assignment with `const text = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`); throw new Error(text);` (the `try`-around-`res.json` and the non-JSON catch keep their current shape; the default per-status string remains the fallback). The `bubbleText`/hint threading is task 03 — do NOT add it here.
|
||||||
|
3. `frontend/assets/app.js` — the SSE `error` frame (~4353): `const err = new Error(errorDetailText(ev.detail, "Something went wrong on my side."));` — the `err.hint = ev.hint;` line byte-identical. A string `detail` (every server-built frame, `ChatErrorEvent`) serializes VERBATIM — pin that the user-visible strings for "Question too long", "dropped the connection" etc. are unchanged.
|
||||||
|
4. `tests/unit/test_frontend_error_detail.py` (new — read `tests/unit/test_frontend_failed_turn.py` first and reuse its harness/assertion style):
|
||||||
|
- the EXACT live 422 body (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"<huge>"}]}`) → the `msg` verbatim, and the result contains NO `input` text and no `"[object Object]"`;
|
||||||
|
- multi-entry array → the `msg`s joined `"; "`; mixed entries (object + plain string) → joined; all-empty array → fallback;
|
||||||
|
- object with `msg` → verbatim; object without → fallback;
|
||||||
|
- string → verbatim (incl. the server's fixed copy — pinned); `""`/null/undefined/number → fallback;
|
||||||
|
- source-contract pins: the two call sites now read through `errorDetailText`; `JSON.stringify` appears NOWHERE in the new helper; the seven guarded copies are byte-identical (hash their regions or assert the untouched lines).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the shape matrix above (the harness the `test_frontend_*` files use); no `app/` change — coverage gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every serialized shape yields a human string or the named fallback — `[object Object]` is unrepresentable at the two chat call sites (unit-proven).
|
||||||
|
- [ ] A 422 `detail` never puts request `input` text into the DOM (pinned); server string frames render byte-identical copy.
|
||||||
|
- [ ] The seven existing guarded readers are untouched; the `err.hint` channel is untouched.
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 03 — The distinct pre-stream failure line: `FAILED_REQUEST_TEXT` (phase-164 D3, LOCKED A3)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D3 — the live bubble said *"My answer didn't make it — the connection dropped"* for a **422**: the server was reachable and answered deliberately; the copy lied, and the owner-confirmed remedy is a distinct honest line for pre-stream HTTP rejections.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pre-stream HTTP rejection (the `!res.ok` settle) bubbles `FAILED_REQUEST_TEXT` — "I couldn't take that — my API rejected the request. Use Retry to ask again." — with the serialized detail on the `.failed-note` and the phase-114 hint on the banner; transport failures (the `fetch` itself rejects) and SSE `error` frames keep `FAILED_TURN_TEXT` byte-identically; the phase-120 persistence contract ("what the user saw is what is stored") extends to the new text.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — new constant beside `FAILED_TURN_TEXT`: `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` docstring: the PRE-STREAM HTTP-rejection settle (the server ANSWERED with a 4xx/5xx — the request died before the model; the 2026-10-07 live 422-after-a-long-answer is the finding); the real detail rides the in-bubble `.failed-note` + the persisted `error` key exactly as `FAILED_TURN_TEXT`'s does; NOT for transport failures (no answer happened at all — `FAILED_TURN_TEXT` is honest there).
|
||||||
|
2. `frontend/assets/app.js` — the pre-stream throw (task 02's site): attach `err.bubbleText = FAILED_REQUEST_TEXT;` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send.";` to the thrown `Error` (the phase-114 `hint` channel already flows from `err.hint` to `showErrorBanner` via the catch's `setUiState` opts — verify, do not duplicate).
|
||||||
|
3. `frontend/assets/app.js` — `finalizeFailedTurn(detail, { acc, thinking, tools, segments, wrap, leavePartialIndex, bubbleText })`: the ZERO-frame branch (`else` — no `wrap`) renders `bubbleText || FAILED_TURN_TEXT` (the `addMessage` text AND the `rememberBrainTurn` record text — same value, the stored-what-you-saw contract) and the note/persistence/Retry/Edit machinery is otherwise untouched (partial-stream settles never receive a `bubbleText` by construction — a pre-stream throw has no `wrap`; still, the partial branch must ignore the key: read it ONLY in the zero-frame branch). Docstring update (the `bubbleText` key, its caller-only meaning).
|
||||||
|
4. `frontend/assets/app.js` — `runTurn`'s catch: pass `bubbleText: err?.bubbleText` into the `finalizeFailedTurn` options (the ONE line-level change; the detail/hint/banner sequencing after the call is untouched).
|
||||||
|
5. `tests/unit/test_frontend_failed_turn.py` (extend; the file's harness):
|
||||||
|
- a thrown `Error` with `bubbleText` → the zero-frame bubble + persisted record text are `FAILED_REQUEST_TEXT`, note = the detail, `failed: true` + the `error` key unchanged in shape (≤500 cap intact);
|
||||||
|
- a thrown `Error` WITHOUT `bubbleText` (the transport-failure shape) → `FAILED_TURN_TEXT` byte-identical (the existing pins re-run green);
|
||||||
|
- the SSE-frame throw (task 02's site) sets NO `bubbleText` → model-side failures keep `FAILED_TURN_TEXT`;
|
||||||
|
- the banner: `err.hint` reaches the opts the same way for both throw kinds (the phase-114 pin re-runs);
|
||||||
|
- `FAILED_REQUEST_TEXT`'s exact string pinned ONCE (the copy is owner-locked); the never-`innerHTML` rule holds.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; the phase-120 suite green WITHOUT modification (the default path byte-identical); coverage gate unchanged (no `app/` code).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Pre-stream HTTP error → bubble `FAILED_REQUEST_TEXT`, note carries the serialized detail (task 02's serializer), hint on the banner, Retry + Edit land, composer re-enables (§7.4).
|
||||||
|
- [ ] Transport failures and SSE error frames are byte-identical to the phase-120 behavior (`FAILED_TURN_TEXT`, same persistence).
|
||||||
|
- [ ] The persisted `bor.chat.v1` record for a new-line failure stores the new text + `failed: true` + the capped `error` — a refresh restores it identically (the same property the existing suite pins for `FAILED_TURN_TEXT`).
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 04 — The dedicated E2E: error honesty + long-answer acceptance; the final gates (phase-164 D4)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D4 + the AGENTS.md per-phase E2E rule (A16).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
**Pre-fix evidence (2026-10-07, recorded per house practice):** owner-live instance after raising `BOR_LLM_MAX_OUTPUT_TOKENS=131072` — the post-long-answer turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."*; code trace: the 422's list-of-objects `detail` hit the unguarded `body.detail` assignment (`app.js` ~3956) → `new Error(<array>)` → `"[object Object]"` (the example was not preserved live — the cap raise removed the repro; the 32 000 ceiling repros it at DEFAULT settings with any >32k-char answer). `HistoryTurn`'s own docstring records the identical failure class once already (4 000 ceiling, phase-42 E2E).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One Playwright suite proves the phase end to end against the REAL client code paths — the distinct pre-stream line with the serialized validation message (never `[object Object]`, never `input` leakage), the fallback and tolerance paths, and the real-server acceptance of a full-length-answer history — and the final pass runs every gate plus the full documentation consistency review.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_chat_error_honesty.py` (new; isolation; deterministic mock LLM; read `tests/e2e/test_frontend_failed_turn*.py`-adjacent suites and any existing `page.route` usage first and reuse the established stubbing approach — if no route-stub precedent exists, intercept `POST /api/chat` with Playwright `page.route` fulfilling the fetch; the assertions must drive the real `app.js` send path):
|
||||||
|
- **assert 1 (the live bug)**: stub `POST /api/chat` → 422 with the EXACT FastAPI shape for a long-history rejection (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"x".repeat(1000)}]}`) → send a question: the bubble text is `FAILED_REQUEST_TEXT` verbatim; the `.failed-note` detail contains the `msg` substring; NO `[object Object]` anywhere in the DOM; the rejected `input` text is NOT in the DOM; the banner shows the hint; Retry + Edit are present; the composer is re-enabled (§7.4).
|
||||||
|
- **assert 2 (fallback)**: stub → 500 with a NON-JSON body → the note/line settle on the `HTTP 500` default string, `FAILED_REQUEST_TEXT` bubble (still pre-stream), no crash.
|
||||||
|
- **assert 3 (acceptance — the D1 fix)**: seed `localStorage` `bor.chat.v1` with a conversation whose LAST brain record carries a ~525k-char answer (the seeded-restore pattern the saved-chat suites use), reload, send a short question → the request is NOT 422 (assert the response status 200 via the route/network log) and the turn settles normally with the mock answer (the locked budget trim may drop the giant turn WHOLE from the prompt — asserting settle, never prompt content, per A4).
|
||||||
|
- **assert 4 (SSE-frame tolerance)**: stub → a valid SSE stream whose `error` frame carries an ARRAY `detail` → the note serializes the `msg` (client tolerance; server-built string frames are pinned in task 02 units).
|
||||||
|
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
|
||||||
|
- `uv run pytest tests/e2e/test_chat_error_honesty.py -v --no-cov` green **in isolation**; the phase-120/114 failed-turn + banner suites and `test_chats_api.py` green (re-cut pins named in the commit body);
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — add the **deploy note**: *the proxy in front of the app must accept chat POSTs sized to the new caps (recommend `client_max_body_size ≥ 16m` — a follow-up turn after a full-cap answer posts ~0.5 MB+, a saved chat up to the 8 MB budget); a proxy with a smaller limit turns a legal request into a 413* — review the error-handling descriptions for accuracy (the failed-turn copy is documented where phase 120's bubbles are), and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` untouched (A5).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the E2E + gates task; its fixtures are test-side; `app/` coverage stays **>90%**.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All four asserts green in isolation; assert 1 proves the exact live message renders honestly end to end (bubble + note + hint + controls).
|
||||||
|
- [ ] The seeded 525k-answer conversation sends, is accepted, and settles — the live "every follow-up dies after a long answer" failure mode is impossible after this phase.
|
||||||
|
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; no existing suite modified except re-cut pins named in the commit body.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full, the proxy body-size deploy note added, every inconsistency fixed; the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (overview) is made by the pipeline gate.
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Phase 165 — Prose word wrap: image descriptions and plain-text docs stop scrolling horizontally
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Need word wrap for image explanations and other long text rather than scrolling horizontally"
|
||||||
|
**Story:** n/a (owner-directed UI defect)
|
||||||
|
**Context:** `frontend/assets/document.js::renderDocument` (the ONE viewer core, imported by `frontend/assets/document-modal.js:73`), `frontend/assets/styles.css` (`.doc-raw` ~L5218, `.doc-md` ~L4929), phases 10 (viewer), 36 (summary panel), 116 (themed raw scrollbar, **locked A1: code never wraps**), 122 task 04 (image docs: `content` = `summary` = the vision description), 148/163 (page strip + the Text/Pages toggle).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An image document's description — and a plain-text (`.txt`) document's body — render as **wrapping prose** inside the 72rem reading column on both viewer surfaces (the `/document.html` page and the in-page modal), so a long sentence is read by scrolling vertically like every other text surface in the app. Code and code-ish content keep their horizontal scroller exactly as phase 116's locked A1 decided.
|
||||||
|
|
||||||
|
## Why it scrolls today (traced 2026-10-08)
|
||||||
|
`renderDocument` has two content branches: `format ∈ {md, markdown}` → `<div class="doc-md">` (wraps, centered 72rem column), **everything else** → `<pre class="doc-raw">`, and `.doc-raw` is `white-space: pre` + `overflow-x: auto`. An image document's `format` is its extension (`png`, `webp`, …) and its `content` is the model's prose description (phase 122 task 03), so the description lands in the `pre` and each generated line becomes one unbreakable line → horizontal scrollbar. A `.txt` document is the same shape.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — the prose set is one named constant (task 01, LOCKED A1).** `frontend/assets/document.js`:
|
||||||
|
```js
|
||||||
|
const PROSE_FORMATS = new Set(["txt", "text"]);
|
||||||
|
function isProseDoc(doc) { return doc.is_image === true || PROSE_FORMATS.has(doc.format); }
|
||||||
|
```
|
||||||
|
`renderDocument`'s content branch becomes three-way: markdown → `.doc-md` (untouched) → prose → `<div class="doc-prose">` (NEW) → everything else → `<pre class="doc-raw">` (untouched). `is_image` wins regardless of format (an image doc is prose by construction, phase 122 A3). `doc.format` values that stay in `.doc-raw`: `py`, `yaml`, `yml`, `json`, `j2`, the quadlet family, `pdf`, and every unknown/extension-less format.
|
||||||
|
|
||||||
|
**D2 — `.doc-prose` is `.doc-md`'s column with prose rhythm (task 01, LOCKED A2).** `frontend/assets/styles.css`, adjacent to `.doc-raw`:
|
||||||
|
`width: 100%; max-width: var(--chat-column); margin-inline: auto; background: var(--surface); border: 1px solid var(--line); border-radius: var(--radius); box-shadow: var(--shadow); padding: 1.5rem 1.75rem; color: var(--ink); font-family: inherit; white-space: pre-wrap; overflow-wrap: anywhere; line-height: 1.55;`
|
||||||
|
The content stays a **text node** (`textContent`, never `innerHTML` — the XSS contract is untouched; the description is model prose with literal newlines, so `pre-wrap` keeps paragraph breaks without parsing markdown). Tokens only (the `:root`-drift test parses built-ins from this file — zero literals). No animation (nothing for `prefers-reduced-motion` to still).
|
||||||
|
|
||||||
|
**D3 — the code contract stays locked and gets pinned (task 02, LOCKED A3).** Phase 116's locked A1 ("keep horizontal scroll (NO line-wrap) for the code content") survives verbatim: `.doc-raw`, `.bubble pre`, `.md-table-wrap` are byte-identical, and phase 116 task 01's themed scrollbar rules (`.doc-modal .doc-raw`) keep applying to what still scrolls. The unit test asserts the exact prose set, so a future format cannot drift into prose by accident.
|
||||||
|
|
||||||
|
**D4 — verification (task 03).** Dedicated Playwright suite `tests/e2e/test_prose_word_wrap.py` (isolated run, deterministic mock LLM, the seeding idiom of `tests/e2e/test_image_documents.py` / `tests/e2e/test_document_viewer.py`): for a seeded image doc AND a seeded `.txt` doc with a > 200-char single line, assert in **both** surfaces (page + modal) that the prose block (a) contains the full text, (b) has `scrollWidth <= clientWidth + 1` (no horizontal overflow — the TODO's literal ask), (c) is taller than one line (proof it wrapped), and that a seeded `.py`/`yaml` doc STILL reports `scrollWidth > clientWidth` (the locked code path).
|
||||||
|
|
||||||
|
**Deferred, recorded, not silently dropped:** the same `pre`-scroll applies to **extracted PDF text** (`format == "pdf"` → `.doc-raw`, e.g. the IKEA-manual corpus). Wrapping it means touching `pre.doc-raw` assertions in three completed suites (`test_pdf_pages_in_sources.py`, `test_pdf_strip_additive_selfheal.py`, `test_garbage_pdf.py`) and the phase-163 strip contract — an owner call, one-line change once made (`PROSE_FORMATS.add("pdf")` + those pins). This phase deliberately does NOT change PDF rendering (no behavior change in completed phases without owner sign-off).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `122_image_documents` (complete) — image docs: `content` = `summary` = vision description, `is_image` on the content payload, `imageSummaryIsContent` suppression.
|
||||||
|
- `116_modal_scrollbar_theme` (complete) — **locked A1** (code keeps horizontal scroll) and the themed scrollbar this phase must leave working.
|
||||||
|
- `163_pdf_strip_additive_selfheal` (complete) — the `.doc-views` wrapper + Text/Pages toggle: the prose block mounts inside the same `slot` the `.doc-raw` path used, so `[data-view="pages"]` hiding must keep working (add `.doc-prose` to the hide list — task 01, it is the additive/PDF case's text view).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_doc_prose_block.md` — the prose-set constant + `renderDocument`'s third branch + `.doc-prose` CSS (page + modal, strip hide list) + unit pins.
|
||||||
|
2. `02_raw_scroll_pins.md` — pin the untouched code scrollers (`.doc-raw`, `.bubble pre`, `.md-table-wrap`, phase-116 scrollbar), pin the exact prose set + the XSS/textContent rule.
|
||||||
|
3. `03_e2e_prose_wrap.md` — `tests/e2e/test_prose_word_wrap.py` + the full gates + the docs pass + the commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (extend, following the house CSS/JS source-assertion style of `tests/unit/test_image_documents.py`, `tests/unit/test_viewer_additive_toggle.py`, `tests/unit/test_document_viewer.py`): `isProseDoc` truth table (image × format, txt/text, py/yaml/json/j2/quadlet/pdf/unknown/extension-less); `renderDocument` output shape for each branch (`className`, `textContent`, no `innerHTML`); the `.doc-prose` rule contains `pre-wrap` + `overflow-wrap: anywhere` + `var(--chat-column)` and zero color literals; the `.doc-raw` rule still contains `white-space: pre` + `overflow-x: auto`.
|
||||||
|
- E2E: `tests/e2e/test_prose_word_wrap.py` green **in isolation** (`uv run pytest tests/e2e/test_prose_word_wrap.py -v --no-cov`); `tests/e2e/test_image_documents.py`, `test_document_viewer.py`, `test_pdf_pages_in_sources.py`, `test_pdf_strip_additive_selfheal.py`, `test_edit_summaries.py`, `test_summary_in_viewer.py` stay green (PDF/raw paths unchanged).
|
||||||
|
- Coverage: **>90%** on `app/` (this phase is frontend+CSS only; the gate must not move).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An image document's description renders in `.doc-prose`, wraps at the reading column, and shows the FULL description (no clamp, no horizontal scrollbar) on both `/document.html` and the modal.
|
||||||
|
- [ ] A `.txt` document wraps the same way; `py`/`yaml`/`json`/`j2`/quadlet/`pdf`/unknown formats still render in `.doc-raw` with the horizontal scroller and the themed modal scrollbar.
|
||||||
|
- [ ] No `innerHTML` for document content anywhere in the new path; contrast pairs are the existing AA token pairs.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean; the phase E2E green in isolation.
|
||||||
|
- [ ] One `--no-gpg-sign` Conventional Commit; the phase dir moves to `complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — the prose set is exactly `{is_image: true} ∪ {format ∈ txt, text}`** (owner-confirmed 2026-10-07: "image descriptions + plain-text prose"); code and code-ish formats keep scrolling.
|
||||||
|
- **A2 — `.doc-prose` is a text-node `pre-wrap` block in the 72rem column** (no markdown parsing of a description — a description is prose, not markup; the phase-122 rule).
|
||||||
|
- **A3 — phase 116's locked A1 is NOT superseded**: code content keeps `white-space: pre` + `overflow-x: auto`.
|
||||||
|
- **A4 — extracted PDF text stays in `.doc-raw` for now** (see "Deferred" above) — an owner call, not an executor choice.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(viewer): wrap image descriptions and plain-text docs as prose instead of scrolling horizontally"
|
||||||
|
```
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 01 — The prose content block: `PROSE_FORMATS`, `renderDocument`'s third branch, `.doc-prose` (phase 165 D1/D2)
|
||||||
|
|
||||||
|
**Phase:** `165_prose_word_wrap` · **Source:** `TODO.md:3` — "Need word wrap for image explanations and other long text rather than scrolling horizontally"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The viewer renders an image document's description and a `.txt` document's body as a wrapping prose block (`.doc-prose`) instead of `<pre class="doc-raw">`, on both the `/document.html` page and the in-page modal, with the markdown and raw branches untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document.js` (module scope, above `renderDocument`) — the ONE prose rule, phase 165 D1:
|
||||||
|
```js
|
||||||
|
const PROSE_FORMATS = new Set(["txt", "text"]);
|
||||||
|
function isProseDoc(doc) {
|
||||||
|
return doc.is_image === true || PROSE_FORMATS.has(doc.format);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Docstring: the image arm is unconditional (phase 122 A3 — an image doc's `content` IS the vision description, prose by construction, whatever its extension); `format` comes from the server's `doc_format()` (lowercased suffix). Name the TODO defect in the comment ("the description used to land in the `pre` and each generated line became one unbreakable line").
|
||||||
|
2. `frontend/assets/document.js` — `renderDocument`'s content branch (today: `if (doc.format === "md" || doc.format === "markdown") { … .doc-md … } else { … .doc-raw … }`): make it three-way and put the prose arm **between** them —
|
||||||
|
- markdown → the existing `.doc-md` div, byte-identical;
|
||||||
|
- `isProseDoc(doc)` → `<div class="doc-prose">` with `textContent = doc.content` (a **text node**: the XSS contract is unchanged; `pre-wrap` carries the newlines, no markdown parsing — locked A2);
|
||||||
|
- else → the existing `<pre class="doc-raw">`, byte-identical.
|
||||||
|
The prose block appends into the SAME `slot` variable the raw branch uses, so the phase-122 image block (`docImageBlock`) stays FIRST, the phase-36 summary panel (when not suppressed by `imageSummaryIsContent`) stays between them, and the phase-163 additive strip (`docPagesStrip`) still mounts last inside `.doc-views`.
|
||||||
|
3. `frontend/assets/styles.css` — new `.doc-prose` rule immediately after `.doc-raw` (~L5229), phase 165 D2, exact property set:
|
||||||
|
`width: 100%; max-width: var(--chat-column); margin-inline: auto; background: var(--surface); border: 1px solid var(--line); border-radius: var(--radius); box-shadow: var(--shadow); padding: 1.5rem 1.75rem; color: var(--ink); white-space: pre-wrap; overflow-wrap: anywhere; line-height: 1.55;`
|
||||||
|
(`--ink` on `--surface` is the documented 14.5:1 pair; **zero color literals** — the `:root`-drift unit test parses built-ins from this file.) Comment block cites: TODO L3, the `.doc-md` column precedent, `.doc-raw`'s `pre` as the defect, locked A1/A2, "static content — no animation".
|
||||||
|
4. `frontend/assets/styles.css:5361-5364` — add `.doc-views[data-view="pages"] .doc-prose` to the existing hide list (the four-selector rule that hides the text view behind the page strip). Defensive today (image/`txt` docs are never page-qualifying), load-bearing if the owner later adds `pdf` to `PROSE_FORMATS` — say so in the comment.
|
||||||
|
5. `tests/unit/test_image_documents.py` (extend; reuse its `renderDocument` harness) — the image-doc content branch now yields `className === "doc-prose"`, `textContent === description`, and NOT a `doc-raw` sibling; the summary-suppression case (`summary === content`) is unchanged.
|
||||||
|
6. `tests/unit/test_document_viewer.py` (extend) — the format table: `txt` / `text` → `doc-prose`; `md`/`markdown` → `doc-md` (innerHTML through `renderMarkdown`, unchanged); `py`, `yaml`, `json`, `j2`, `conf`, `pdf`, `""`, an unknown suffix → `doc-raw` (proves locked A1/A3 unchanged); the never-`innerHTML` pin for the prose arm (`assert "innerHTML" not in prose_branch`).
|
||||||
|
7. `tests/unit/test_viewer_additive_toggle.py` (extend) — the `[data-view="pages"]` hide-list test enumerates `.doc-summary`, `.doc-md`, `.doc-raw`, `.doc-image`: add `.doc-prose` to the expected selectors and assert the rule stays ONE grouped declaration.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: items 5–7 + the CSS pins (`.doc-prose` contains `pre-wrap`, `overflow-wrap: anywhere`, `var(--chat-column)`; `.doc-raw` still contains `white-space: pre` and `overflow-x: auto`).
|
||||||
|
- Coverage: **>90%** on `app/` unchanged (no `app/` change in this task).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `isProseDoc` is the only place the prose rule lives (grep-verified: no second `txt` literal in the content branch).
|
||||||
|
- [ ] Image docs and `.txt` docs render `.doc-prose` on page + modal (both call `renderDocument`); every other format's markup is byte-identical to pre-task.
|
||||||
|
- [ ] No `innerHTML` on document content in the new path; full `uv run pytest` green.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 02 — Pin the untouched scrollers: code keeps horizontal scroll (phase 165 D3, LOCKED A1/A3)
|
||||||
|
|
||||||
|
**Phase:** `165_prose_word_wrap` · **Source:** `TODO.md:3` — the "other long text" half of the item, bounded by phase 116's locked A1 ("keep horizontal scroll (NO line-wrap) for the code content … wrapping breaks code readability")
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The word-wrap fix is provably scoped: every surface that must keep scrolling horizontally still declares it, the prose set is pinned as data, and the chat/table scrollers are asserted byte-identical — so "we fixed wrapping" can never later mean "we wrapped code" or "we wrapped wide tables" by accident.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `tests/unit/test_prose_word_wrap.py` (the house CSS/JS source-assertion style — read `tests/unit/test_markdown_tables.py` and `tests/unit/test_date_editor.py` first; they parse `frontend/assets/styles.css` and the JS files as text):
|
||||||
|
- **the prose set is exactly `{txt, text}` + `is_image`**: assert the `PROSE_FORMATS` literal contains exactly `"txt"`, `"text"` (a set-membership assertion over the source line, so adding a format is a deliberate, test-editing act); assert `isProseDoc` reads `doc.is_image` first;
|
||||||
|
- **code scrollers present and unchanged**: `.doc-raw {` block still contains `white-space: pre` and `overflow-x: auto`; `.bubble pre` still contains `overflow-x: auto` (no `white-space` added); `.md-table-wrap { overflow-x: auto; }` and `.md-table { … min-width: max-content … }` still match the phase-44 declaration;
|
||||||
|
- **phase 116 A1 stays locked**: the four `.doc-modal .doc-raw` scrollbar rules (`scrollbar-width`, `scrollbar-color`, `::-webkit-scrollbar` ×3) are all still present and still scoped to `.doc-modal`;
|
||||||
|
- **`.doc-prose` never reaches for the raw scroller**: the `.doc-prose` block contains no `overflow-x` and no `white-space: pre;` (only `pre-wrap`).
|
||||||
|
2. `tests/unit/test_prose_word_wrap.py` — the contrast/AA pin for the new block: `.doc-prose`'s `color` is `var(--ink)` and its background `var(--surface)` (the documented 14.5:1 pair), and the block contains no hex/`rgb(` literal (the phase-92 zero-literal invariant).
|
||||||
|
3. Repo hygiene pass: `grep -rn "doc-raw" frontend/` — confirm the only remaining writers of `doc-raw` are the raw branch of `renderDocument` and the phase-116 CSS; if `frontend/doc-edit.html` / `doc-edit.js` / `shared.*` render document content, verify they are NOT a second content renderer that needs the same rule (they are not expected to be — record the finding in the test's docstring).
|
||||||
|
4. ASSUMPTION: PDF-extracted text (`format == "pdf"`) is deliberately OUT of the prose set — it stays in `.doc-raw` because three completed suites pin `pre.doc-raw` for PDFs and the phase-163 strip contract; the deferral is recorded in `00_phase.md` ("Deferred") and in this test's docstring, so it is an owner decision on record, not an oversight.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new file (all assertions are static-source assertions — no app dependency); run the phase-116/44/163-owning unit files to prove nothing moved: `uv run pytest tests/unit -k "additive_toggle or markdown_tables or date_editor or image_documents or document_viewer" -v`.
|
||||||
|
- Coverage: **>90%** on `app/` unchanged.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_prose_word_wrap.py -v` green.
|
||||||
|
- [ ] The pin set proves: code/raw/tables keep `overflow-x: auto`; `.doc-prose` has none; the prose set is `{txt, text}` + image.
|
||||||
|
- [ ] Full `uv run pytest` green; ruff + pyright clean; no `app/` change.
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 03 — The wrapping proof: `tests/e2e/test_prose_word_wrap.py`, migrating the two suites that pinned `pre.doc-raw` for prose, final gates (phase 165 D4)
|
||||||
|
|
||||||
|
**Phase:** `165_prose_word_wrap` · **Source:** `TODO.md:3` — "Need word wrap for image explanations and other long text rather than scrolling horizontally"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A dedicated, isolated Playwright suite proves the TODO's literal claim (no horizontal scrollbar, full text visible, both surfaces, both prose classes) and the two completed suites whose fixtures ARE prose are migrated to the new contract — while the code path that keeps scrolling is still proven to scroll.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `tests/e2e/test_prose_word_wrap.py` — module-scoped app fixtures + `_reset_db`/seed idiom copied from `tests/e2e/test_document_viewer.py` (read it first; `_seed_long_line_doc` shows the direct-seed pattern) and the image seeding of `tests/e2e/test_image_documents.py`:
|
||||||
|
- **seed**: one image doc (`is_image`, `content` = `summary` = a ~900-char single-paragraph vision description with `image_path` + a served `image_url`) and one `notes/prose-long.txt` doc whose content is a single ~900-char line plus a second paragraph (proves BOTH intra-line wrap and paragraph breaks); plus one `notes/code-long.py` doc with one ~900-char line (the negative control).
|
||||||
|
- **assert, per prose doc, on BOTH surfaces** (`/document.html?source=…&path=…` and the same-page modal via the Sources row / chip click):
|
||||||
|
- `#doc-content .doc-prose` (and `#doc-modal-content .doc-prose`) count == 1 and `to_have_text(<full description>)` — nothing clamped, nothing truncated;
|
||||||
|
- `el.scrollWidth <= el.clientWidth + 1` — **the TODO's literal defect: no horizontal overflow**;
|
||||||
|
- `el.getBoundingClientRect().height > 2.2 × line-height` — the text really wrapped;
|
||||||
|
- the computed `white-space === "pre-wrap"` and the block's max-width resolves to the 72rem column (never wider than the container);
|
||||||
|
- the image doc: the `<img class="doc-image-img">` still renders ABOVE the prose block (phase 122 ordering) and the summary panel stays suppressed (`summary === content`).
|
||||||
|
- **negative control**: `notes/code-long.py` renders `pre.doc-raw` with `scrollWidth > clientWidth` (phase 116 A1 alive).
|
||||||
|
- `--no-cov` isolation run per AGENTS.md rule 9.
|
||||||
|
2. `tests/e2e/test_document_viewer.py` — MIGRATE the phase-116 scrollbar proof: `_seed_long_line_doc()` seeds `notes/long-line.txt`, which is now prose. Change the seed to `notes/long-line.json` (same long line, a code-ish format), keep every assertion (`pre.doc-raw`, real `scrollWidth > clientWidth`, the themed `scrollbar-width`/`scrollbar-color` reading, the screenshots) and update the docstring + the file's outline lines 27/42 so they say the raw block is a CODE doc and that prose wraps (cross-reference phase 165). Do NOT weaken the overflow assertion.
|
||||||
|
3. `tests/e2e/test_image_documents.py:390,410` — MIGRATE the viewer assertion: the image doc's description now lives in `.doc-prose` (keep `to_have_text(seeded["description"])`), and the docstring line that says "the plain-content path (`pre.doc-raw`)" becomes the prose path; the rest of the suite (thumbnail, alt, chips, figure) is untouched.
|
||||||
|
4. `tests/unit/test_image_documents.py:1042` — update the comment/assert pair that names "the existing plain-content path — doc-raw via textContent" to the prose path (task 01 already added the shape assertion; here just remove the stale claim).
|
||||||
|
5. Docs pass + gates: `README.md` — the document-viewer section gains one sentence (prose formats wrap, code keeps the horizontal scroller; the phase-116 themed scrollbar applies to what still scrolls). `AGENTS.md` — verify the §7 UI list needs no change (the 72rem reading column already covers this). Then the full gate set: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (> 90%), `uv run ruff check . && uv run pyright`, the phase E2E in isolation, plus the regression set that owns the surfaces touched: `tests/e2e/test_document_viewer.py`, `tests/e2e/test_image_documents.py`, `tests/e2e/test_pdf_pages_in_sources.py`, `tests/e2e/test_pdf_strip_additive_selfheal.py`, `tests/e2e/test_edit_summaries.py`, `tests/e2e/test_summary_in_viewer.py`, `tests/e2e/test_chat_answer_clusters.py` (each in isolation).
|
||||||
|
6. Move `.agents/phases/todo/165_prose_word_wrap/` → `.agents/phases/complete/` and commit with the `00_phase.md` message (AGENTS.md rule 8, `--no-gpg-sign`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the new suite green in isolation + the seven regression suites green in isolation (the migration in items 2–3 must keep each suite's ORIGINAL claim true, just on the right element).
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` change; the gate must not move).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_prose_word_wrap.py -v --no-cov` green — image description and `.txt` body wrap with zero horizontal overflow on page + modal, full text present.
|
||||||
|
- [ ] The `code-long.py` control still overflows horizontally (locked A1 pinned end-to-end, not just in CSS).
|
||||||
|
- [ ] `test_document_viewer.py` and `test_image_documents.py` pass with their phase-116/122 claims intact on the new elements — no assertion deleted, only re-pointed.
|
||||||
|
- [ ] `uv run pytest` green, coverage > 90%, ruff + pyright clean, one atomic `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# Phase 166 — No empty answer box before the first token
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 — "There's an empty box where the LLM response will go that shows up before the LLM responds. It looks ugly." (the owner attached `image.png`: mid-turn the chat shows `Thinking` ▸, `Tool calls (2)` ▸, an expanded `Thinking` run with text, and BELOW it a bordered, padded, empty rounded box ≈ 40px tall)
|
||||||
|
**Story:** n/a (owner-directed UI defect)
|
||||||
|
**Context:** `frontend/assets/styles.css` (`.bubble` ~L501, `.bubble typing`, `.msg-body` flex gap ~L489), `frontend/assets/app.js` (`addMessage` L1386, `addTyping` L1433, `ensureReasoningWrap` L1610-1625, the `if (!wrap) wrap = addMessage("brain", "")` sites at ~4063 thinking / ~4164 tool / ~4269 first delta, `finalizeFailedTurn` ~3841, the restore path ~2864), `frontend/assets/shared.js:113` (the same `.msg/.msg-body/.bubble` structure), phases 06 (loading feedback), 17/117/158/161 (the per-run timeline that inserts BEFORE `.bubble`), 120/164 (failed-turn bubbles).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
While a turn is in flight, nothing paints an empty answer bubble: the box appears **at the first answer token**, and every visible-feedback affordance the app already owns (the typing indicator, the reasoning container with its live status, `#send-status`, the elapsed clock) keeps working exactly as before. Same rule covers the shared page's empty persisted answer, which renders the same empty box today.
|
||||||
|
|
||||||
|
## Why the box exists (traced 2026-10-08)
|
||||||
|
The first `thinking`/`tool` frame of a turn calls `addMessage("brain", "")` (`app.js:4063` / `:4164`) — an EMPTY bubble is created on purpose, because `ensureReasoningWrap` uses it as the DOM anchor (`body.insertBefore(cont, body.querySelector(".bubble"))`). `.bubble` carries `padding: 0.7rem 1rem` + `border` + `background` + `box-shadow`, so an empty one paints a box; the `.msg-body` flex gap adds the visible separation. `renderMarkdown("")` returns `""` (verified), so the restore path produces the same empty element — and the shared page (`shared.js:113`) too.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — hide it in CSS, do not restructure the JS (task 01, LOCKED A1).** `frontend/assets/styles.css`, in the Messages block right after `.bubble p`:
|
||||||
|
```css
|
||||||
|
.msg .bubble:empty { display: none; }
|
||||||
|
```
|
||||||
|
`:empty` matches only an element with NO child nodes, which is exactly what `addMessage("brain", "")` produces (`<div class="bubble"></div>`); the first delta writes `innerHTML` and the box returns on the same frame. `display: none` also drops the bubble from the `.msg-body` flex layout, so the gap collapses (no orphan whitespace). The typing indicator is `.bubble typing` WITH three `<span>` children, so it is never `:empty` and its animation is untouched. Scoped to `.msg .bubble` — the viewer's `.doc-*` blocks and the composer are unaffected.
|
||||||
|
|
||||||
|
**D2 — the JS anchor stays (task 02).** Creating the empty wrap and anchoring the reasoning container before it is the phase-158/161 contract, pinned by `tests/unit/test_frontend_tool_states.py:61`, `tests/unit/test_thinking_timeline.py:475,522`, `tests/unit/test_frontend_edit_message.py:971,1072,1177`. Changing it (create-the-bubble-at-first-delta) would re-own those anchors for zero user-visible gain over D1 — so `app.js` is **not** modified by this phase; task 02's job is to prove the feedback contract holds with the box hidden.
|
||||||
|
|
||||||
|
**D3 — §7.4 feedback contract, verified not assumed (task 02).** With the box hidden, a live turn still shows, in order: `#typing-indicator` (role=status, `TYPING_LABEL()`, elapsed-seconds suffix from `thinkingClock`) → the open `details.reasoning` container with its live status span ("Thinking…" / "Calling <name>…" / "N steps") → the answer bubble at the first delta. Nothing else in the app depends on the empty bubble being painted: `scrollReveal` measures the `.msg` wrap (not the bubble), the phase-161 window pins measure `.reasoning-body`, the phase-18 follow-bottom logic is user-intent-gated, and `#messages` remains the single `aria-live="polite"` region (an `aria-live` announcement fires when content lands — at which point the bubble is visible again).
|
||||||
|
|
||||||
|
**D4 — verification (task 03).** Dedicated suite `tests/e2e/test_no_empty_box.py`: stub `POST /api/chat` with a controllable SSE stream (`thinking` → `tool` → a pause → `delta`), assert `page.locator(".msg.brain .bubble").first` is `not_to_be_visible()` (or has zero box) during the pre-delta window while `#typing-indicator` / `details.reasoning[open]` / `#send-label` ARE in their expected states, and that after the first `delta` the same locator becomes visible and holds the token. Plus a restore case (a saved chat whose last brain record has empty text renders no empty box) and the shared-page case.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `161_reasoning_bubble` + `158_thinking_timeline` (complete) — the container-before-the-bubble anchor this phase keeps, and the unit pins that must stay green untouched.
|
||||||
|
- `06_story_loading_feedback` + `109_turn_progress_loader` + `120_failed_turn_retry` (complete) — the visible-feedback states that must not regress when the box stops painting.
|
||||||
|
- `130_chat_answer_clusters` (complete) — `.msg-meta` rows are siblings of `.bubble`, never children: hiding an empty bubble never hides a chip/action row (pin it).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_hide_empty_bubble.md` — the CSS rule + its provenance comment + the unit pins (styles.css source assertions + the typing-indicator non-emptiness pin).
|
||||||
|
2. `02_feedback_contract_check.md` — prove §7.4 endures: the existing state-machine pins stay green, the shared/restore empty-answer corners are covered, and nothing measures the hidden box.
|
||||||
|
3. `03_e2e_no_empty_box.md` — `tests/e2e/test_no_empty_box.py` + the regression set + the full gates + the commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: new `tests/unit/test_no_empty_answer_bubble.py` (the house "read the source file, assert on the rules" style — see `tests/unit/test_thinking_scroll.py`): the `.msg .bubble:empty { display: none; }` declaration exists exactly once and sits in the Messages block; `.bubble` itself keeps `padding`/`border`/`background`/`box-shadow` (the rule hides, it does not restyle); `addTyping`'s markup still contains the three `<span>` dots (⇒ never `:empty`); `app.js` still contains the three `addMessage("brain", "")` anchors (D2 — the rule is CSS-only).
|
||||||
|
- E2E: `tests/e2e/test_no_empty_box.py` green **in isolation**; the owning regression set green in isolation: `test_thinking_timeline.py`, `test_turn_progress_loader.py`, `test_chat_history.py`, `test_share_chat.py`, `test_save_share_ux.py`, `test_failed_turn_retry.py`, `test_chat_answer_clusters.py`, `test_chat_error_honesty.py`.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` change; the gate must not move).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Mid-turn (thinking and/or tool frames, zero deltas) NO empty answer box paints — the reasoning container and the typing indicator are the only brain-side boxes.
|
||||||
|
- [ ] The answer bubble appears with the first delta, in the same DOM position, with the source chips / actions / figures below it exactly as phase 130 arranged them.
|
||||||
|
- [ ] A restored or shared answer whose persisted text is empty paints no empty box either; every non-empty bubble is byte-identical.
|
||||||
|
- [ ] No `app.js` change; the phase-158/161 anchor unit pins pass untouched; `uv run pytest` green, coverage > 90%, ruff + pyright clean.
|
||||||
|
- [ ] One `--no-gpg-sign` Conventional Commit; the phase dir moves to `complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — CSS-only suppression** (`.msg .bubble:empty { display: none }`); the JS continues to create the empty bubble as the reasoning anchor (owner-confirmed 2026-10-07: "hide until the first token", not "restructure the anchor").
|
||||||
|
- **A2 — nothing replaces the box**: no skeleton, no shimmer — the existing loading feedback (typing indicator → reasoning container → status label) is the turn's visible state (PLAN §7.4), owner-confirmed 2026-10-07.
|
||||||
|
- **A3 — the rule extends to the shared page and the restore path** (same `.msg .bubble` structure; an empty persisted answer is the same defect), owner-confirmed 2026-10-07.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): stop painting an empty answer bubble before the first token"
|
||||||
|
```
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Task 01 — `.msg .bubble:empty { display: none }`: the empty answer bubble stops painting (phase 166 D1)
|
||||||
|
|
||||||
|
**Phase:** `166_hide_empty_answer_bubble` · **Source:** `TODO.md:4` — "There's an empty box where the LLM response will go that shows up before the LLM responds. It looks ugly." (attached `image.png`: Thinking ▸ / Tool calls (2) / an expanded Thinking run, then an empty bordered box)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One CSS rule suppresses the empty answer bubble during a turn (and for an empty persisted answer), with the box returning the instant the first token lands — no JS change, no restyle of a non-empty bubble, no change to any loading affordance.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — in the `---------- Messages ----------` block, directly after `.bubble p { margin: 0.2rem 0; }` (~L509), add:
|
||||||
|
```css
|
||||||
|
/* Phase 166 (TODO L4): the answer bubble an in-flight turn creates is
|
||||||
|
EMPTY by construction — app.js:4063/:4164 call addMessage("brain","")
|
||||||
|
so ensureReasoningWrap has an anchor to insert the per-run timeline
|
||||||
|
BEFORE. With .bubble's own padding + border + background + shadow an
|
||||||
|
empty one paints a ~40px box, which is the "empty box before the LLM
|
||||||
|
responds" the owner reported. :empty matches only a child-less
|
||||||
|
element — exactly innerHTML === "" (renderMarkdown("") is "", so the
|
||||||
|
restore and the shared page take the same path) — and the first
|
||||||
|
delta writes content, so the box appears AT the first token.
|
||||||
|
display:none also drops the flex item, so .msg-body's gap collapses.
|
||||||
|
The typing indicator is never :empty (three dot <span>s) — its
|
||||||
|
animation is untouched. Scoped to .msg .bubble: the viewer's .doc-*
|
||||||
|
blocks and the composer never match. */
|
||||||
|
.msg .bubble:empty { display: none; }
|
||||||
|
```
|
||||||
|
No new tokens, no animation (nothing for `prefers-reduced-motion` to still).
|
||||||
|
2. `tests/unit/test_no_empty_answer_bubble.py` (new — static source-assertion style, copy the harness of `tests/unit/test_thinking_scroll.py` / `tests/unit/test_markdown_tables.py`):
|
||||||
|
- the declaration appears **exactly once**, spelled `.msg .bubble:empty { display: none; }`, and sits after `.bubble {` and before `.bubble pre {` (provenance ordering, the `test_date_editor.py:411` `css.find(...)` idiom);
|
||||||
|
- the `.bubble {` rule still declares `padding`, `border-radius`, `background`, `border`, `box-shadow` (the fix HIDES, it never restyles);
|
||||||
|
- `.bubble typing` / the typing-dot rules are still present and unchanged (the indicator stays visible);
|
||||||
|
- `frontend/assets/app.js` still contains all three `addMessage("brain", "")` call sites and `ensureReasoningWrap` still does `body.insertBefore(cont, body.querySelector(".bubble"))` (D2: the anchor survives — this is the pin that stops a future "cleanup" from moving the fix into JS and breaking the timeline);
|
||||||
|
- `frontend/assets/markdown.js` — `renderMarkdown("")` and `renderMarkdown(" ")` both return `""` (execute the file the way the existing markdown unit tests do), which is WHY the restore/shared paths are covered by the same rule;
|
||||||
|
- no CDN / no `url(http` introduced (the AGENTS.md rule-6 idiom these files already assert).
|
||||||
|
3. `frontend/assets/app.js` — **no change** (explicitly). If the executor believes a JS change is needed, stop and flag it: LOCKED A1 (the reasoning-container anchor and the phase-158/161 pins are the reason this is CSS-only).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new file; then `uv run pytest tests/unit -k "thinking_timeline or tool_states or edit_message or frontend" -v` — the phase-158/161/159 anchor pins pass untouched.
|
||||||
|
- Coverage: **>90%** on `app/` unchanged (no `app/` change).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The rule exists exactly once, scoped, with the provenance comment naming TODO L4, `image.png`, and the anchor it preserves.
|
||||||
|
- [ ] `addTyping()`'s bubble is provably non-empty (unit-pinned) so the loading state never disappears.
|
||||||
|
- [ ] `git diff` for this task touches `frontend/assets/styles.css` + one new unit test file only.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — §7.4 audit: the turn is never feedback-less with the box hidden (phase 166 D3)
|
||||||
|
|
||||||
|
**Phase:** `166_hide_empty_answer_bubble` · **Source:** `TODO.md:4` — hiding the empty box must not remove the owner's only mid-turn signal; PLAN §7.4 ("never stale": every state has a defined UI) and AGENTS.md rule 10
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove — with tests, not prose — that every mid-turn state still has a visible affordance with the empty bubble suppressed, that no code path measures the hidden box, and that the restore and shared-page corners render no empty box either.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_no_empty_answer_bubble.py` (extend) — the **feedback inventory** pins, each naming the state it guarantees:
|
||||||
|
- `addTyping()` markup contains `role="status"`, `aria-label="${TYPING_LABEL()}"` and the three `<span></span>` dots (the pre-thinking state);
|
||||||
|
- `thinkingClock` still writes the `Ns` `.typing-elapsed` suffix into `#typing-indicator .bubble` (the long-tool-call state) — and note in the comment that this bubble is never `:empty`, so the clock keeps painting;
|
||||||
|
- `setReasoningStatus("Thinking…")` / `"Calling "` / the settle `"N steps"` literals are still wired at their three call sites (the thinking/tool state), and `ensureReasoningWrap` sets `cont.open = true` (the container is visible while live);
|
||||||
|
- `#send-status` / `#send-label` state copy is untouched (the composer-side signal), and `#messages` keeps `aria-live="polite"` (`frontend/index.html:167`) — the ONE conversation live region; an announcement fires when content lands, at which point the bubble is visible again (comment this reasoning).
|
||||||
|
2. `tests/unit/test_no_empty_answer_bubble.py` (extend) — the **"nothing measures the hidden box"** audit, executed as source assertions with the finding recorded per site:
|
||||||
|
- `scrollReveal(wrap)` receives the `.msg` WRAP (not the bubble) — pin the call site `if (scroll) scrollReveal(wrap);`;
|
||||||
|
- the phase-161 window pins operate on `.reasoning-body` (`isReasoningPinned` / `scrollReasoningToBottom`) — pin that neither takes a `.bubble`;
|
||||||
|
- `.bubble` is only ever read for CONTENT (`bubble.textContent`, `bubble.innerHTML`, `bubble.querySelector(".msg-image")`, `bubble.classList`) in `app.js`/`shared.js` — grep-assert there is no `getBoundingClientRect`/`offsetHeight`/`clientHeight` on a `.bubble` selector;
|
||||||
|
- `.msg-meta` (chips, actions, figures, `.failed-note`, `.stopped-note`) are appended to `.msg-body` — pin one append site each so a future refactor cannot move them INTO the bubble (where hiding an empty one would hide them).
|
||||||
|
3. `tests/unit/test_no_empty_answer_bubble.py` (extend) — the **restore / shared corners**:
|
||||||
|
- `app.js` restore renders brain bubbles through `addMessage("brain", renderMarkdown(m.text), true)` — with `m.text === ""` the bubble is child-less ⇒ `:empty` ⇒ hidden (assert the call shape, plus the `renderMarkdown("")` fact already pinned in task 01);
|
||||||
|
- `shared.js` builds the identical `<div class="bubble">${html}</div>` structure (its line ~113), so the same `.msg .bubble:empty` scope covers the shared page — assert the rule's scope prefix `.msg ` matches both (the shared page uses the same `.msg`/`.msg-body` classes).
|
||||||
|
4. Findings note: append a short `AUDIT NOTE:` block to the test module docstring listing the four mid-turn states and the affordance that carries each after this phase (idle/typing → typing dots + clock; thinking/tool → reasoning container + status span; first token → the bubble appears; terminal → answer/deflection/stopped/failed copy). This is the record the next agent reads instead of re-deriving it.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the extended file + `uv run pytest tests/unit -k "frontend or reasoning or thinking or tool" -v` green; no `app/` change, coverage gate unmoved.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every mid-turn state has a pinned, visible affordance with the empty bubble suppressed (the four-state inventory is in the test docstring).
|
||||||
|
- [ ] No code path measures a `.bubble` box (grep-pinned), so `display:none` cannot shift a scroll or a pin.
|
||||||
|
- [ ] The restore and shared paths are covered by the same scope (pinned), with no `shared.js` change needed.
|
||||||
|
- [ ] Full `uv run pytest` green.
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 03 — `tests/e2e/test_no_empty_box.py`: the mid-turn proof, a paced mock turn, and the final gates (phase 166 D4)
|
||||||
|
|
||||||
|
**Phase:** `166_hide_empty_answer_bubble` · **Source:** `TODO.md:4` — "There's an empty box where the LLM response will go that shows up before the LLM responds. It looks ugly."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated Playwright suite proves the defect is gone with a race-free timeline: during the pre-delta window (thinking frames + a tool call, no answer token) the brain message paints NO answer box, the reasoning container and the typing/loading affordances DO paint, and the box appears exactly with the first delta. Plus the deterministic mock turn that makes that window long enough to assert, the regression set, and the phase gates.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/mock_llm.py` — a new dedicated paced trigger (the `TURN_PROGRESS_TRIGGER` precedent, its block documents the convention): `EMPTY_BUBBLE_TRIGGER = "paint the pre-answer window"` (verify the phrase appears in NO existing E2E question and NO fixture file — the phase-109 suite's disjointness check, grep before committing), streaming only, with baked-in delays so the windows are ≥ 1 s:
|
||||||
|
- request 1 (tools offered, no tool results): `TURN_*`-style local constants (defined next to the trigger, never inline magic): a ~1.2 s pre-delay, then 3 `reasoning_content` chunks at ~0.4 s (≈ 1.2 s of thinking with **no** `delta.content`), then one synthetic `ls` tool call (`call_0`, `finish_reason: "tool_calls"`) — i.e. request 1 NEVER emits a content delta;
|
||||||
|
- request 2 (a `tool`-role result present): a ~2.0 s gap, then 3 short `delta.content` chunks (0.05 s apart) + `finish_reason: "stop"`;
|
||||||
|
- the module docstring's trigger inventory gains the entry; every other suite's classification is untouched (stateless discrimination from the messages, like the phase-109 flow).
|
||||||
|
2. New `tests/e2e/test_no_empty_box.py` (module-scoped `app_server`/`app_url` + `_reset_db(mock_llm, seed=True)` + `login()` copied from `tests/e2e/test_reasoning_bubble.py`, which is the closest live-stream suite — read its `_submit`/`_wait_answer` helpers first):
|
||||||
|
- **the defect window**: submit `EMPTY_BUBBLE_TRIGGER` question; while the thinking/tool frames stream (`expect(page.locator("details.reasoning")).to_be_visible()` first, as the sync point) assert on `.msg.brain .bubble`: `be_not_visible()` AND a zero-height box (`evaluate("el => el.getBoundingClientRect().height") === 0`) — the box does not paint; assert `details.reasoning[open]` is visible, its status span carries the live literal (`Thinking…` / `Calling `), and `#send-label` reads the in-flight state;
|
||||||
|
- **the settle**: after `done`, the SAME locator is visible, `height > 0`, and contains the answer text; the container folds (`details.reasoning:not([open])`), the chips/actions rows are present below the bubble (phase 130 ordering survives);
|
||||||
|
- **no-thinking control**: a plain answer question (no thinking, no tools) shows `#typing-indicator` before the settle (the typing bubble has dots ⇒ visible) and no empty box at any point — assert `#typing-indicator .bubble` height > 0 while in flight;
|
||||||
|
- **restore**: save the answered chat, reload `/?chat=<id>` — the answered turn renders its bubble normally (non-empty), and NO `.bubble:empty` exists in the DOM (`page.locator(".msg .bubble:empty").count() === 0`);
|
||||||
|
- **shared page**: share the chat, open `/shared/<token>` — the same zero-`:empty` assertion (LOCKED A3);
|
||||||
|
- screenshot of the mid-turn state (the owner's `image.png` before/after record) into the suite's screenshot dir the way `test_document_viewer.py` does.
|
||||||
|
3. Regression run (each **in isolation**, `--no-cov`): `tests/e2e/test_thinking_timeline.py`, `test_reasoning_bubble.py`, `test_turn_progress_loader.py`, `test_chat_history.py`, `test_share_chat.py`, `test_save_share_ux.py`, `test_failed_turn_retry.py`, `test_chat_answer_clusters.py`, `test_chat_error_honesty.py`, `test_markdown_tables.py` (the `.bubble pre` scroller), `test_wide_desktop_column.py` (the chat column metrics).
|
||||||
|
4. Docs + gates: `README.md` — one sentence in the chat-turn/loading section (the answer bubble appears at the first token; mid-turn feedback is the typing indicator + the reasoning container). Then `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (> 90%), `uv run ruff check . && uv run pyright`, the phase E2E in isolation.
|
||||||
|
5. Move `.agents/phases/todo/166_hide_empty_answer_bubble/` → `.agents/phases/complete/` and commit with the `00_phase.md` message (`--no-gpg-sign`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the new suite green in isolation (its windows are ≥ 1 s by construction — no `sleep`-style flake), plus the ten regression suites above.
|
||||||
|
- Mock: the new trigger changes NO other suite (grep-proven disjoint phrase; the full E2E smoke `tests/e2e/test_smoke.py` green).
|
||||||
|
- Coverage: **>90%** on `app/` (mock + tests only here; no `app/` change in the phase).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Mid-turn: `height === 0` / not visible on `.msg.brain .bubble` while thinking + tool frames stream; the reasoning container, the status literal, and the composer state are all visible (screenshotted).
|
||||||
|
- [ ] The box appears with the first delta and holds the answer; chips/actions/figures keep their positions below it.
|
||||||
|
- [ ] Zero `.bubble:empty` elements after a restore and on the shared page.
|
||||||
|
- [ ] `uv run pytest` green, coverage > 90%, ruff + pyright clean, all ten regression suites green in isolation, one atomic `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# Phase 167 — API payload diet: gzip, a lazily-drilled KB tree, revalidation, and a measured byte budget
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L5 — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data."
|
||||||
|
**Story:** n/a (owner-directed performance defect)
|
||||||
|
**Context:** `app/api/docs.py` (`GET /api/docs/tree` L919, the pure builder `build_kb_tree` L780 / `_level_children` L671 / `_folder_counts` L626 / `_subtree_max` L760, `GET /api/docs` L129), `app/schemas.py` (`KbTree` L576, `KbTreeFolder` L501, `KbTreeSource` L539, `KbTreeFile` L447 + the `_drop_image_fields` omission precedent), `app/main.py` (middleware stack: `SessionMiddleware` → routers → `configure_caching` → `SecurityHeadersMiddleware` outermost), `app/core/caching.py` (the `/api/*` pass-through contract), `app/core/logging.py` (§9 log style), `frontend/assets/sources.js` (`loadTree` L1625, `renderLevel` L1888, `treeStats` L1703, `folderExistsIn` L1667, `resetVanishedLocation` L1646), `app/models.py` (`SourcesMeta` — deliberately NOT the ETag source, see A3), phases 94/97/98/99/106/122 (the tree contract), A15 (SSE), A19 (caching), A20 (security headers).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`GET /api/docs/tree` stops shipping 2 MB: the first load carries what the RAG view actually renders (sources + one level, with server-computed totals), deeper levels load on drill, every response the client already has revalidates to a 304, and **every** non-SSE response ships gzipped (~8-12× on this JSON). Then the same lens is pointed at the rest of the API with a byte-budget log line and a measured audit, so "any other endpoints that could load a ton of data" becomes a recorded finding with fixes for the offenders instead of a guess.
|
||||||
|
|
||||||
|
## Why it is 2 MB (traced 2026-10-08)
|
||||||
|
`list_kb_tree` returns the FULL recursive tree in one body (the deliberate phase-97 "the RAG view drills CLIENT-side, zero per-level requests" contract): every indexed document as a file node (`path`, `title`, `chunks`, `created_at`, `indexed_at`, + `is_image`/`image_url`/`summary` on image nodes) AND **every stored `folder_summaries` row inlined on its folder/source node**, with no compression anywhere in the stack (the app has no GZip middleware — `app/main.py` adds only Session / caching / security-headers; nothing sets `Content-Encoding`). At the live corpus (≈1 k documents, hundreds of folders with AI descriptions, image docs whose `summary` is a full vision description) that is ~2 MB of uncompressed JSON per Sources-page load, and the browser re-downloads it after every sync and on every shell boot.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — selective gzip, SSE never touched (task 01, LOCKED A1).** New `app/core/compression.py`, a **pure-ASGI** middleware (`BaseHTTPMiddleware` is forbidden here — it buffers):
|
||||||
|
```python
|
||||||
|
class SelectiveGZipMiddleware:
|
||||||
|
def __init__(self, app, *, minimum_size: int, skip_paths: tuple[str, ...] = ("/api/chat",)) -> None:
|
||||||
|
self.app = app
|
||||||
|
self.gzip = GZipMiddleware(app, minimum_size=minimum_size) # starlette, reused, not forked
|
||||||
|
async def __call__(self, scope, receive, send):
|
||||||
|
if scope["type"] != "http" or scope["path"] in self.skip_paths:
|
||||||
|
await self.app(scope, receive, send); return
|
||||||
|
await self.gzip(scope, receive, send)
|
||||||
|
```
|
||||||
|
Exact-path skip (never a prefix — `/api/chat-images` must still compress). Starlette's `GZipMiddleware` already: requires `Accept-Encoding: gzip`, respects `minimum_size`, skips a response that already sets `content-encoding`, appends `Vary: Accept-Encoding`, and streams with `compressobj` + a final flush. `app/main.py`: new `Settings.compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`, `0` = the feature OFF (skip the middleware entirely), negative → fail startup loudly — the `agent_max_rounds` pattern, PLAN §13), wired by `app.add_middleware(...)` **before** the `SecurityHeadersMiddleware` line so SecurityHeaders stays outermost (A20) and gzip sits outside the caching layer (an already-rewritten HTML page compresses; `/api/*` still passes the caching middleware byte-identically). No production nginx change is required; if nginx also gzips, it will not double-encode an already-encoded body.
|
||||||
|
|
||||||
|
**D2 — a depth-limited tree + server totals (task 02, LOCKED A2).** `app/api/docs.py`:
|
||||||
|
- `build_kb_tree(..., depth: int | None = None)` and `_level_children(..., depth)` — `None` = unlimited (the legacy behavior every existing unit test keeps, byte-identical); a positive budget stops the recursion. A folder node whose own children were NOT built carries `truncated=True`, `children=[]`, its recursive `documents` from `counts[sub]` (unchanged) and — the correctness trap — its `updated_at` from a NEW recursive max-date map computed in `_folder_counts` (because `_subtree_max(children)` over an empty child list would be `None`, which would silently break the phase-106 D9 column). Its `summary` still rides the node (the level row shows a subfolder's description).
|
||||||
|
- `GET /api/docs/tree` gains `depth: int = Query(1, ge=1, le=8)`. **The default is what breaks the existing pins:** the full consumer list is exactly nine files (`grep -rl "docs/tree" tests app` → `tests/integration/test_docs_api.py`, `tests/integration/test_docs_api_dates.py`, `tests/unit/test_kb_tree_builder.py`, `tests/unit/test_kb_tree_ui.py`, `tests/unit/test_frontend_router.py`, `tests/e2e/test_kb_tree.py`, `tests/e2e/test_navbar_refresh.py`, `tests/e2e/test_document_dates.py`, `tests/e2e/test_git_source_dates.py`), and the TWO DATE SUITES walk straight through the tree with no depth param (`test_docs_api_dates.py` reaches `a` → `a/b` → deeper; `test_git_source_dates.py::_find_file` recurses by design) — they assert phase-106 D9 dates, not payload shape, so they migrate by requesting `?depth=8` explicitly with every assertion kept (task 02 item 9; the E2E one is named in task 06 item 4).
|
||||||
|
- `KbTree` gains **`totals: KbTreeTotals`** (`documents`, `chunks`, `last_indexed: str | None`) computed in the endpoint from the SAME catalogue rows (no extra query) — `sources.js::treeStats()` cannot sum a truncated tree client-side (it walks children) and the three stat cards must stay honest.
|
||||||
|
- `truncated: bool = False` on `KbTreeFolder`/`KbTreeSource` with a `@model_serializer(mode="wrap")` omission when `False` (the `_drop_image_fields` precedent): an untruncated node's wire shape stays byte-identical, and only the depth-limited nodes add the key.
|
||||||
|
|
||||||
|
**D3 — the subtree route + revalidation (task 03, LOCKED A3/A4).**
|
||||||
|
- `GET /api/docs/tree/node?source=<name>&path=<folder>&depth=1` (admin-gated like `/api/docs/tree`, `path=""` = the source root): one node in the `KbTreeFolder`/`KbTreeSource` shape (its direct children + `truncated` flags), 404 `{"detail": "unknown node"}` for an unregistered+unindexed source or a folder that fails the phase-94 existence rule. The RAG view's drill target; the catalogue query is the same one `GET /api/docs/tree` runs (documented: one query per drill, no N+1).
|
||||||
|
- Revalidation, both tree routes (and `GET /api/docs`): build → serialize once (`model_dump_json()` / json) → `ETag: W/"<sha256(body)[:32]>"` + `Cache-Control: private, must-revalidate`; `If-None-Match` match (weak/strong and `*` tolerant) → `Response(304)` with the same two headers. Helpers live in a new `app/core/http_cache.py` (`weak_etag(body)`, `etag_matches(header, tag)`) so the pattern is one function, not a copy. **Why not `sources_meta.version`** (LOCKED A3): bumping that generation marks every saved chat `stale` (phase 53) — the wrong tool — and the admin PATCHes (`/api/documents/summary`, `/api/documents/date`, `/api/folders/summary`) change the tree WITHOUT bumping it, so a version-derived tag would serve stale trees; a content hash is exact, needs no migration, and the tree is built anyway. This is a deliberate, recorded extension of A19 (which governs HTML pages + `/assets/*` + the caching pass-through) — `/api/*` gains a validator on two catalogue routes only; the SSE stream and every mutating route are untouched.
|
||||||
|
|
||||||
|
**D4 — the client drills lazily (task 04).** `frontend/assets/sources.js`: `loadTree()` → `fetch("/api/docs/tree?depth=1")` (still ONE first load), a `levelCache` (Map keyed `source + "/" + folder`), and a drill that fetches `GET /api/docs/tree/node?…` **only** when the target node reports `truncated === true` and is not cached — merged into the cached node before `renderLevel()`. Never-stale (§7.4): the drill's in-flight state disables the row's link + shows one "Loading…" row (the `.kb-*` empty/loading idiom), a failure shows a `role="alert"` line with a Retry affordance and leaves the current level rendered, and the race token (`loadSeq`) extends to the node fetch (a stale drill response may never paint). `folderExistsIn`/`resetVanishedLocation` stop walking a full tree: the node route's 404 IS the "the folder vanished" signal → reset to the top level. `treeStats()` reads `kbTree.totals` (fallback to the existing walk when `totals` is absent — the defensive no-crash path).
|
||||||
|
|
||||||
|
**D5 — the byte budget and the audit (task 05).** New `app/core/response_budget.py`, pure ASGI, one log line per `/api/*` response in the §9 single-line style:
|
||||||
|
`api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41`
|
||||||
|
It counts `http.response.body` chunk bytes as they pass (zero buffering, SSE-safe: `duration_ms` for a stream is the stream's life, documented), logs at INFO when `bytes >= settings.log_large_response_bytes` (`BOR_LOG_LARGE_RESPONSE_BYTES`, default 65536, `0` = log every `/api/*` response, negative fails startup) and at DEBUG otherwise. `GET /api/chat` gets a `stream=1` variant of the line. New `scripts/api_payload_audit.py` (`python -m scripts.api_payload_audit --base-url … --user … --password …`): logs in, hits the candidate endpoints (`/api/docs/tree` at depth 1 and unlimited, `/api/docs`, `/api/documents/content` for the largest doc, `/api/chats`, `/api/chats/{id}` for the largest chat, `/api/git-sources`, `/api/suggestions`, `/api/sync/status`, `/api/tokens`, `/api/config`), and prints uncompressed AND gzip bytes + duration per endpoint with a stated budget (256 KB uncompressed = "look at it", 64 KB = "fine"). Findings land in this directory as `AUDIT.md` (endpoint, before/after bytes, the remedy applied or the accepted-why) — the TODO's "any other endpoints" answered with numbers, not vibes. The audit runs against the local 1,000-doc live-replica snapshot (the `restore-test-db` skill) so the numbers are the production order of magnitude, not the fixture KB's.
|
||||||
|
|
||||||
|
**D6 — verification (task 06).** New `tests/e2e/test_kb_tree_lazy_load.py`: the Sources page's FIRST `/api/docs/tree` request carries `depth=1` + `totals` and no deep children (route-level byte assertion + `via response.json()` node walk), drilling two levels deep issues the node fetch (exactly once per level, cached the second time), the stat cards match `totals` at every level, a 404 node (folder pruned under us — a `page.route` stub) resets to the top level with no stale breadcrumb, `Accept-Encoding: gzip` yields `Content-Encoding: gzip` + a `Vary` header, and the second visit revalidates to a 304 (`page.route` observes the 304 + the body unchanged from cache). Plus the migrated pins: `tests/unit/test_kb_tree_ui.py` (asserts exactly ONE `fetch("/api/docs/tree")` occurrence), `tests/unit/test_frontend_router.py:591`, `tests/e2e/test_navbar_refresh.py:115` (the `#nav-sources` → `/api/docs/tree` expectation), and the phase-97/99 integration shape tests.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `97_kb_tree_catalog` + `99_kb_tree_table_and_back_nav` (complete) — the one-fetch/drill-client-side contract this phase **deliberately supersedes** (owner-confirmed 2026-10-07, TODO L5); `106_document_dates` (D9: a truncated node's `updated_at` must stay the recursive max — D2's trap), `98_sync_summary_visibility` (D3 `summary_pending` on every node, including truncated ones), `122_image_documents` (task 04's image map, which stays a bounded select), `156_table_ux_polish` (the bounded scroll panel + sticky header the drill re-renders into), `82_security_headers` / A19 / A20 (the middleware order this phase must not disturb), A15 (SSE).
|
||||||
|
- No migration: nothing in the DB changes.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_gzip_middleware.md` — `app/core/compression.py` + `Settings.compress_min_bytes` + wiring + the compression/SSE/passthrough tests.
|
||||||
|
2. `02_tree_depth_and_totals.md` — `KbTreeTotals`, the `truncated` omission serializer, `build_kb_tree(depth=…)`, the recursive max-date map, `depth` on `/api/docs/tree`, builder + integration tests.
|
||||||
|
3. `03_tree_node_endpoint_etag.md` — `app/core/http_cache.py`, `GET /api/docs/tree/node`, ETag/304 on the two tree routes + `/api/docs`, integration matrix.
|
||||||
|
4. `04_client_lazy_drill.md` — `sources.js` level cache + per-level fetch + §7.4 drill states + `totals` stat cards + the migrated unit pins.
|
||||||
|
5. `05_response_budget_audit.md` — `app/core/response_budget.py` + `scripts/api_payload_audit.py` + `AUDIT.md` with the measured numbers + the remedies for the offenders (and the accepted-why for the rest) + README/env docs.
|
||||||
|
6. `06_e2e_payload_diet.md` — `tests/e2e/test_kb_tree_lazy_load.py`, the migrated E2E pins, the full gate set, the commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the builder depth/truncation/max-date matrix (`tests/unit/test_kb_tree_builder.py`), the schema omission wire-shape pins, `tests/unit/test_caching.py` extended (the `/api/*` pass-through survives gzip coexistence), config validator pins, `tests/unit/test_kb_tree_ui.py` extended for the drill (cache, single fetch per level, race token, loading/alert states, `totals` stat cards).
|
||||||
|
- Integration: depth + truncated + totals + node-route matrix (200/404/403), the migrated `tests/integration/test_docs_api_dates.py` (explicit `?depth=8`, every D9 date assertion intact), the ETag matrix (no header → 200 + `ETag`; match → 304 no body; mismatch → 200; `*`; weak/strong equivalence; a folder-summary PATCH changes the tag), gzip (`Content-Encoding: gzip`, `Vary`, under-`minimum_size` identity, `Accept-Encoding: identity` identity, the SSE stream NOT compressed and arriving frame-by-frame), and the response-budget log line (caplog: INFO over the threshold, DEBUG under, `stream=1` for `/api/chat`).
|
||||||
|
- E2E: `tests/e2e/test_kb_tree_lazy_load.py` green **in isolation**; the owning regression set green in isolation: `test_kb_tree.py` (migrated), `test_kb_tree_nav.py`, `test_navbar_refresh.py`, `test_document_dates.py`, `test_git_source_dates.py` (its `_find_file` recurses — it needs the explicit `?depth=8`), `test_sync_summary_visibility.py`, `test_table_ux_polish.py`, `test_sources_midstream_bug.py`, `test_ls_tree_drilldown.py` (the agent's `ls` view must stay byte-identical), `test_smoke.py` (the no-CDN integration test stays green).
|
||||||
|
- Coverage **>90%** on `app/` (the new `compression.py` / `http_cache.py` / `response_budget.py` are small and fully pinned).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/docs/tree` (default `depth=1`) + `Content-Encoding: gzip` transfers < 40 KB on the audited live-replica corpus (measured by `scripts/api_payload_audit.py`, recorded in `AUDIT.md`) — from ~2 MB — and a repeat load answers **304** with an empty body.
|
||||||
|
- [ ] Drilling two folders deep renders correct rows (counts, dates, descriptions, thumbnails, pending markers) with exactly one node fetch per uncached level; a vanished folder resets honestly.
|
||||||
|
- [ ] The stat cards are right at every level (server `totals`), and the tree's per-node `updated_at` is still the recursive max even when the node's children are truncated.
|
||||||
|
- [ ] `POST /api/chat` streams uncompressed and frame-by-frame (unit + E2E proof); HTML pages still serve `no-cache` with the `?v=` rewrite; every response still carries the A20 headers.
|
||||||
|
- [ ] `AUDIT.md` lists every candidate endpoint with before/after bytes and its verdict; offenders over the stated budget are fixed or explicitly accepted with a reason.
|
||||||
|
- [ ] `uv run pytest` green, coverage > 90%, `uv run ruff check . && uv run pyright` clean, the phase E2E green in isolation, one `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — gzip is selective and pure-ASGI**: `POST /api/chat` (SSE, A15) is skipped by EXACT path; no `BaseHTTPMiddleware`; `BOR_COMPRESS_MIN_BYTES` (`0` = off, negative = fail loudly) — owner-confirmed 2026-10-07 ("gzip + ETag + lazy drill").
|
||||||
|
- **A2 — `depth` (default 1, max 8) + `totals` on the tree, `truncated` omitted when false**: the depth-limited response keeps every per-node field the view renders (counts, `updated_at`, `summary`, `summary_pending`, image keys) and only omits un-rendered DEPTH; the recursive max-date map is mandatory so phase 106 D9 does not silently regress — owner-confirmed 2026-10-07.
|
||||||
|
- **A3 — the phase-97 "ONE fetch, drill client-side" contract is superseded** by one initial load + one fetch per drilled level (the deliberate cost of a small first payload; the drill UX, breadcrumb, and scroll-panel behavior stay as phases 97/99/156 built them) — owner-confirmed 2026-10-07.
|
||||||
|
- **A4 — the ETag is a weak hash of the response body, NOT `sources_meta.version`**: bumping the generation would mark every saved chat `stale` (phase 53), and the admin summary/date/folder PATCHes mutate the tree without bumping it. Recorded as a scoped extension of A19 (validators on two read-only catalogue routes; SSE and mutations untouched) — owner-confirmed 2026-10-07.
|
||||||
|
- **A5 — the byte budget is a log line + a script, not a hard 413**: no endpoint starts rejecting large payloads this phase (the app's own documents are legitimately large); the audit's 256 KB "look at it" threshold drives follow-up work, recorded in `AUDIT.md`.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ scripts/ tests/ README.md .env.example .agents/phases/ && git commit --no-gpg-sign -m "perf(api): gzip non-SSE responses, a depth-limited KB tree with per-level drill and ETag revalidation, and a measured response byte budget"
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 01 — Selective gzip: `app/core/compression.py`, `BOR_COMPRESS_MIN_BYTES`, SSE untouched (phase 167 D1, LOCKED A1)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every HTTP response except the `POST /api/chat` SSE stream compresses with gzip, with a config kill switch, the existing middleware order and contracts (A19 caching pass-through, A20 security headers outermost) untouched, and a test that proves the chat stream still arrives frame-by-frame and uncompressed.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `app/core/compression.py` — the middleware of phase 167 D1, **pure ASGI** (never `BaseHTTPMiddleware`, which buffers and would break streaming responses):
|
||||||
|
- `class SelectiveGZipMiddleware` holding `self.app` and a composed `starlette.middleware.gzip.GZipMiddleware(self.app, minimum_size=minimum_size)`; `__call__` delegates to the gzip path unless `scope["type"] != "http"` or `scope["path"] in skip_paths` (EXACT match — `/api/chat-images` must still compress), in which case it calls `self.app` directly;
|
||||||
|
- `DEFAULT_SKIP_PATHS = ("/api/chat",)` module constant, docstring names A15 (SSE) and the buffering reason; do NOT re-implement compression math — Starlette's `GZipMiddleware` owns `Accept-Encoding` negotiation, `minimum_size`, the already-`content-encoding` skip, the `Vary: Accept-Encoding` append, and the `compressobj` streaming + final flush;
|
||||||
|
- module docstring: the TODO's 2 MB defect, the ~8-12× JSON expectation, the SSE exclusion, and "nginx may also gzip; an already-encoded body is never double-encoded".
|
||||||
|
2. `app/config.py` — `compress_min_bytes: int = 860` (`BOR_COMPRESS_MIN_BYTES`) beside the other ints, with the `agent_max_rounds` validator pattern (`_compress_min_bytes_non_negative`: negative → `ValueError("compress_min_bytes must be >= 0 (0 = compression disabled)")`) and the docstring stating `0` = the feature is OFF (no middleware), `> 0` = the Starlette minimum-size floor (860 = the classic nginx default, below which compression can inflate).
|
||||||
|
3. `app/main.py` — wire it, added **before** the `app.add_middleware(SecurityHeadersMiddleware)` line so SecurityHeaders stays outermost (A20, the existing comment says "last add_middleware"): construct `SelectiveGZipMiddleware` only when `settings.compress_min_bytes > 0` (the `0` = OFF path adds nothing at all), with a comment naming TODO L5, the SSE skip, and the position (outside `configure_caching`, inside SecurityHeaders → the rewritten HTML pages compress too, `/api/*` still passes the caching layer byte-identically before compression).
|
||||||
|
4. `.env.example` — `BOR_COMPRESS_MIN_BYTES=860` with the three-value comment (0 off, negative refuses startup, the 860 rationale).
|
||||||
|
5. New `tests/unit/test_compression.py`:
|
||||||
|
- a tiny app with a ~50 KB JSON route + a ~50 byte route, wrapped in the middleware: `Accept-Encoding: gzip` → `Content-Encoding: gzip` + a body that `gzip.decompress`es to the original + `Vary` mentions `accept-encoding`; `Accept-Encoding: identity` → no `Content-Encoding`; the small route → identity (under the floor); a response that already set `content-encoding` → untouched;
|
||||||
|
- `skip_paths`: a `/api/chat`-named route in the test app is NOT compressed and streams incrementally (assert the received chunks arrive before the response completes — the `list(stream)` idiom the existing SSE tests use), while a `/api/chat-images/x` route IS compressed (the exact-match pin);
|
||||||
|
- `scope["type"] == "lifespan"`/`"websocket"` pass-through;
|
||||||
|
- config: `compress_min_bytes = 0` → `create_app` adds no gzip layer (assert via `app.user_middleware`'s classes), `-1` → the validator raises naming the field.
|
||||||
|
6. Integration pin `tests/integration/test_docs_api.py` (or the middleware-owning integration file — locate via `SecurityHeadersMiddleware` in `tests/`): the real app's `GET /api/health`-class small response stays identity; a large `/api/docs/tree` (seed ≥ 50 docs) returns `Content-Encoding: gzip` and decompresses to the documented JSON; the A20 headers (`CSP`, `X-Content-Type-Options`, `X-Frame-Options`/`frame-ancestors`) are STILL present on a gzipped response (the outer-layer invariant); the HTML page contract (A19: `no-cache`, `?v=` rewrite, no validators) survives decompression.
|
||||||
|
7. ASSUMPTION: `Settings.compress_min_bytes` (name + default 860) — the TODO named no knob; the house kill-switch convention (`0` off, negative loud) is what makes it owner-controllable without a code change.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration as above; `uv run pytest tests/integration/test_chat_api.py -v` (or the suite that owns the SSE contract) stays green — the stream is not compressed and not buffered.
|
||||||
|
- Coverage: **>90%** on `app/core/compression.py` (every branch listed above is pinned).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `curl -H 'Accept-Encoding: gzip' …/api/docs/tree | wc -c` beats the identity size by an order of magnitude on any corpus above the floor; `Vary: Accept-Encoding` is present.
|
||||||
|
- [ ] `POST /api/chat` responses carry NO `Content-Encoding` and stream frame-by-frame (test-proven), and `/api/chat-images` still compresses.
|
||||||
|
- [ ] `BOR_COMPRESS_MIN_BYTES=0` removes the middleware entirely; `-1` refuses startup naming the field.
|
||||||
|
- [ ] A20 headers, A19 page/asset caching, and the `/api/*` pass-through are unchanged; `uv run pytest` green; ruff + pyright clean.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 02 — Depth-limited tree + server totals: `truncated`, the recursive max-date map, `KbTreeTotals` (phase 167 D2, LOCKED A2)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET …/api/docs/tree loads 2 MB in production which causes long loading times."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The pure builder can stop at a depth budget: a node whose children were not built says so (`truncated: true`) while keeping every field the Sources view renders (recursive count, recursive `updated_at`, its own `summary`, `summary_pending`, the image affordance), the response gains the `totals` the stat cards need, and `GET /api/docs/tree` defaults to `depth=1`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py` — `KbTreeTotals(BaseModel)`: `documents: int = Field(ge=0)`, `chunks: int = Field(ge=0)`, `last_indexed: str | None = None` (docstring: the catalogue-wide totals the RAG view's three stat cards read, because a depth-limited tree cannot be summed client-side — the phase-97 `treeStats()` walk needed the whole tree).
|
||||||
|
2. `app/schemas.py` — `KbTree` gains `totals: KbTreeTotals = KbTreeTotals()` (docstring: additive top-level key, TODO L5) and `KbTreeFolder` / `KbTreeSource` gain `truncated: bool = False` with an OMIT-WHEN-FALSE serializer. Follow the file's existing omission idiom (`_drop_image_fields` + `@model_serializer(mode="wrap")`): add the `truncated` key only when `True`, so an untruncated node's wire shape is byte-identical to today (every existing wire pin stays green). `KbTreeFile` never truncates (a file has no children).
|
||||||
|
3. `app/api/docs.py::_folder_counts` — extend the return to `(folders, counts, max_dates)` where `max_dates[folder]` is the folder's RECURSIVE MAX document `created_at` (the same ISO-8601 strings, so the lexicographic max is the chronological one — the phase-106 D9 rule), computed from the rows in the same pass. Rationale in the docstring: a truncated node has NO built children, so `_subtree_max(children)` would read `None` and silently lose the D9 column; the aggregate map is the truncation-proof source. Keep `_subtree_max` for the expanded path UNCHANGED and add a unit test asserting the two agree for every expanded folder (one concept, two derivations, cross-pinned).
|
||||||
|
4. `app/api/docs.py::_level_children` — add `depth: int | None`; per subfolder: when `depth is None` or `depth > 1`, build the node as today with the recursion at `depth - 1`; when the budget is exhausted, build `KbTreeFolder(path=sub, documents=counts[sub], updated_at=max_dates[sub], summary=summaries.get((source, sub)), summary_pending=<unchanged rule>, children=[], truncated=True)`. Update the docstring with the depth rule + the truncated-node field set (the level row renders a subfolder's name, count, date and description WITHOUT its contents).
|
||||||
|
5. `app/api/docs.py::build_kb_tree` — thread `depth` (`None` = unlimited: every existing unit call and every existing expectation is byte-identical); `_source_node` passes the budget down and never marks the SOURCE node itself truncated at `depth ≥ 1` (the source's direct children are always built — the top level of the RAG view is source → level 1).
|
||||||
|
6. `app/api/docs.py::list_kb_tree` — `depth: int = Query(1, ge=1, le=8)` (docstring: 1 = sources + one level, the view's first paint; 8 = the practical KB ceiling; the endpoint computes `totals` from the catalogue rows it already fetched: `documents = len(rows)`, `chunks = sum(chunks)`, `last_indexed = max(indexed_at)` → `None` for an empty KB) and passes it to the builder. `GET /api/docs` and the agent's `ls` path are untouched (pin it: `app/rag/agent.py` imports are unchanged).
|
||||||
|
7. `tests/unit/test_kb_tree_builder.py` (extend): depth `None` vs `1` vs `2` vs deep on the multi-level fixture — the truncated node's exact field set (count recursive, `updated_at` = the deep max, summary present, `summary_pending` present, `children == []`, `truncated is True`), the totals math incl. the empty KB (`last_indexed is None`), the `_subtree_max`/`max_dates` cross-check on every expanded folder, and the phase-94/98 invariants (existence rule, `missing_folder_summaries` cross-check) at each depth.
|
||||||
|
8. `tests/integration/test_docs_api.py` (extend): `GET /api/docs/tree` default → depth 1 (deep children absent, `truncated` present on a subfolder that has them, `totals` correct vs `GET /api/docs`), `?depth=3` → deeper, `?depth=0` / `?depth=9` / `?depth=abc` → 422, anonymous → 403 unchanged, and the wire-shape pin: an UNtruncated node's JSON keys are exactly the pre-phase set (no `truncated` key).
|
||||||
|
9. Migrate the TWO date suites that consume the tree with no depth param (they break on the new default — the phase changes the default, so it owns the repair; every assertion stays, only the URL gains `?depth=8`):
|
||||||
|
- `tests/integration/test_docs_api_dates.py` (~L157): `admin_client.get("/api/docs/tree")` → `"/api/docs/tree?depth=8"`, keeping the whole `a` → `a/b` → deeper D9 walk (deep date beats shallow at every level) and the 0-document-source `updated_at is None` case; add a one-line comment naming phase 167 (the default is the view's first paint, `depth=8` is the whole tree).
|
||||||
|
- `tests/e2e/test_git_source_dates.py` (~L419): the `page.request.get(f"{app_url}/api/docs/tree")` call gains `?depth=8` so `_find_file` (which recurses by design) still reaches `old/…` and `recent/…`; the folder `updated_at` date assertions stay verbatim.
|
||||||
|
- Add the truncation-proof counterpart so the guarantee is tested BOTH ways: at the DEFAULT depth, a truncated folder node's `updated_at` still equals the deep max (this is exactly what the task-02 `max_dates` map buys — the date contract survives the payload diet), pinned in `test_docs_api_dates.py` and in `tests/unit/test_kb_tree_builder.py`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration above; `uv run pytest tests/unit/test_kb_tree_builder.py tests/integration/test_docs_api.py -v` green.
|
||||||
|
- Coverage **>90%** on the modified `app/api/docs.py` / `app/schemas.py` lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Default `/api/docs/tree` returns sources + ONE level with correct `totals`, and every node that has unloaded children carries `truncated: true`.
|
||||||
|
- [ ] A truncated node still carries its recursive count, its recursive `updated_at`, its `summary`, and `summary_pending` — phase 98/106 semantics intact at every depth (unit + integration).
|
||||||
|
- [ ] `depth=None` (builder default) reproduces today's tree byte-for-byte; untruncated nodes' wire shape is unchanged.
|
||||||
|
- [ ] `uv run pytest` green; coverage > 90%; ruff + pyright clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 03 — The subtree route and revalidation: `GET /api/docs/tree/node`, `app/core/http_cache.py`, ETag/304 (phase 167 D3, LOCKED A3/A4)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "Fix that any any other endpoints that could load a ton of data."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One small admin-gated route serves a single tree node's direct children (the drill's fetch target), and the catalogue read routes (`/api/docs/tree`, `/api/docs/tree/node`, `/api/docs`) carry a content-exact weak `ETag` + `Cache-Control: private, must-revalidate` so a repeat load costs a 304 with no body — without touching `sources_meta` (which would mark saved chats stale) or the SSE/mutating surface.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `app/core/http_cache.py` (small, total, no I/O):
|
||||||
|
- `weak_etag(body: str | bytes) -> str` → `W/"<sha256 hexdigest[:32]>"` (docstring: content-exact by construction; 32 hex chars is the file's truncation convention and collisions are not a correctness risk for a revalidated cache — a false match only ever costs one stale repaint on a 2 MB namespace, and the tag is over the EXACT served bytes);
|
||||||
|
- `etag_matches(header: str | None, tag: str) -> bool` → tolerant of `*`, a comma-separated list, and weak/strong equivalence (`W/"x"` matches `"x"` both directions), whitespace-tolerant, `None`/empty → `False`;
|
||||||
|
- `revalidate_headers(etag: str) -> dict[str, str]` → `{"ETag": etag, "Cache-Control": "private, must-revalidate"}` (ONE place for the pair; docstring names A19's scope — HTML pages stay `no-cache` with no validators, `/assets/*` stay immutable, this is the read-only API arm).
|
||||||
|
2. `app/api/docs.py` — `GET /api/docs/tree/node`:
|
||||||
|
- signature: `source: str = Query(...)`, `path: str = Query("")` (source-relative; `""` = the source root node), `depth: int = Query(1, ge=1, le=8)`, `db=Depends(get_db)`, `_admin=Depends(require_admin)` (`403 {"detail": "admin only"}` — the same gate as `/api/docs/tree`), `request: Request`;
|
||||||
|
- composition: the SAME catalogue query as `list_kb_tree` (one query, no N+1 — the file's docstring paragraph says so), the same `folder_summaries` + image-docs bounded selects, then build the single node through the pure helpers: resolve the source (registry name or indexed source), `_folder_counts(rows)` for that source, `_level_children(source, path, folders, counts, rows, summaries, images, depth=depth)`; the node is a `KbTreeSource` when `path == ""` else a `KbTreeFolder` with `documents=counts[path]`, `updated_at=max_dates[path]`, `summary`, `summary_pending` (the phase-98 D3 rule), `truncated` per the depth budget;
|
||||||
|
- 404 `{"detail": "unknown node"}` when: the source is neither registered nor indexed, OR the folder fails the phase-94 existence rule (`path` not in `folders` and `path != ""`). Never a filesystem path built from `path` (docstring: the row-only rule, the phase-10 traversal argument reused);
|
||||||
|
- `response_model=None` on the decorator with the union return annotated, and the body returned as a raw `Response` (below) so the serializer is not run twice.
|
||||||
|
3. `app/api/docs.py` — ETag wiring on the three read routes (`/api/docs/tree`, `/api/docs/tree/node`, `/api/docs`): build the model → `body = model.model_dump_json()` (for `/api/docs`: `DocList.model_dump_json()`) → `etag = weak_etag(body)` → `if etag_matches(request.headers.get("if-none-match"), etag): return Response(status_code=304, headers=revalidate_headers(etag))` → else `Response(content=body, media_type="application/json", headers=revalidate_headers(etag))`. Docstring per route: the client is `fetch` (a 304 is served from its cache transparently), and `must-revalidate` guarantees a sync or an admin edit is seen on the next load — the tag changes because the body changes. Keep `response_model=KbTree`/`DocList` on the decorators for the OpenAPI schema (a returned `Response` short-circuits serialization — note that in the docstring).
|
||||||
|
- ASSUMPTION: `GET /api/documents/content` is NOT given a validator this phase (one document's body, on demand, already compressed; the audit in task 05 decides whether it needs more) — recorded so the scope is explicit.
|
||||||
|
4. `tests/integration/test_docs_api.py` (extend):
|
||||||
|
- the node route: 200 for a source root (`path=""`, the `KbTreeSource` shape), 200 for a nested folder (correct `documents`/`updated_at`/`summary`/`summary_pending`/`children` at `depth=1` with `truncated` on the deeper subfolders), `?depth=2` deeper, 404 `unknown node` for an unknown source / a nonexistent folder / a `path` that is a FILE's path, 403 anonymous, and a cross-check that the node's children equal the same level of the unlimited `/api/docs/tree?depth=8` response (ONE concept, two routes, pinned equal);
|
||||||
|
- ETag matrix on all three routes: first GET → 200 + `ETag` + `Cache-Control: private, must-revalidate`; re-send with `If-None-Match` → 304 + empty body + both headers; a wrong tag → 200; a `W/`-vs-strong mismatch pair → 304; `*` → 304; after `PATCH /api/folders/summary` (and after `PATCH /api/documents/date`) the tag CHANGES and the next GET is a 200 with the new tree (the A4 proof — a version-derived tag would have failed exactly here);
|
||||||
|
- gzip coexistence: a gzipped 200 still carries the `ETag` (the header survives the compression layer) and a conditional request returns a body-less 304.
|
||||||
|
5. `tests/unit/test_http_cache.py` (new): `weak_etag` stability/length/prefix (`W/"…"`), bytes-vs-str equality, `etag_matches` for `None`, empty, `*`, single, comma-list, weak↔strong both directions, whitespace, and a tag that merely shares a prefix → `False`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration + unit above; `uv run pytest tests/integration/test_docs_api.py tests/unit/test_http_cache.py -v` green; `tests/integration/test_caching.py` stays green (the `/api/*` pass-through and the HTML rewrite are untouched by a route-level header).
|
||||||
|
- Coverage **>90%** on `app/core/http_cache.py` and the new route lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/docs/tree/node?source=X&path=Y` returns exactly that folder's direct children with the same per-node fields the full tree produces (equality pinned against `depth=8`).
|
||||||
|
- [ ] A repeat request with `If-None-Match` answers **304, no body**; a folder-summary or document-date edit changes the tag and forces the next 200.
|
||||||
|
- [ ] Anonymous gets `403 {"detail":"admin only"}`; an unknown source/folder gets `404 {"detail":"unknown node"}`; nothing builds a filesystem path from `path`.
|
||||||
|
- [ ] `uv run pytest` green; coverage > 90%; ruff + pyright clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 04 — The client drills lazily: level cache, per-level fetch, §7.4 drill states, `totals` stat cards (phase 167 D4, LOCKED A3)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET …/api/docs/tree loads 2 MB in production which causes long loading times."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The RAG view loads one small tree, fetches a folder's contents the first time it is opened, caches it, keeps the breadcrumb/scroll-panel/editing behavior identical, and never shows a stale or half-updated level — the UX cost of A3 paid in full so the user cannot tell the tree is no longer one payload.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/sources.js::loadTree()` (~L1625) — `fetch("/api/docs/tree?depth=1")`; on success: store the tree, CLEAR `levelCache` (a refresh invalidates every drilled level — the sync/refresh call sites at ~L1258, ~L1386, ~L2192, ~L2213 keep working unchanged), then `resetVanishedLocation()` + `renderLevel()` as today. Keep the `loadSeq` race token and the `renderEmpty()` failure path.
|
||||||
|
2. `frontend/assets/sources.js` — new module state + the drill:
|
||||||
|
- `const levelCache = new Map();` keyed `source + "/" + folder` (a node's `path`), populated ONLY by a node fetch;
|
||||||
|
- `function needsLevelFetch(node) { return node.truncated === true && !levelCache.has(key); }`;
|
||||||
|
- `enterFolder(source, folder)` (the folder-row click path, today a pure `current = …; renderLevel()`): when the target needs a fetch → set the in-flight state (`drillPending = {source, folder}`, the row link gets `aria-busy="true"` + `aria-disabled="true"` so a double click cannot start a second fetch), `await fetch("/api/docs/tree/node?source=…&path=…")`, merge the result (`Object.assign(targetNode, node)` + `levelCache.set(key, true)`), clear the flags, then `current = …; renderLevel()`. A stale response (token changed) never paints (the existing `my === loadSeq` convention extended to the drill).
|
||||||
|
- In-flight rendering: render the level the user is LEAVING unchanged and show one loading row in the folders panel using the file's existing status/empty idiom (a `role="status"` `.kb-level-note`-style line "Loading <folder>…" — reuse an existing class if one fits, otherwise add ONE `.kb-level-loading` rule to `styles.css` with tokens only, no animation beyond the existing house spinner if one exists);
|
||||||
|
- failure (non-OK or a network throw): `role="alert"` line "Couldn't load <folder> — Retry." with a real `<button type="button">Retry</button>` that re-runs the same fetch, and the CURRENT level stays rendered (never blank, never a stale breadcrumb — PLAN §7.4 "every failure path re-enables its controls").
|
||||||
|
3. `frontend/assets/sources.js` — the existence rules stop walking a full tree:
|
||||||
|
- `folderExistsIn(src, folderPath)` (~L1667) is replaced by `folderKnownIn(src, folderPath)`: true when the path's whole ancestor chain is present in what the client holds (the loaded tree + cached levels), so `resetVanishedLocation()` can still answer after a re-fetch WITHOUT assuming a full tree; when a drill fetch answers `404` the folder is gone → `current = { source: null, folder: null }` + `renderLevel()` (the honest reset, the phase-97 "never stale" contract carried over), and the failing `levelCache` entry is not cached;
|
||||||
|
- `currentLevelNode()` (~L1680) keeps walking, but through cached nodes (unchanged semantics for loaded levels).
|
||||||
|
4. `frontend/assets/sources.js::treeStats()` (~L1703) — read `kbTree.totals` (`documents`, `chunks`, `last_indexed`) when present, and keep the existing recursive walk ONLY as the defensive fallback when `totals` is absent (a tree served by a pre-phase process); the three stat cards (`statDocs`, `statChunks`, `statLast`) render identically at every level (pin the exact formatting call — `fmtDate(st.last)` — unchanged).
|
||||||
|
5. `frontend/assets/sources.js` docstring (the module header, ~L55-103) — rewrite the "ONE fetch of GET /api/docs/tree returns the whole tree" paragraph for the depth-1 + per-level model, citing TODO L5 and phase 167, and keep the phase-79 race-token, phase-90 upload and phase-98 pending narratives accurate.
|
||||||
|
6. `tests/unit/test_kb_tree_ui.py` (MIGRATE — it currently asserts `js.count('fetch("/api/docs/tree")') == 1` at ~L250 and the "ONE fetch" contract at ~L9/L27): the fetch assertion becomes `fetch("/api/docs/tree?depth=1")` exactly once + `fetch("/api/docs/tree/node?` exactly once, and the "drills client-side" docstring assertions become the level-cache assertions: `levelCache` declared once, cleared inside `loadTree`, consulted by `needsLevelFetch`, written only after a 200; the 404 → reset branch exists; `drillPending` guards a second click; the Retry button is a real `<button>`; `treeStats` prefers `totals`. Preserve every OTHER phase-97/99 pin (breadcrumb, sticky header, scroll panel, editor reset, empty state).
|
||||||
|
7. `tests/unit/test_frontend_router.py:591` — the shell-route pin (`'fetch("/api/docs/tree")' in body`) moves to the `?depth=1` URL, keeping its original intent (the Sources view loads the catalogue from the tree endpoint, not the flat one).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the migrated + new pins above; run the owning set for collateral: `uv run pytest tests/unit -k "kb_tree or frontend_router or table_ux or source" -v`.
|
||||||
|
- No `app/` change → coverage gate unmoved (> 90% on `app/` stays green).
|
||||||
|
- Frontend house rules hold: `createElement` + `textContent` only (no HTML strings), `aria-busy`/`aria-disabled` on the in-flight link, `role="status"`/`role="alert"` for the two states, `:focus-visible` on the new Retry button via the global rule, tokens-only CSS if a new class is added.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] First paint = ONE `depth=1` request; the second visit to a folder is free (cache hit, zero network — unit-pinned).
|
||||||
|
- [ ] A drilled level shows correct rows (counts, dates, descriptions, pending markers, image thumbnails) inside the phase-156 bounded panel with the sticky header intact.
|
||||||
|
- [ ] In-flight = `aria-busy` + one status line; failure = `role="alert"` + working Retry with the current level still rendered; a 404 resets to the top level with no stale breadcrumb; a stale response never paints.
|
||||||
|
- [ ] The three stat cards match the server `totals` at every level, and the pre-phase fallback path exists.
|
||||||
|
- [ ] Full `uv run pytest` green; ruff + pyright clean.
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 05 — The byte budget: `/api/*` response-size log line + `scripts/api_payload_audit.py` + `AUDIT.md` (phase 167 D5, LOCKED A5)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "Fix that **any any other endpoints that could load a ton of data**."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The app can answer "which endpoint is heavy, right now, in production?" from its own logs, and the owner's "any other endpoints" gets a measured, written answer: every candidate endpoint's uncompressed and gzipped size against a stated budget, with each offender either fixed or explicitly accepted.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `app/core/response_budget.py` — **pure ASGI** (no buffering, SSE-safe):
|
||||||
|
- wraps `send`, counting the bytes of every `http.response.body` message's `body` chunk, and `receive`-independent timing (`time.monotonic()` at entry → at `more_body=False` completion);
|
||||||
|
- logs ONE §9-style line per `/api/*` response on the `http.response.body` completion (for a stream that is when the stream ends — say so in the docstring):
|
||||||
|
`api method=GET path=/api/docs/tree status=200 bytes=204511 enc=gzip duration_ms=41`
|
||||||
|
(`enc` = the response's `content-encoding` or `identity`; add ` stream=1` for `media_type == "text/event-stream"` responses, where `bytes` is the streamed total and `duration_ms` is the stream's life);
|
||||||
|
- level: INFO when `bytes >= settings.log_large_response_bytes`, DEBUG otherwise; non-`/api/` paths and non-HTTP scopes pass through silently; an exception in the wrapped app still logs (finally-style, `bytes` as counted) and re-raises unchanged;
|
||||||
|
- `304`s log `bytes=0` (the win this phase bought is visible in the log, not hidden).
|
||||||
|
2. `app/config.py` — `log_large_response_bytes: int = 65536` (`BOR_LOG_LARGE_RESPONSE_BYTES`) with the `agent_max_rounds` validator (negative → `ValueError` naming the field; `0` = log EVERY `/api/*` response at INFO). `.env.example` entry with the three-value comment.
|
||||||
|
3. `app/main.py` — `app.add_middleware(ResponseBudgetMiddleware)` **before** the gzip `add_middleware` call, so the stack is SecurityHeaders (outermost) → gzip → response-budget → caching → session → routes. Consequence to state in the comment: `bytes=` is the bytes the ROUTE produced (uncompressed) while `enc=` reports what went on the wire — the two together answer "how fat is it, and how fat did the client see it".
|
||||||
|
4. New `scripts/api_payload_audit.py` (`python -m scripts.api_payload_audit`), stdlib + `httpx` only, `app.core.debugging` conditional import preserved if it touches boot (AGENTS.md rule 7 — it does not import the app factory, so just follow the other scripts' shape, see `scripts/eval_retrieval.py`):
|
||||||
|
- args: `--base-url` (default `http://localhost:8000`), `--username`/`--password` (or `BOR_ADMIN_*` env reads, never hardcoded), `--budget` (default `262144`), `--path-tree` etc. only if needed;
|
||||||
|
- logs in (the real `/api/login` session cookie), then for each candidate: `GET /api/docs/tree?depth=1`, `/api/docs/tree?depth=8`, `/api/docs/tree/node?source=<first>&path=<first folder>`, `GET /api/docs`, `GET /api/documents/content` for the LARGEST document (ask `/api/docs` then pick by chunks) — its `content` length is the size, `GET /api/chats`, `GET /api/chats/{id}` for the largest chat, `/api/git-sources`, `/api/suggestions`, `/api/sync/status`, `/api/tokens`, `/api/config`, `/api/whoami` — measuring BOTH `identity` and `gzip` request variants (explicit `Accept-Encoding`) via `response.content` length + wall time, plus one conditional re-request for the ETag routes (expect 304/0 bytes);
|
||||||
|
- prints a table (endpoint, identity bytes, gzip bytes, ratio, ms, verdict) where the verdict is `ok` / `LOOK (> budget)` / `304`, exits 0 (it is a report, not a gate) — and prints the corpus it measured (document count from `/api/docs`) so numbers are interpretable.
|
||||||
|
5. `.agents/phases/todo/167_api_payload_diet/AUDIT.md` (new, committed with the phase) — the measured record: how it was run (the local DB restored to the **1,000-document live-replica snapshot** via the `restore-test-db` skill, so the order of magnitude matches production), the full table before (uncompressed, no middleware) and after (depth-1 + gzip + ETag), and one verdict line per endpoint: **fixed** (tree: 2 MB → X KB, second load 304) / **accepted with why** (`/api/documents/content` — one document on demand, compressed; `/api/chats/{id}` — the owner opening one saved chat; `/api/sync/status` — tiny but polled, so the log line is what matters) / **follow-up** (anything still over `--budget`, with the proposed remedy named). Do not hand-write numbers: paste the script's output.
|
||||||
|
6. Remedies in scope if the audit shows an offender (apply, don't just note): add the task-03 ETag pair to any other read route that is large and re-fetched (e.g. `/api/git-sources` if it grows past the budget); raise nothing, reject nothing (LOCKED A5).
|
||||||
|
7. `README.md` — an "Operational notes" subsection: the log line format + how to read it (`grep ' :: api ' `), `BOR_LOG_LARGE_RESPONSE_BYTES`, running the audit script against a restored replica, and the deploy note that the app gzips itself (so nginx `gzip on` is optional, and `proxy_set_header Accept-Encoding` passthrough is fine).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_response_budget.py`, new): the line's exact shape for a large 200 (INFO), a small 200 (DEBUG), a 304 (`bytes=0`), a non-`/api/` path (nothing logged), a streaming response (`stream=1`, bytes = sum of chunks, logs on completion), the `enc=gzip` vs `enc=identity` reading, an app exception (logged + re-raised), and the config validator (`0` → INFO for everything, negative → refuses startup).
|
||||||
|
- Script smoke: `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` against a running dev app (document count + at least the tree rows printed; the script must fail loudly with a clear message when the app is unreachable or the login is rejected).
|
||||||
|
- Coverage **>90%** on `app/core/response_budget.py`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] One parseable `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per `/api/*` response at the configured level, with the SSE stream logged once at stream end and never buffered.
|
||||||
|
- [ ] `AUDIT.md` exists in this phase directory with real numbers (before/after), the corpus it measured, and a verdict per candidate endpoint.
|
||||||
|
- [ ] Every endpoint over the stated budget is either fixed in this phase or carries a named follow-up with its reason.
|
||||||
|
- [ ] README documents the log line, the env var, the audit script, and the self-gzip deploy note; `uv run pytest` green; ruff + pyright clean.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Task 06 — `tests/e2e/test_kb_tree_lazy_load.py`, the migrated pins, the full gates (phase 167 D6)
|
||||||
|
|
||||||
|
**Phase:** `167_api_payload_diet` · **Source:** `TODO.md:5` — "GET <https://brain.reeseapps.com/api/docs/tree> loads 2 MB in production which causes long loading times. Fix that any any other endpoints that could load a ton of data."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated Playwright suite proves the owner-visible outcome — a small first Sources load, correct content when drilling, no refetch, an honest reset when a folder disappears, gzip on the wire, a 304 on the second visit — and every pin the superseded phase-97 contract owned (D3/A3) is migrated rather than deleted. Then the phase gates close it out.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New `tests/e2e/test_kb_tree_lazy_load.py` (module-scoped `app_server`/`app_url`; seed a multi-level KB directly through the DB the way `tests/e2e/test_kb_tree.py` does — ≥ 3 sources, a 3-deep folder chain, one folder with a stored `folder_summaries` row, one pending folder, and one image doc if `BOR_IMAGES=true` is cheap in this suite's env; read `tests/e2e/test_kb_tree.py` and `tests/e2e/test_kb_tree_nav.py` for the seeding + drill helpers first):
|
||||||
|
- **the first load is small**: collect the network (`page.on("response")`) for the Sources view boot → exactly one `/api/docs/tree?depth=1` request; its JSON has `totals` with the expected `documents`/`chunks`, its top-level source nodes carry `documents` counts that include the deep files (recursive), and NO node deeper than level 1 exists while level-1 subfolders report `truncated === true` (assert on the parsed body, not on bytes);
|
||||||
|
- **the stat cards** match `totals` at the top level AND after drilling two levels (the numbers never change — they are catalogue-wide);
|
||||||
|
- **the drill**: click into a level-1 folder → exactly one `/api/docs/tree/node?source=…&path=…` request; rows render (subfolders sorted, then files in catalog order) with the stored description in the level block, the pending marker on the pending folder, and the phase-156 sticky header/scroll panel intact; click into a deeper folder → a second node fetch; go BACK to the first folder → **zero** new requests (cache);
|
||||||
|
- **the revalidation**: reload the view → the browser re-sends `If-None-Match` (observe the request header) and the response is `304` with an empty body while the table still renders (the `page.route`/`expect_response` idiom);
|
||||||
|
- **gzip**: assert the tree response's `Content-Encoding` is `gzip` and `Vary` includes `accept-encoding` (Chromium sends `Accept-Encoding: gzip, deflate, br` by default — pin what actually arrives, and if the harness negotiates identity, assert the server side in the integration suite instead and say so in the test docstring);
|
||||||
|
- **the vanished folder** (§7.4): a `page.route` stub that answers the node fetch with `404 {"detail":"unknown node"}` → the alert line + Retry button appear, the level the user was on stays rendered; click a parent crumb → the level is fine; then let the real route answer and Retry → the level loads;
|
||||||
|
- **in-flight**: a `page.route` handler that delays the node response → the clicked link is `aria-busy`/`aria-disabled` and the status line shows, and a second click does NOT issue a second request.
|
||||||
|
2. `tests/e2e/test_kb_tree.py` (MIGRATE, do not delete): the suite's "ONE fetch of `/api/docs/tree`, drill is client-side" assertions become the depth-1 + per-level assertions (the file's outline comment block first, then each assertion); every OTHER claim it owns (breadcrumb text, back navigation, table columns, dates, summary editor round-trip, pending markers, empty state, the anonymous gate) stays, re-aimed at the level the user is on. Each edit is a re-point, never a removal — state that in the commit body.
|
||||||
|
3. `tests/e2e/test_navbar_refresh.py:115` — the `("#nav-sources", "/sources.html", "/api/docs/tree")` expectation matches the new URL (keep the substring `/api/docs/tree` if the assertion is a substring test — verify, then pin precisely); re-run that suite.
|
||||||
|
4. `tests/e2e/test_kb_tree_nav.py`, `test_document_dates.py`, `test_git_source_dates.py`, `test_sync_summary_visibility.py`, `test_table_ux_polish.py`, `test_sources_midstream_bug.py`, `test_ls_tree_drilldown.py` (the agent's `ls` must be untouched) — run in isolation, repair ONLY where a condition assumed the full tree loaded (e.g. a test that expects a deep folder's rows before drilling — add the drill step; a test that walks the tree JSON — give it `?depth=8`, the repair task 02 item 9 defines), never by weakening an assertion.
|
||||||
|
5. Final gates + docs, in one pass:
|
||||||
|
- `uv run pytest` (unit + integration) green;
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → `app/` > 90% (report the number in the commit body);
|
||||||
|
- `uv run ruff check . && uv run pyright` clean;
|
||||||
|
- `uv run pytest tests/e2e/test_kb_tree_lazy_load.py -v --no-cov` green in isolation + the six regression suites in isolation + `tests/e2e/test_smoke.py`;
|
||||||
|
- `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` unchanged (retrieval is untouched — record it);
|
||||||
|
- `AUDIT.md` (task 05) reflects the FINAL state after these fixes; `README.md` + `AGENTS.md` reviewed in full once and every inconsistency from this phase fixed (the tree endpoint table, the env var list, the deploy note about self-gzip);
|
||||||
|
- UI Structure Check (AGENTS.md rule 5) for the drill: the level stays inside the 72rem container, the bounded scroll panel + sticky header survive a lazy render, the loading/alert lines are `role="status"`/`role="alert"`, the Retry button meets 44px + `:focus-visible`, contrast pairs are the existing AA tokens.
|
||||||
|
6. Move `.agents/phases/todo/167_api_payload_diet/` → `.agents/phases/complete/` (with `AUDIT.md`) and commit with the `00_phase.md` message, `--no-gpg-sign`, one atomic commit (AGENTS.md rule 8).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the new suite green in isolation + the six migrated/regression suites + `test_smoke.py` in isolation.
|
||||||
|
- Coverage **>90%** on `app/` including `compression.py`, `http_cache.py`, `response_budget.py` and the new route/builder lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Boot = one `depth=1` request whose body has `totals` and no deep children; drill = one node fetch per uncached level, zero on a cache hit; reload = 304.
|
||||||
|
- [ ] Every level renders the same data the pre-phase full tree rendered (counts, dates, descriptions, pending markers, thumbnails) — proven by the migrated `test_kb_tree.py` claims, none dropped.
|
||||||
|
- [ ] Failure paths are honest and recoverable (alert + Retry, level preserved, vanished folder resets the breadcrumb), and the agent's `ls` output is unchanged.
|
||||||
|
- [ ] The measured numbers are in `AUDIT.md`, the coverage number is in the commit body, `uv run pytest` green, ruff + pyright clean, one atomic `--no-gpg-sign` commit, phase dir in `complete/`.
|
||||||
@@ -0,0 +1,236 @@
|
|||||||
|
# Phase 167 · Task 05 — Measured API payload audit (the byte budget)
|
||||||
|
|
||||||
|
Owner TODO L5: "GET `/api/docs/tree` loads 2 MB in production … Fix that **any any
|
||||||
|
other endpoints** that could load a ton of data." This file is the measured answer —
|
||||||
|
every number below is pasted from `scripts/api_payload_audit.py` (or the `curl` /
|
||||||
|
server-log captures), never hand-written. LOCKED A5: the budget is a log line plus a
|
||||||
|
report, never a 413 — nothing was rejected, nothing was capped.
|
||||||
|
|
||||||
|
## How it was run
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| Date | 2026-10-08 |
|
||||||
|
| Corpus | the **1,000-document live-replica snapshot** (`data/bor_live_kb_replica.dump.sql`, the `restore-test-db` skill) — fingerprint verified `docs=1000`, `chunks=8866`, `sources=deploy,homelab,homepage,ServMon,skills,vibe-bot`; the snapshot's `alembic=0011` was upgraded to head `0027` afterwards (the phase-119/162 precedent) |
|
||||||
|
| App | `uv run uvicorn app.main:app --port 8000` (local dev, this working tree — tasks 01-04 of this phase included) |
|
||||||
|
| BEFORE run | the same app with `BOR_COMPRESS_MIN_BYTES=0` — the gzip layer is **not added at all**, i.e. byte-for-byte the transport the pre-phase app had |
|
||||||
|
| AFTER run | the same app with the shipped defaults (`BOR_COMPRESS_MIN_BYTES=860`, depth default 1, ETag on the three catalogue reads) |
|
||||||
|
| Command | `uv run python -m scripts.api_payload_audit --base-url http://localhost:8000` (budget default `262144`) |
|
||||||
|
| Budget log capture | `BOR_LOG_LARGE_RESPONSE_BYTES=0` (the loud mode — every `/api/*` response at INFO), used only for the log excerpts below |
|
||||||
|
|
||||||
|
The script is **read-only**: it signs in through `POST /api/login` (the session
|
||||||
|
cookie — the single-admin login is password-only, PLAN A10, so there is no
|
||||||
|
`--username`) and GETs. It never mutates, so it is safe against a production target
|
||||||
|
(`--base-url https://…`).
|
||||||
|
|
||||||
|
### Corpus caveat (read this before comparing with the 2 MB report)
|
||||||
|
|
||||||
|
The 2026-09-05 snapshot carries **0 `folder_summaries` rows and 0 image documents**
|
||||||
|
(verified: `select count(*) from folder_summaries` → `0`,
|
||||||
|
`select count(*) from documents where is_image` → `0`). The production 2 MB payload
|
||||||
|
was 1,000 documents **plus** the hundreds of inlined AI folder descriptions
|
||||||
|
(phase 127/155) and the image documents' full vision descriptions (phase 122/124) —
|
||||||
|
none of which exist in the snapshot. **So the identity bytes below are a lower bound
|
||||||
|
on the production payload**, and what transfers from this measurement is the
|
||||||
|
*structure*, i.e. the ratios: the depth limit cut the tree by **31×** (it ships the
|
||||||
|
ONE level the RAG view renders instead of the whole recursion below it — and every
|
||||||
|
level of a summary-carrying folder is where the production payload's fat lives),
|
||||||
|
gzip bought a further **5.5× to 20×** across these payloads (**7.5×** on the depth-1
|
||||||
|
tree, **11×** on the full tree), and a repeat load costs **0 bytes**. The verdict
|
||||||
|
table below names the live re-run as the follow-up.
|
||||||
|
|
||||||
|
## BEFORE — no gzip middleware (`BOR_COMPRESS_MIN_BYTES=0`), full tree available
|
||||||
|
|
||||||
|
> Read the BEFORE table's `gzip` column as "no compressor existed": ratio
|
||||||
|
> `1.000` on every row, the route's bytes ARE the transfer. Its `304` verdicts
|
||||||
|
> are NOT a pre-phase property — the conditional re-requests hit the validators
|
||||||
|
> tasks 01-04 added, which this run left in place because only the transport was
|
||||||
|
> being switched off (the honest way to measure "the same bytes, uncompressed").
|
||||||
|
> Pre-phase there were no validators at all, so every repeat load cost the
|
||||||
|
> identity column again.
|
||||||
|
|
||||||
|
```
|
||||||
|
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||||
|
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||||
|
endpoint identity gzip ratio ms verdict
|
||||||
|
----------------------------------------------------------------------------------------------------------------
|
||||||
|
tree depth=1 (default) 8,474 8,474 1.000 12 304
|
||||||
|
tree depth=8 (full) 265,147 265,147 1.000 37 304
|
||||||
|
tree node homelab/active 20,844 20,844 1.000 11 304
|
||||||
|
docs catalogue 246,366 246,366 1.000 14 304
|
||||||
|
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 892,124 1.000 8 LOOK (> 262144)
|
||||||
|
note: content field 772573 chars
|
||||||
|
saved chats list 12 12 1.000 1 ok
|
||||||
|
saved chat (one) — — — — n/a
|
||||||
|
note: no saved chats
|
||||||
|
git sources 188 188 1.000 1 ok
|
||||||
|
suggestions 74 74 1.000 1 ok
|
||||||
|
sync status (polled) 272 272 1.000 1 ok
|
||||||
|
tokens 13 13 1.000 2 ok
|
||||||
|
config 188 188 1.000 1 ok
|
||||||
|
whoami 37 37 1.000 1 ok
|
||||||
|
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||||
|
```
|
||||||
|
|
||||||
|
What the pre-phase browser actually paid for the **Sources page first paint** is the
|
||||||
|
sum of the rows it fetches on load, and the pre-phase tree route answered the FULL
|
||||||
|
recursive tree by default (`tree depth=8 (full)` is that shape):
|
||||||
|
`265,147 + 246,366 + 188 + 188 + 37 = 511,926 bytes identity`, and the pre-phase app
|
||||||
|
published **no validators at all**, so every repeat load re-paid the same 512 KB.
|
||||||
|
|
||||||
|
## AFTER — shipped defaults (depth-1 default + gzip + ETag)
|
||||||
|
|
||||||
|
api_payload_audit: base=http://localhost:8000 budget=262,144 bytes (LOOK threshold)
|
||||||
|
api_payload_audit: corpus documents=1000 chunks_total=8866 sources=6 tree totals documents=1000 chunks=8866 saved_chats=0 biggest_doc=tests/fixtures/advisory/bodhi_exiv2.json
|
||||||
|
endpoint identity gzip ratio ms verdict
|
||||||
|
----------------------------------------------------------------------------------------------------------------
|
||||||
|
tree depth=1 (default) 8,474 1,129 0.133 13 304
|
||||||
|
tree depth=8 (full) 265,147 23,520 0.089 41 304
|
||||||
|
tree node homelab/active 20,844 1,064 0.051 11 304
|
||||||
|
docs catalogue 246,366 45,107 0.183 23 304
|
||||||
|
document content (biggest: tests/fixtures/advisory/bodhi_exiv2.json) 892,124 60,776 0.068 14 LOOK (> 262144)
|
||||||
|
note: content field 772573 chars
|
||||||
|
saved chats list 12 12 1.000 2 ok
|
||||||
|
saved chat (one) — — — — n/a
|
||||||
|
note: no saved chats
|
||||||
|
git sources 188 188 1.000 1 ok
|
||||||
|
suggestions 74 74 1.000 2 ok
|
||||||
|
sync status (polled) 272 272 1.000 1 ok
|
||||||
|
tokens 13 13 1.000 2 ok
|
||||||
|
config 188 188 1.000 1 ok
|
||||||
|
whoami 37 37 1.000 1 ok
|
||||||
|
api_payload_audit: ms is the gzip (real-client) variant; identity is the route's own bytes — the pair the ' :: api ' log line reports.
|
||||||
|
```
|
||||||
|
|
||||||
|
Sources-page first paint, same sum against the AFTER gzip column:
|
||||||
|
`1,129 + 45,107 + 188 + 188 + 37 = 46,649 bytes on the wire` (from 511,926 identity,
|
||||||
|
and every one of those five rows revalidates to **304 / 0 bytes** on the next load).
|
||||||
|
The tree itself — the endpoint TODO L5 named — goes **265,147 → 1,129 = 235×**
|
||||||
|
(31× from the depth limit × 7.5× from gzip), and the second visit is free.
|
||||||
|
|
||||||
|
## `curl` proof on the audited corpus
|
||||||
|
|
||||||
|
```
|
||||||
|
$ curl -s -H 'Accept-Encoding: identity' -D- -o b1.json http://localhost:8000/api/docs/tree
|
||||||
|
etag: W/"53c86d5d554efd8487ad52ca6cd6f913"
|
||||||
|
cache-control: private, must-revalidate
|
||||||
|
content-length: 8474
|
||||||
|
vary: Cookie, Accept-Encoding
|
||||||
|
8474 bytes # wc -c b1.json
|
||||||
|
|
||||||
|
$ curl -s -H 'Accept-Encoding: gzip' -D- -o b2.gz http://localhost:8000/api/docs/tree
|
||||||
|
etag: W/"53c86d5d554efd8487ad52ca6cd6f913" # the validator survives the encoding
|
||||||
|
content-encoding: gzip
|
||||||
|
1129 bytes # wc -c b2.gz
|
||||||
|
|
||||||
|
$ curl -s -H 'If-None-Match: W/"53c86d5d554efd8487ad52ca6cd6f913"' -o /dev/null \
|
||||||
|
-w 'status=%{http_code} size=%{size_download}\n' http://localhost:8000/api/docs/tree
|
||||||
|
status=304 size=0
|
||||||
|
```
|
||||||
|
|
||||||
|
## The byte-budget log line (the answer to "which endpoint is heavy, right now")
|
||||||
|
|
||||||
|
Captured with `BOR_LOG_LARGE_RESPONSE_BYTES=0` (every `/api/*` response at INFO);
|
||||||
|
the shipped default is `65536`, so at rest only the big ones reach the log.
|
||||||
|
`grep ' :: api ' ` isolates them. `bytes=` is what the ROUTE produced, `enc=` what
|
||||||
|
went on the wire — the two together are the "how fat, and how fat did the client see
|
||||||
|
it" pair.
|
||||||
|
|
||||||
|
```
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=identity duration_ms=14
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree status=304 bytes=0 enc=identity duration_ms=11
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=identity duration_ms=34
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=265147 enc=gzip duration_ms=40
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs/tree/node status=200 bytes=20844 enc=gzip duration_ms=10
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/docs status=200 bytes=246366 enc=gzip duration_ms=22
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/documents/content status=200 bytes=892124 enc=gzip duration_ms=12
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/sync/status status=200 bytes=272 enc=identity duration_ms=0
|
||||||
|
INFO app.response_budget :: api method=GET path=/api/whoami status=200 bytes=37 enc=identity duration_ms=0
|
||||||
|
```
|
||||||
|
|
||||||
|
`POST /api/chat` (SSE, A15) is not in the audit table — a stream has no bounded
|
||||||
|
payload to budget — and it is skipped by the gzip layer by exact path (which is why
|
||||||
|
its line reads `enc=identity`). One real turn against this corpus, measured:
|
||||||
|
|
||||||
|
```
|
||||||
|
INFO app.response_budget :: api method=POST path=/api/chat status=200 bytes=39527 enc=identity duration_ms=45002 stream=1
|
||||||
|
```
|
||||||
|
|
||||||
|
39,527 bytes of SSE frames streamed, `duration_ms` the stream's life (this capture
|
||||||
|
ended because the measuring `curl --max-time 45` hung up — which is itself the
|
||||||
|
other half of the contract: a stream the client walks away from still writes
|
||||||
|
exactly ONE line, with the bytes that actually got out, never one line per frame
|
||||||
|
and never a missing line). Pinned by
|
||||||
|
`tests/unit/test_response_budget.py::test_a_cancelled_stream_still_logs_one_line_with_what_got_out`
|
||||||
|
and by the integration pin
|
||||||
|
`tests/integration/test_chat_api.py::test_chat_turn_logs_one_budget_line_with_the_stream_flag`.
|
||||||
|
|
||||||
|
## Verdict per candidate endpoint
|
||||||
|
|
||||||
|
| Endpoint | Measured (identity / wire) | Verdict |
|
||||||
|
|---|---|---|
|
||||||
|
| `GET /api/docs/tree` (default `depth=1`) | 265,147 → **8,474** identity, **1,129** gzip, repeat **304 / 0** | **FIXED** — the TODO L5 defect. Depth-limited default (task 02) + selective gzip (task 01) + ETag (task 03). Phase criterion "< 40 KB gzipped" met at 1,129 B with a 1,000-document corpus. |
|
||||||
|
| `GET /api/docs/tree?depth=8` | 265,147 / 23,520 gzip, repeat 304 / 0 | **FIXED (bounded by the route's own contract)** — the explicit full tree stays available for tooling and the migrated date suites; it compresses 11× and revalidates to 0. Nothing asks for it in the UI anymore (`grep` of `frontend/` shows only `depth=1` + the node route). |
|
||||||
|
| `GET /api/docs/tree/node` (the drill) | 20,844 / 1,064 gzip, repeat 304 / 0 | **FIXED** — one level per click (task 03/04), so the level the RAG view renders is ~1 KB on the wire instead of a re-ship of the corpus. |
|
||||||
|
| `GET /api/docs` (the catalogue) | 246,366 / 45,107 gzip, repeat 304 / 0 | **FIXED within scope**, and a **watch item**: 246 KB identity is 6 KB under the 256 KB LOOK bar. It already carries the ETag pair (task 03) and gzips 5.5×. Follow-up if the corpus grows: `depth`-style paging or a per-source filter param — named, not applied (A5: raise nothing, reject nothing). |
|
||||||
|
| `GET /api/documents/content` (biggest doc, 772 KB JSON) | 892,124 / 60,776 gzip | **ACCEPTED with why** — one document, on demand, when the owner opens it from a citation; the wire cost is 60 KB gzipped, and LOCKED A5 says the app's own documents are legitimately large. **Follow-up (named, not applied):** task 03 recorded this route as deliberately un-validated; if the `:: api ` log shows repeat 200s for the same document, give it the ETag pair keyed on `(source, path, content_hash)` (the row is already loaded, so the hash is free). |
|
||||||
|
| `GET /api/chats` | 12 (empty corpus) | **ok** — one row per saved chat, no payloads in the list (phase 50 shape). Re-measure on the live corpus with the saved chats. |
|
||||||
|
| `GET /api/chats/{id}` | n/a — `no saved chats` in the snapshot | **ACCEPTED with why** — the owner opening ONE saved conversation; the payload is that chat's raw `bor.chat.v1` JSONB, which is exactly what the page must render. Not measurable on this snapshot (the replica carries 0 saved chats) — **follow-up**: re-run this script against the live app, where the chats exist. |
|
||||||
|
| `GET /api/git-sources` | 188 / 188 (under the 860 floor → identity, correctly) | **ok** — the registry is a handful of rows; no remedy needed, and task 05 item 6's conditional ETag is therefore NOT applied (it grows past the budget only with hundreds of sources). |
|
||||||
|
| `GET /api/suggestions` | 74 | **ok** — three opening questions (phase 103). |
|
||||||
|
| `GET /api/sync/status` | 272 | **ACCEPTED with why (polled)** — tiny, but the UI polls it every 2 s, which is precisely why the byte-budget line exists: the log shows a chatty poll, and 272 B says the poll is not a payload problem. |
|
||||||
|
| `GET /api/tokens` | 13 | **ok** — secret-free list, one row per token (phase 79 A4). |
|
||||||
|
| `GET /api/config` | 188 | **ok** — display strings only. |
|
||||||
|
| `GET /api/whoami` | 37 | **ok** — two fields. |
|
||||||
|
| `POST /api/chat` (SSE) | not auditable as a payload (skipped by gzip, A15) | **UNCHANGED by design** — the log line covers it with ` stream=1`; frame-by-frame delivery is pinned by `tests/unit/test_compression.py` and `tests/integration/test_chat_api.py`. |
|
||||||
|
|
||||||
|
## What the audit changed, and what it proved
|
||||||
|
|
||||||
|
* **No new remedy was needed beyond tasks 01-04.** The only over-budget row
|
||||||
|
(`/api/documents/content`) is the documented accepted case, and
|
||||||
|
`/api/git-sources` measures 188 bytes, so task 05 item 6's conditional ETag
|
||||||
|
was **not** applied — a validator on a sub-860-byte response costs a hash
|
||||||
|
and saves nothing (it would not even compress).
|
||||||
|
* **The browser confirms the wire.** `tests/e2e/test_kb_tree_lazy_load.py` (task 06)
|
||||||
|
drives the real Sources view in Chromium and pins what this file measured: the
|
||||||
|
boot is ONE `?depth=1` request whose body carries `totals` and no deep children,
|
||||||
|
the response arrives `Content-Encoding: gzip` with `Accept-Encoding` in `Vary`
|
||||||
|
(Chromium's own negotiation — no stub), a reload re-sends `If-None-Match` and the
|
||||||
|
wire answers `304` for a fraction of the first transfer while the table still
|
||||||
|
renders from the cache, and a level the payload only names costs exactly one
|
||||||
|
`/api/docs/tree/node` fetch (zero on every re-entry). Because Chromium hands
|
||||||
|
`fetch` the cached 200 on a revalidation, that last pair is asserted on the CDP
|
||||||
|
Network events — the DevTools-panel truth, not Playwright's post-cache view.
|
||||||
|
* **The depth limit is where the win is, not gzip.** depth-1 vs depth-8 on
|
||||||
|
the same corpus is 8,474 vs 265,147 identity bytes (**31×**) — gzip is the
|
||||||
|
second ~11× on top of it (265,147 → 23,520), and the ETag makes the repeat
|
||||||
|
load free. Had the phase shipped gzip alone, the Sources page would still
|
||||||
|
have paid 23 KB/request forever, and on the summary-inflated production
|
||||||
|
corpus proportionally more.
|
||||||
|
* **`/api/docs` is the next one to watch** (246 KB identity, 6 KB under the
|
||||||
|
LOOK bar, already validated + compressed). The byte-budget line at its
|
||||||
|
default `65536` threshold logs it at INFO on every uncached load, so the
|
||||||
|
corpus growth that pushes it over shows up in the log before it shows up
|
||||||
|
for the owner.
|
||||||
|
* **Two limits of this measurement, stated instead of hidden:** (1) the
|
||||||
|
snapshot has no folder summaries and no image docs, so the identity
|
||||||
|
numbers are a lower bound on the production payload (see the caveat
|
||||||
|
above); (2) it has no saved chats, so `/api/chats/{id}` is `n/a` rather
|
||||||
|
than measured — the script is read-only by design (it must stay safe
|
||||||
|
against the deployed app), so it does not invent a chat to size. Both are
|
||||||
|
re-measurable with one command against the live corpus.
|
||||||
|
|
||||||
|
## Re-running this audit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. the corpus (destructive to the local DB — see the skill's own backup step)
|
||||||
|
# restore-test-db skill: data/bor_live_kb_replica.dump.sql → brain_of_reese
|
||||||
|
uv run alembic upgrade head # the snapshot carries an old alembic version
|
||||||
|
# 2. the app
|
||||||
|
uv run uvicorn app.main:app --port 8000
|
||||||
|
# 3. the measurement (BOR_ADMIN_PASSWORD from .env; --password to override)
|
||||||
|
uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
|
||||||
|
# the BEFORE transport, for a before/after pair like the tables above:
|
||||||
|
BOR_COMPRESS_MIN_BYTES=0 uv run python -m scripts.api_payload_audit
|
||||||
|
```
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# Phase 168 — Admin Settings panel: edit any `BOR_*` setting in the browser, DB over env, live on Save
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "I want an admin panel that can edit anything in .env and save it to the database. The database value should override the env var. The change should take effect on clicking 'save' on that page."
|
||||||
|
**Story:** n/a (owner-directed feature)
|
||||||
|
**Context:** `app/config.py` (75 `Settings` fields, `env_prefix="BOR_"`, `env_file=".env"`, `extra="ignore"`, `get_settings()` `@lru_cache`d, ~40 call sites across `app/`), `app/db.py` (module-level `settings = get_settings()` + `create_engine`), `app/main.py` (`settings = get_settings()` at import, middleware wired at boot, `_shell_routes`), `app/core/theming.py::effective_settings` (the DB-over-env resolver precedent, B1), `app/models.py::UiSettings` (the single-row admin-persistence precedent), `app/api/ui_settings.py` + `app/api/tokens.py` (router-wide `require_admin`), `app/core/auth.py::require_admin`, `frontend/index.html` (`#view-*` shell views; `#view-tokens` L1077, `#view-theme` L1269), `frontend/assets/router.js` (`VIEW` / `VIEW_PATH` / `VIEW_MODULES`), `frontend/assets/header.js` (~L197-230, the ship-hidden / reveal-for-admin nav contract), `frontend/assets/styles.css` (`.table-wrap` L2825 + `.table-wrap table thead th` sticky L2841 — the phase-156 bounded-scroll precedent), `app/core/caching.py::HTML_PAGES` (L169), `alembic/versions/` (next free: **0028**), `tests/e2e/conftest.py` (the `os.environ["BOR_PDF_PICTORIAL"] = "0"` global-pin precedent, ~L52).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
An admin opens **Settings** in the navbar, edits any setting that today lives only in `.env` (75 fields, grouped, type-correct inputs), clicks **Save**, and the new value is (a) stored in Postgres, (b) preferred over the env var from that instant on with **no restart**, and (c) visible in the running UI. Per-row **Reset** deletes the DB row so the env value wins again. The `.env` file is never written by the app.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — Storage: one `app_settings` table (task 01).** `key` = the **bare `Settings` field name** (`relevance_threshold`), `value` = the **env-native string form** (`"0.55"`, `"true"`, `"md,txt"`, a JSON array for the one `list[str]` field), `updated_at`. Storing bare strings (not typed columns, not a wide row) is what makes "anything in .env" work for all 75 fields with zero per-field DDL, and keeps the coercion in exactly one place (the pydantic validators). The `BOR_` prefix stays a *presentation* concern (`env_var = "BOR_" + key.upper()`), never stored — a stored prefix is a double-prefix typo waiting to happen.
|
||||||
|
|
||||||
|
**D2 — The overlay lives INSIDE `get_settings()` (task 02).** The ~40 existing call sites (`app/rag/retriever.py`, `app/api/chat.py`, middlewares, `scripts/`) must not change — that is the only way "anything" becomes live rather than "the three fields someone remembered to thread through". `get_settings()` keeps its `@lru_cache` and its zero-arg signature (`tests/{integration,e2e}/*.py` call `get_settings.cache_clear()` in 6 files — that surface must survive), and resolves: `Settings()` (env/`.env`, exactly today) → overlay the DB rows → validate → return. `invalidate_settings()` clears the cache and runs the reset hooks (D3). Hot-path cost after the first call: one `lru_cache` dict hit.
|
||||||
|
|
||||||
|
**D3 — "Takes effect on Save" = cache + the two in-process singletons (task 03).** Everything except two objects is read per use, so clearing the cache is enough for ~70 fields. The exceptions are `app/api/chat.py::_llm` (an `LLMClient` built once from `get_settings()` — holds `llm_base_url` / `effective_api_key` / `llm_chat_model` / `llm_embed_model` / timeouts / retries, L318-326) and `_chat_semaphore` (built once from `chat_max_concurrent`, L302-315). The overlay exposes a hook registry; `app/main.py::create_app()` registers the chat reset (`_llm = None`, `_chat_semaphore = None`) so `app/core/` never imports `app/api/` (dependency direction, PLAN §3). Settings wired at boot cannot be applied in-process (engine pools, `SessionMiddleware`, the static mount, the gzip middleware, logging) — they stay **editable** but carry a `restart_required` badge (LOCKED A4).
|
||||||
|
|
||||||
|
**D4 — The catalog is code, derived-checked, and complete (task 02).** `app/core/settings_catalog.py` declares, per field: `group` (mirrors `app/config.py`'s own section comments: App & branding, HTTP & logging, Database, LLM, Chat history, RAG & retrieval, Agent tools, Import & sources, Images, PDF, Docs push, Admin & security), `label`, `secret`, `restart_required`, `overridable`. A unit test asserts the catalog covers **exactly** `Settings.model_fields` (a new setting added without catalog metadata fails the test → it can never silently disappear from the panel).
|
||||||
|
|
||||||
|
**D5 — Fail loud on write, fail soft on read (tasks 02/04, LOCKED A5).** A write validates the FULL candidate set — every stored override plus the new one, layered over the env base — through the real `Settings` validators, so a cross-field typo (`lexical_support_floor` above a DB-overridden `relevance_threshold`) is a 422 that names the field and writes nothing. A read that finds an unusable row (hand-edited, or a validator changed in a later release) skips THAT key, logs `settings override ignored key=… reason=…` and applies the rest: a DB value must never brick boot, because a bricked app cannot be un-bricked from the panel.
|
||||||
|
|
||||||
|
**D6 — The kill switch, and the E2E poison guard (task 02).** `BOR_SETTINGS_DB=0` (env-only, never itself DB-overridable, read straight from `os.environ`) disables the overlay entirely — the pre-phase behavior, the house `agent_max_rounds`/`compress_min_bytes` convention. `tests/e2e/conftest.py` pins it to `0` for **every** E2E app the same way it already pins `BOR_PDF_PICTORIAL`: a row left behind by one suite would otherwise silently retune every later suite's app subprocess. The phase-168 suite opts back in on its own app fixture (the `test_pictorial_pdf.py` pattern) and `TRUNCATE app_settings` before AND after (the `test_admin_theme_tab.py:286` precedent).
|
||||||
|
|
||||||
|
**D7 — The API (task 04).** `app/api/admin_settings.py`, prefix `/admin/settings`, router-wide `dependencies=[Depends(require_admin)]` (the `ui_settings`/`tokens` pattern: anonymous AND token users 403): `GET` the catalog + effective values with a `source` badge (`db` | `env` | `default`), `PUT /{key}` set one value, `DELETE /{key}` reset to env, `DELETE` reset all (the panel's "Clear all overrides"). Secrets are **write-only** (LOCKED A4): `GET` reports `is_set`, never the plaintext, and the write log line carries the key and the action, never the value.
|
||||||
|
|
||||||
|
**D8 — The view (task 05).** The shell's eighth view: `#view-settings` in `frontend/index.html` + `frontend/assets/settings.js`, `/settings.html` added to `main.py::_shell_routes` and `caching.HTML_PAGES`, `VIEW`/`VIEW_PATH`/`VIEW_MODULES` in `router.js`, and `#nav-settings` (ship-hidden) in `index.html` / `document.html` / `login.html` / `shared.html` revealed by `header.js` — the exact phase-91 Theme-view wiring. Layout per PLAN §7 + AGENTS rule 5: the 72rem `.container`, one `.table-wrap` card (bounded `max-height: min(70vh, 64rem)`, sticky `thead`) with a `<tbody>` per group, type-correct controls (`<input type=number>` for int/float, a `true/false` `<select>` for bool, `<textarea>` for the CSV/JSON and long strings, a password input for secrets), a text source badge (B5: never color alone), a client-side `type=search` filter (the phase-101 tokens pattern), a page-level **Save settings** (dirty rows only), per-row **Reset**, and a `role="status"` / `role="alert"` live region (§7.4).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `91_admin_theme_tab` + `92_theme_save_and_coverage` + `93_theme_semantic_completion` (complete) — the DB-over-default resolver (B1), the admin-only shell view + gate pattern this view copies, and the nav reveal contract.
|
||||||
|
- `79_api_tokens` (complete) — router-wide `require_admin` + the bounded, searchable admin catalogue table with per-row actions.
|
||||||
|
- `156_table_ux_polish` (complete) — `.table-wrap` bounded scroll + sticky header, the `cell-clip` ellipsis contract for long action labels.
|
||||||
|
- `167_api_payload_diet` (complete) — the response byte budget: the new `GET /api/admin/settings` (~75 rows ≈ 20 KB) must stay under `BOR_LOG_LARGE_RESPONSE_BYTES` and the audit script's 256 KB bar; never a 413.
|
||||||
|
- `82_security_headers` + `81_login_rate_limit` (complete) — the CSP/`default-src 'self'` posture the new JS must live inside (no CDN, rule 6).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_app_settings_table.md` — `AppSetting` model + reversible migration `0028_app_settings.py` + its migration test.
|
||||||
|
2. `02_settings_overlay_resolver.md` — `app/core/settings_catalog.py` + `app/core/settings_overlay.py`, `get_settings()` becomes the cached env+DB resolver, `invalidate_settings()`, the `BOR_SETTINGS_DB` kill switch + the E2E conftest pin, the catalog-completeness unit pins.
|
||||||
|
3. `03_live_apply_on_save.md` — the reset-hook registry, the chat `_llm` / `_chat_semaphore` reset registered from `create_app()`, and the integration proof that a saved change bites **without a restart**.
|
||||||
|
4. `04_admin_settings_api.md` — `/api/admin/settings` GET / PUT `/{key}` / DELETE `/{key}` / DELETE all, the schemas, the full-candidate validation, the masked secrets, the value-free log line, the README API note.
|
||||||
|
5. `05_settings_view_ui.md` — the Settings shell view (`#view-settings`, `settings.js`, router + shell route + `HTML_PAGES` + nav reveal, `.table-wrap` catalogue, Save/Reset + live region).
|
||||||
|
6. `06_e2e_settings_panel.md` — `tests/e2e/test_admin_settings_panel.py`, the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_settings_catalog.py` (catalog ⇄ `Settings.model_fields` equality; the masked set is exactly `llm_api_key` + `admin_password` + `session_secret` + `database_url` (the last one is read-only but its URL carries credentials); the non-overridable set is exactly `database_url` + `embedding_dim`), `tests/unit/test_settings_overlay.py` (merge/coercion/validation-skip/`BOR_SETTINGS_DB=0`), `tests/unit/test_settings_frontend_view.py` (static source pins in the house style: the view section exists, `router.js` maps `/settings.html`, no CDN, `settings.js` builds DOM with `createElement`/`textContent`).
|
||||||
|
- **Integration:** `tests/integration/test_migration_0028.py` (up/down, A13), `tests/integration/test_admin_settings_api.py` (auth matrix anonymous/token-user/admin, 404 unknown key, 422 validator failures naming the field, masked secrets in the response body, the DB-over-env flip visible through `GET /api/config`, reset-to-env, `get_settings()` reflecting a write with no restart), plus the existing `get_settings.cache_clear()` suites staying green untouched.
|
||||||
|
- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`.
|
||||||
|
- **E2E:** `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_theme_tab.py`, `test_api_tokens.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_admin_auth.py`, `test_cache_busting.py`.
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
- **Log contract (AGENTS rule 10):** every settings write logs one line (`settings action=set|clear|clear_all key=… source=db|env`) with no value; `GET /api/admin/settings` appears in the `grep ' :: api '` byte audit and under the 256 KB bar (`uv run python -m scripts.api_payload_audit`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/settings.html` is an admin-only shell view: anonymous sees the sign-in gate, a token user never sees the nav link, and all four API routes 403 for both.
|
||||||
|
- [ ] Editing a value + **Save settings** stores it in `app_settings`, flips the row's badge from `env`/`default` to `DB`, and the effect is visible in the running app with **no restart** (pinned E2E: `BOR_INPUT_PLACEHOLDER` shows up in the composer; `BOR_RELEVANCE_THRESHOLD` / `BOR_LLM_CHAT_MODEL` are read from the overlay by a live request).
|
||||||
|
- [ ] **Reset** deletes the row and the env value wins again; **Clear all overrides** empties the table; `.env` is never opened for writing by any code path (unit-pinned: no `open(` on the env file in `app/`).
|
||||||
|
- [ ] Secrets never leave the server (`GET` body has no plaintext, `page.content()` has no secret), no write log line contains a value, and a bogus value is a 422 that names the field and stores nothing.
|
||||||
|
- [ ] A stored-but-invalid row (inserted directly, bypassing the API) is ignored at read with a logged reason — the app still boots and `/api/health` is 200.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-09 during phase authoring)
|
||||||
|
- **A1 — DB-only persistence:** the panel writes `app_settings` and **never** touches `.env`; the env var stays the base and the DB row wins.
|
||||||
|
- **A2 — the overlay sits inside `get_settings()`** (cached, `cache_clear()`-compatible) so all ~40 consumers go DB-over-env at once, behind the `BOR_SETTINGS_DB=0` kill switch.
|
||||||
|
- **A3 — `database_url` and `embedding_dim` are NOT DB-overridable** (read-only rows, reason shown: the engine is built from `database_url`, so an override is a chicken-and-egg brick; A6 fixes `chunks.embedding` at 768, so a dim override would corrupt retrieval).
|
||||||
|
- **A4 — boot-wired settings stay editable with a "restart required" badge** (pools, session middleware, static mount, gzip, log level); **secrets are masked and write-only** and never appear in a response body or a log line.
|
||||||
|
- **A5 — validate the full candidate set on write (422, nothing stored), fail soft on read** (skip the bad key, log it, boot anyway).
|
||||||
|
- **A6 — one story, one phase, one Playwright file** (AGENTS rule 4) — the backend tasks are covered by unit + integration, the live-effect contract by `tests/e2e/test_admin_settings_panel.py`.
|
||||||
|
|
||||||
|
## Assumptions (the TODO left these open; the executor does not guess)
|
||||||
|
- **K1:** the key is the bare field name; `BOR_UPPER` is derived for display/API presentation only.
|
||||||
|
- **K2:** the one `list[str]` field (`suggestions`) is edited as a JSON array in a `<textarea>` and stored in that same JSON form (pydantic parses it exactly like the env var); every other field is a scalar or CSV string.
|
||||||
|
- **K3:** a missing `app_settings` table (fresh deployment, migrations not applied) means "no overrides" and logs at DEBUG once — only an unusable *value* logs at ERROR.
|
||||||
|
- **K4:** "Save settings" sends only the dirty rows; the first failing row aborts the batch, shows its 422 detail, and keeps the other rows' edits in the form for correction.
|
||||||
|
- **K5:** history/audit beyond `updated_at` (who changed what, when, old value) is out of scope — single-admin homelab (PLAN §1 "out of scope").
|
||||||
|
- **K6:** the overlay is single-process (PLAN A12: in-process state, restart-clearing accepted, no Valkey); with more than one uvicorn worker a save is visible in the worker that served it and in every worker after its next resolution — the deployment runs one app process, and the panel notes nothing about it.
|
||||||
|
- **K7:** `.env` keys that are not `Settings` fields (`extra="ignore"`, e.g. `AIPI_KEY`, `DEBUGPY*`) are not editable — they have no `app/` consumer to retune. `BOR_LLM_API_KEY` remains the way to set the key (it wins over `$AIPI_KEY` via `effective_api_key`).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"
|
||||||
|
```
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# Task 01 — `app_settings`: the table that holds DB-side setting overrides
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "I want an admin panel that can edit anything in .env and save it to the database."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One narrow key/value table stores the admin's overrides, keyed by the bare `Settings` field name, with the value kept in its **env-native string form** — the storage half of phase-168 D1, with zero per-field DDL for the 75 settings.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — append the `AppSetting` class (after `FolderSummary`, the file's tail) in the house docstring style, naming the phase and the TODO line:
|
||||||
|
```python
|
||||||
|
class AppSetting(Base):
|
||||||
|
"""One admin setting override (phase 168, task 01).
|
||||||
|
|
||||||
|
``TODO.md`` L3: the admin Settings panel persists what used to be
|
||||||
|
``.env``-only knobs here, and the row WINS over the environment
|
||||||
|
(resolved by ``app.core.settings_overlay``, consumed by
|
||||||
|
``app.config.get_settings``). …
|
||||||
|
"""
|
||||||
|
|
||||||
|
__tablename__ = "app_settings"
|
||||||
|
|
||||||
|
#: The BARE ``Settings`` field name (``relevance_threshold``) — never
|
||||||
|
#: the ``BOR_…`` form: the prefix is presentation
|
||||||
|
#: (``"BOR_" + key.upper()``), so a stored prefix could double up.
|
||||||
|
key: Mapped[str] = mapped_column(String(100), primary_key=True)
|
||||||
|
#: The value in the SAME textual form the env var would carry
|
||||||
|
#: (``"0.55"``, ``"true"``, ``"md,txt"``, a JSON array for the one
|
||||||
|
#: ``list[str]`` field) — coercion and validation belong to
|
||||||
|
#: pydantic, in exactly one place (phase 168 D1/A5).
|
||||||
|
value: Mapped[str] = mapped_column(Text, nullable=False)
|
||||||
|
#: When the admin last saved it (the ``folder_summaries`` /
|
||||||
|
#: ``kb_overview.updated_at`` house style: server-side now()).
|
||||||
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), server_default=func.now(), nullable=False
|
||||||
|
)
|
||||||
|
```
|
||||||
|
Import `func` from `sqlalchemy` if the module does not already (it imports `DateTime`, `String`, `Text`, `Integer`, `select`-adjacent names — check the header first, do not duplicate an import).
|
||||||
|
No index: the whole table is read by primary key order in one `SELECT` per settings resolution (the `ui_settings` "display fact, never a query surface" note, 0027).
|
||||||
|
2. `alembic/versions/0028_app_settings.py` — `revision = "0028"`, `down_revision = "0027"`, docstring in the 0027 style (what the table is for, why the value is TEXT, why no index, why the downgrade is a clean drop). `upgrade()` = `op.create_table("app_settings", sa.Column("key", sa.String(length=100), primary_key=True), sa.Column("value", sa.Text(), nullable=False), sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False))`; `downgrade()` = `op.drop_table("app_settings")` with the A13 comment (dropping it leaves the 0027 schema byte-identical; the values are re-enterable from the panel).
|
||||||
|
3. `tests/integration/test_migration_0028.py` (new) — copy the `test_migration_0017.py` harness verbatim (the `alembic` fixture with the `db_available()` skip + the teardown `upgrade(cfg, "head")`, `_version`, the `information_schema` assertion helpers) and pin:
|
||||||
|
- 0027 → 0028 creates `app_settings` with `key` VARCHAR(100) NOT NULL and PK, `value` TEXT NOT NULL, `updated_at` TIMESTAMPTZ NOT NULL with the `now()` server default, and `alembic_version = '0028'`;
|
||||||
|
- the 0027 schema survives (`documents.page_strip_mode` still present);
|
||||||
|
- rows round-trip (insert `relevance_threshold`/`"0.55"` and `import_extensions`/`"md,txt"`, read them back);
|
||||||
|
- a duplicate `key` raises `IntegrityError` (the PK is the uniqueness guard — two rows for one setting would make "which one wins" undefined);
|
||||||
|
- `NULL` value is rejected (NOT NULL — "clear the override" is a DELETE, never an empty value, so the resolver never has to distinguish them);
|
||||||
|
- downgrade to 0027 → the table is GONE, the rest survives; re-upgrade → the table is back (A13 round-trip).
|
||||||
|
4. `.env.example` — add a short comment block at the top (under the header, before the first `BOR_` line) stating the precedence rule and the kill switch: the app reads `BOR_*` from the environment, an admin override saved in the database wins per key, the panel never writes this file, and `BOR_SETTINGS_DB=0` disables DB overrides (set it in the container env, not here — a `0` committed here is a silent "my change did nothing"). Task 02 implements what this comment describes; write it so it stays true.
|
||||||
|
ASSUMPTION (K3): `.env.example` is documentation only — nothing parses it at runtime except pydantic's `env_file=".env"`, so a comment cannot change behavior.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `uv run pytest tests/integration/test_migration_0028.py -v` (needs `podman compose up -d db`).
|
||||||
|
- Unit: no new module yet; `uv run pytest tests/unit -q` stays green (adding a model cannot change behavior — if something moved, it is a bug in this task).
|
||||||
|
- Coverage: **>90%** on `app/` — the added model is exercised by the migration test's insert/read plus task 02's resolver tests.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run alembic upgrade head` applies 0028 and `uv run alembic downgrade 0027` removes `app_settings` cleanly.
|
||||||
|
- [ ] `tests/integration/test_migration_0028.py` green (including the duplicate-key and NOT-NULL pins).
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change anywhere: `get_settings()` still reads env only (that is task 02), and no route exists yet.
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# Task 02 — The overlay: `get_settings()` resolves DB-over-env, behind a kill switch
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…save it to the database. The database value should override the env var."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A cached resolver that layers the `app_settings` rows over the environment and hands every existing consumer a valid `Settings` — no consumer changes, no restart, and a `BOR_SETTINGS_DB=0` switch that puts the pre-phase behavior back exactly.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/core/settings_catalog.py` (new) — the presentation + capability metadata, code-owned and complete:
|
||||||
|
- `@dataclass(frozen=True, slots=True) class SettingMeta`: `key: str`, `group: str`, `label: str`, `secret: bool = False`, `restart_required: bool = False`, `overridable: bool = True`.
|
||||||
|
- `CATALOG: dict[str, SettingMeta]` covering **every one** of `Settings.model_fields` (75 today). Groups, in this display order, with these members (they mirror `app/config.py`'s own section comments — if the executor disagrees with one placement, keep the count and note it in the commit body):
|
||||||
|
| Group | Fields |
|
||||||
|
|---|---|
|
||||||
|
| App & branding | `app_name` `app_version` `environment` `input_placeholder` `footer_text` `suggestions` |
|
||||||
|
| HTTP & logging | `log_level`↻ `log_large_response_bytes` `compress_min_bytes`↻ `static_dir`↻ |
|
||||||
|
| Database | `database_url`⛔ `db_pool_size`↻ `db_pool_max_overflow`↻ `db_pool_recycle`↻ `chat_max_concurrent` |
|
||||||
|
| LLM (aipi) | `llm_base_url` `llm_api_key`🔒 `llm_chat_model` `llm_embed_model` `llm_summary_model` `stream_thinking` `llm_retries` `llm_retry_delay` `llm_timeout` |
|
||||||
|
| Chat history | `history_max_turns` `history_max_chars` |
|
||||||
|
| Retrieval & honesty | `embedding_dim`⛔ `top_n_docs` `relevance_threshold` `lexical_support_floor` `source_usefulness_floor` `related_max_docs` `suggested_docs` `suggestion_preview_chars` `hybrid_vector_candidates` `hybrid_lexical_candidates` `rrf_k` `recency_boost` `recency_half_life_days` `name_hit_bonus` |
|
||||||
|
| Chunking & embedding | `chunk_target_chars` `chunk_overlap_chars` `embed_batch_size` `embed_question_max_chars` |
|
||||||
|
| Prompt budgets | `max_output_tokens` `steering_max_chars` `summary_max_chars` `kb_overview_max_chars` `overview_input_max_chars` `folder_summary_input_max_chars` |
|
||||||
|
| Agent tools | `agent_max_rounds` `read_max_chars` |
|
||||||
|
| Import & sources | `import_extensions` `git_sources` `sources_dir` `upload_dir` `upload_max_mb` |
|
||||||
|
| Images | `images` `image_extensions` `image_dir` `chat_image_dir` `chat_image_max_mb` `answer_images` |
|
||||||
|
| PDF | `pdf_pictorial` `pdf_pictorial_min_chars` `pdf_image_skip_blank` `pdf_page_dir` |
|
||||||
|
| Docs push | `docs_repo` `docs_branch` `docs_base_branch` `docs_work_dir` |
|
||||||
|
| Admin & security | `admin_password`🔒 `session_secret`🔒↻ `session_max_age`↻ `session_cookie`↻ |
|
||||||
|
`↻` = `restart_required=True` (wired once at boot: logging, the gzip middleware in `app/main.py`, the static mount, `create_engine` in `app/db.py`, `SessionMiddleware`); `⛔` = `overridable=False` (LOCKED A3 — `database_url` is what builds the engine that would read the override, `embedding_dim` is fixed in the `chunks` table by A6); `🔒` = `secret=True`, plus `database_url` as a fourth masked field (its URL carries credentials) even though it is read-only.
|
||||||
|
- `ENV_PREFIX = "BOR_"`, `def env_var(key: str) -> str` → `"BOR_" + key.upper()`; `GROUPS: tuple[str, ...]` (the table order above); `SECRETS`, `RESTART_REQUIRED`, `NON_OVERRIDABLE` derived from `CATALOG` (never re-listed by hand).
|
||||||
|
- Module docstring: the phase, the TODO line, why the catalog exists (the panel is generated, not hand-written) and the completeness pin.
|
||||||
|
2. `app/core/settings_overlay.py` (new) — the mechanism:
|
||||||
|
- `resolve_settings() -> Settings` — the whole overlay: `base = Settings()`; `if not enabled(): return base`; `rows = load_overrides()`; `return merge(rows)[0]`. **Never call `app.config.get_settings()` here** — `Settings()` is the env-only entry point (recursion guard, pinned by test).
|
||||||
|
- `enabled() -> bool` — `os.environ.get("BOR_SETTINGS_DB", "1")` is not one of `0/false/no/off` (the house bool parsing), read **per call** from `os.environ` so tests can flip it; the flag itself is not a `Settings` field, so it can never be turned off from the database (the escape hatch cannot disable itself).
|
||||||
|
- `load_overrides() -> dict[str, str]` — `SELECT key, value FROM app_settings ORDER BY key` through a module-level lazily-created engine: `create_engine(Settings().database_url, poolclass=NullPool, pool_pre_ping=True)` (NullPool: one connection per resolution, none held — the app's real pool belongs to `app/db.py` and must not be used here: importing `app.db` from the overlay would be a cycle, `app/db.py` imports `app.config`, and the pool whose size this table can change must not be the one that reads the table). Fail-soft: `ProgrammingError`/undefined-table → `{}` + one DEBUG line per process (K3 — a pre-migration deployment); any other `SQLAlchemyError`/`OSError` → `{}` + one WARNING per process naming the exception class ("settings overrides unavailable — env values in effect"). Both "once per process" guards are module-level flags, reset by `reset_state_for_tests()`.
|
||||||
|
- `merge(rows: Mapping[str, str]) -> tuple[Settings, list[tuple[str, str]]]` — returns `(settings, skipped)` where `skipped` is `(key, reason)` pairs. Build the kwargs from the rows: drop keys not in `Settings.model_fields` (an orphan row from a removed setting) and keys with `overridable=False` (a hand-inserted `database_url` row must never hijack the engine — LOCKED A3), coerce each value (`_coerce(field, raw)`: `json.loads` for the one `list[str]` field `suggestions`, the raw string for everything else — pydantic-settings lax-coerces `str`→`int`/`float`/`bool`, and init kwargs rank ABOVE env in pydantic-settings, which is precisely "the database value overrides the env var"). Then `Settings(**kwargs)`; on `ValidationError` drop the named field (`error["loc"][0]`), append it to `skipped` with the validator's message, and retry — so one bad row costs one setting, not the whole table (LOCKED A5). Log each skipped key once at ERROR: `settings override ignored key=… reason=…`.
|
||||||
|
- `register_invalidator(fn: Callable[[], None]) -> None` + `run_invalidators()` — the hook list task 03 uses; a hook that raises is logged, never propagated (a failed reset must not undo a successful save).
|
||||||
|
- `reset_state_for_tests() -> None` — clears the module flags/engine for the unit suites.
|
||||||
|
3. `app/config.py` — keep the `Settings` class 100% untouched and swap the resolver tail:
|
||||||
|
```python
|
||||||
|
@lru_cache
|
||||||
|
def get_settings() -> Settings:
|
||||||
|
"""Effective settings: env/.env with the admin's DB overrides layered
|
||||||
|
on top (phase 168, TODO L3). … cached exactly as before —
|
||||||
|
``get_settings.cache_clear()`` still works — and
|
||||||
|
``invalidate_settings()`` is the save-side wrapper that also runs the
|
||||||
|
runtime reset hooks (task 03)."""
|
||||||
|
from app.core.settings_overlay import resolve_settings # lazy: app.core imports app.config
|
||||||
|
|
||||||
|
return resolve_settings()
|
||||||
|
|
||||||
|
|
||||||
|
def invalidate_settings() -> None:
|
||||||
|
get_settings.cache_clear()
|
||||||
|
from app.core.settings_overlay import run_invalidators
|
||||||
|
|
||||||
|
run_invalidators()
|
||||||
|
```
|
||||||
|
The lazy imports are load-bearing (documented in the comment): `app.core.*` imports `app.config` at module scope, and `app/db.py` → `app.config` runs at import time; a module-scope `app.config` → `app.core` import would deadlock the import graph. The 6 existing `get_settings.cache_clear()` test call sites keep working unchanged (they only retune env, so skipping the hooks there is correct).
|
||||||
|
4. `app/main.py` — inside `create_app()`, immediately after `ensure_admin_configured(settings)`: `invalidate_settings()` then `effective = get_settings()` and ONE boot log line, `settings overrides loaded count=N` (or `source=env-only disabled` when the kill switch is off) — the prime that turns a boot-time DB miss into a retry rather than a permanently cached env-only view. Keep the module-level `settings = get_settings()` as is (it is the pre-boot value the middleware wiring uses today).
|
||||||
|
5. `tests/conftest.py` — next to the existing `os.environ.setdefault("BOR_ADMIN_PASSWORD", …)` block, add `os.environ.setdefault("BOR_SETTINGS_DB", "0")` with a comment: the shared test DB must never retune the in-process suites (the same poison-guard reasoning as the phase-140 `BOR_PDF_PICTORIAL` pin in `tests/e2e/conftest.py`), and tests that WANT the overlay set the var + `get_settings.cache_clear()` themselves.
|
||||||
|
6. `tests/e2e/conftest.py` — same pin beside the `os.environ["BOR_PDF_PICTORIAL"] = "0"` line (~L52): `os.environ["BOR_SETTINGS_DB"] = "0"`, comment it as the cross-suite poison guard (one suite's saved row would silently change every later suite's app subprocess). The phase-168 E2E suite overrides it to `"1"` in its own module app fixture (the `test_pictorial_pdf.py:339` `env["BOR_PDF_PICTORIAL"] = "1"` pattern) — task 06 does that.
|
||||||
|
7. `tests/unit/test_settings_catalog.py` (new) — `set(CATALOG) == set(Settings.model_fields)` (both directions, the diff in the failure message — this is the pin that stops a new setting from being invisible in the panel); `every group in GROUPS exists in CATALOG and every CATALOG group is in GROUPS`; `SECRETS == {llm_api_key, admin_password, session_secret, database_url}`; `NON_OVERRIDABLE == {database_url, embedding_dim}`; `env_var("relevance_threshold") == "BOR_RELEVANCE_THRESHOLD"`; no label is empty and no label duplicates another.
|
||||||
|
8. `tests/unit/test_settings_overlay.py` (new) — pure/unit level, no DB: `enabled()` matrix (`"0"/"false"/"no"/"off"` off; unset/`"1"/"true"` on); `_coerce` for int/float/bool/str and `suggestions` JSON (a malformed JSON value is skipped, not fatal); `merge` happy path over the real env (`{"relevance_threshold": "0.55"}` → float 0.55, everything else byte-equal to `Settings()`), `merge` skips an unknown key and a `NON_OVERRIDABLE` key with a reason, `merge` drops one invalid value and keeps the rest, a cross-field violation (`lexical_support_floor` above a DB-overridden `relevance_threshold`) lands in `skipped` instead of raising, bool spellings (`"false"`, `"0"`) behave like the env, `load_overrides()` returns `{}` + one WARNING when the engine raises, `register_invalidator`/`run_invalidators` runs hooks in order and survives a raising hook, and a pin that `app/core/settings_overlay.py` contains no `get_settings(` call (the recursion guard) and that no file this phase owns writes `.env` (LOCKED A1 — assert the absence of an `open(` / `.env` write in `app/core/settings_overlay.py`, `app/config.py` and — once it exists — `app/api/admin_settings.py`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the two new files; then `uv run pytest tests/unit -q` (the `get_settings` surface is unchanged for every existing pin) and `uv run pytest tests/integration/test_api.py -q` (it calls `get_settings.cache_clear()` — it must pass untouched).
|
||||||
|
- Integration: `get_settings()` with `BOR_SETTINGS_DB=0` is byte-identical to `Settings()` — pin it in `tests/integration/test_admin_settings_api.py` (task 04) or here if the DB is up.
|
||||||
|
- Coverage: **>90%** on the new modules — the fail-soft branches (undefined table, generic error, invalid value, non-overridable row) each need a test.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run python -c "from app.config import Settings, get_settings; print(get_settings().relevance_threshold)"` prints the env value, and `BOR_RELEVANCE_THRESHOLD=0.11 …` still prints 0.11 (the env path is untouched).
|
||||||
|
- [ ] With a live DB and one hand-inserted `app_settings` row + `BOR_SETTINGS_DB=1` + `invalidate_settings()`, `get_settings()` reports the DB value; with `BOR_SETTINGS_DB=0` it reports the env value.
|
||||||
|
- [ ] A hand-inserted row for `database_url` or `embedding_dim` changes nothing and logs the skip.
|
||||||
|
- [ ] `tests/{conftest,e2e/conftest}.py` pin the switch off; the whole `uv run pytest` suite passes with a *populated* `app_settings` table present (the poison guard proven by running the suite twice: once with the table empty, once with a garbage row in it that must change nothing).
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Task 03 — Live apply: Save re-points the two in-process singletons
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "The change should take effect on clicking \"save\" on that page."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
After `invalidate_settings()`, the next request behaves under the new value with **no restart**: the ~70 per-use settings are covered by the cache clear alone, and the two objects that captured a `Settings` snapshot — the shared `LLMClient` and the chat concurrency semaphore — are rebuilt.
|
||||||
|
|
||||||
|
## Why only two objects (traced 2026-10-09, do not re-derive)
|
||||||
|
Every other setting is read through `get_settings()` at the point of use (`app/rag/retriever.py:408,700,807,860`, `app/core/response_budget.py:172`, `app/api/sync.py:528`, the `LLMClient()` constructions in `app/api/sync.py:532`, `app/api/docs.py:480`, `app/api/git_sources.py:944`, the path fields resolved per operation), so clearing the cache is the whole job for them. `app/core/caching.py`'s `functools.cache` `asset_version()` is the deploy token (A19), not a setting. The genuine snapshots are:
|
||||||
|
|
||||||
|
- `app/api/chat.py:318-326` — `_llm: LLMClient | None`, built once by `get_llm()`; `LLMClient.__init__` (`app/rag/llm.py:246-254`) holds `self.settings` and bakes `llm_base_url`, `effective_api_key` and `llm_timeout` into the `AsyncOpenAI` client, while `llm_chat_model` / `llm_embed_model` / `llm_summary_model` are read off that captured snapshot on every call.
|
||||||
|
- `app/api/chat.py:301-315` — `_chat_semaphore: asyncio.Semaphore | None`, built once from `chat_max_concurrent`.
|
||||||
|
|
||||||
|
Boot-wired values (`db_pool_*`, `session_*`, `static_dir`, `log_level`, `compress_min_bytes`, `database_url`) are LOCKED as `restart_required` / non-overridable — this task does not attempt them (A3/A4).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/chat.py` — add, directly below `get_llm()` (and documented as the phase-168 counterpart of the lazy-init comments above it):
|
||||||
|
```python
|
||||||
|
def reset_runtime_caches() -> None:
|
||||||
|
"""Drop the settings-derived singletons (phase 168, task 03).
|
||||||
|
|
||||||
|
Called by ``invalidate_settings()`` after the admin saves. The next
|
||||||
|
turn rebuilds the client from the CURRENT ``get_settings()`` — an
|
||||||
|
in-flight turn keeps the client it started with (its ``AsyncOpenAI``
|
||||||
|
transport is mid-request; swapping it would kill the stream), which
|
||||||
|
is the same accepted-semantics note as A12's in-process state.
|
||||||
|
``_chat_active`` is deliberately untouched: it counts live turns, and
|
||||||
|
the new ``chat_max_concurrent`` applies as permits are taken.
|
||||||
|
"""
|
||||||
|
global _llm, _chat_semaphore
|
||||||
|
_llm = None
|
||||||
|
_chat_semaphore = None
|
||||||
|
```
|
||||||
|
2. `app/main.py` — in `create_app()`, right after the task-02 prime line: register the hook and log it once.
|
||||||
|
```python
|
||||||
|
# Phase 168 (task 03): the settings panel's Save must retune the running
|
||||||
|
# process, and app/core must never import app/api (PLAN §3's direction) —
|
||||||
|
# so create_app(), the composition root, wires the reset hook.
|
||||||
|
register_invalidator(reset_runtime_caches)
|
||||||
|
```
|
||||||
|
with `from app.api.chat import reset_runtime_caches` added to the existing `app.api.chat` import block and `from app.core.settings_overlay import register_invalidator` alongside the other `app.core` imports. Importing `app.api.chat` here is already done (the router import at the top of the file), so this adds no new import cost or cycle.
|
||||||
|
3. `tests/unit/test_settings_live_apply.py` (new) — source/behaviour pins in the house style:
|
||||||
|
- `reset_runtime_caches()` nulls both globals: set `app.api.chat._llm = object()` and `app.api.chat._chat_semaphore = object()`, call it, assert both are `None`; assert `_chat_active` is untouched by the call;
|
||||||
|
- `get_llm()` rebuilds after a reset and returns the SAME instance on two calls with no reset (the singleton contract still holds — this pin stops a future "always construct" shortcut that would break connection pooling);
|
||||||
|
- `_get_chat_semaphore()` rebuilds after a reset and reflects a changed `chat_max_concurrent` (drive it with `monkeypatch.setenv("BOR_CHAT_MAX_CONCURRENT", "3")` + `get_settings.cache_clear()`, assert `_chat_semaphore._value == 3`), then restore;
|
||||||
|
- `app/main.py`'s source contains `register_invalidator(reset_runtime_caches)` inside `create_app` (the static pin that stops the wiring being "cleaned up" — without it a Save silently stops retuning the LLM client);
|
||||||
|
- `invalidate_settings()` runs registered hooks and clears the cache (call it after registering a counter, assert the counter moved and that `get_settings()` afterwards re-resolves);
|
||||||
|
- a hook that raises does not propagate out of `invalidate_settings()` (a failed reset must not turn a committed save into a 500).
|
||||||
|
4. `tests/integration/test_admin_settings_live.py` (new; DB required — use the existing `db` fixture + `pytest.skip` idiom from `tests/integration/test_ui_settings_api.py`) — the end-to-end proof of "no restart", in-process:
|
||||||
|
- start from an empty `app_settings` and `monkeypatch.setenv("BOR_SETTINGS_DB", "1")` + `invalidate_settings()`, teardown `DELETE FROM app_settings` + `invalidate_settings()` + `get_settings.cache_clear()` (never leave a row behind for the suites that run after);
|
||||||
|
- `relevance_threshold`: read `get_settings().relevance_threshold` (the env/test value), `INSERT` the row `('relevance_threshold','0.5')` directly, `invalidate_settings()`, assert `get_settings().relevance_threshold == 0.5` and that a retrieval call that consults it (`app/rag/retriever.py`'s gate reads it at `:408`) now behaves under 0.5 — assert through the function the way `tests/integration/test_recency_boost.py` does, no chat turn needed;
|
||||||
|
- `llm_chat_model`: with the row `('llm_chat_model','gpt-test')`, `invalidate_settings()`, then `app.api.chat.get_llm().settings.llm_chat_model == "gpt-test"` **and** the rebuilt client's `AsyncOpenAI` base_url/timeout come from the new snapshot (set `('llm_timeout','7')` in the same batch and assert `client._client.timeout.connect == 7.0`, the way `app/rag/llm.py` builds it) — this is the pin that a Save actually swaps the client;
|
||||||
|
- `chat_max_concurrent`: same shape, assert the rebuilt semaphore's `_value` (via `_get_chat_semaphore()`) follows the saved number;
|
||||||
|
- the kill-switch counterpart: with rows present but `BOR_SETTINGS_DB=0`, `invalidate_settings()` → the env values are back (the switch really is the whole feature's kill switch);
|
||||||
|
- `import_extensions` (the `scripts/` consumer): after the save, `get_settings().import_extension_set` reflects the DB value — the importer and Sync read the same overlay (no separate code path).
|
||||||
|
5. `.env.example` — extend task 01's comment block with the one sentence that matters operationally: values marked "restart required" in the panel (DB pool sizes, session cookie wiring, log level, compression, the static dir) apply on the next `podman compose restart app`, everything else applies immediately.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `uv run pytest tests/unit/test_settings_live_apply.py -v`, then `uv run pytest tests/unit -q`; the existing chat pins (`tests/unit/test_chat_*`, `tests/integration/test_chat_api.py`, `test_chat_concurrency.py`) pass untouched — the reset only runs when `invalidate_settings()` is called, which nothing in them calls.
|
||||||
|
- Integration: `uv run pytest tests/integration/test_admin_settings_live.py -v` with the DB up; then `uv run pytest tests/integration -q` to prove the teardown left nothing behind.
|
||||||
|
- Coverage: **>90%** on `app/api/chat.py`'s new function and `app/core/settings_overlay.py`'s hook paths.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Saving a value (row insert + `invalidate_settings()`) changes what the NEXT request does — pinned for a retrieval setting, the LLM client (model + timeout), and the chat concurrency cap, with no process restart.
|
||||||
|
- [ ] An in-flight turn is never swapped mid-stream (the docstring's contract; the unit test proves `get_llm()` hands out the same object until a reset).
|
||||||
|
- [ ] `register_invalidator(reset_runtime_caches)` is wired in `create_app()` and unit-pinned; `app/core/` imports no `app/api/` symbol (grep-clean).
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Task 04 — `/api/admin/settings`: the admin-only read/write surface
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…can edit anything in .env and save it to the database."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One admin-only router that serves the whole catalogue (every setting, its effective value, and where that value came from) and writes single values back to `app_settings` — validated by the real `Settings` validators before anything is stored, masked where the value is a secret.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py` — append the four models (docstrings in the `UiSettingsIn`/`UiSettingsOut` style, naming the phase and the route):
|
||||||
|
- `AdminSettingItem`: `key: str`, `env_var: str`, `group: str`, `label: str`, `type: Literal["int","float","bool","str","json"]`, `secret: bool`, `restart_required: bool`, `overridable: bool`, `source: Literal["db","env","default"]`, `value: str | None` (the effective value in env-native string form; **`None` when masked**), `masked: bool`, `has_value: bool` (for a masked field: is a non-empty value in effect at all), `has_override: bool` (a row exists), `db_value: str | None` (the stored string when it is a stored non-secret override, else `None`), `default: str` (the code default stringified — for a masked field `""`).
|
||||||
|
- `AdminSettingsOut`: `items: list[AdminSettingItem]`, `totals: {"count": int, "overridden": int}`, `overlay_enabled: bool` (the `BOR_SETTINGS_DB` state — when `false` the panel shows a banner instead of pretending a save took effect).
|
||||||
|
- `AdminSettingIn`: `value: str` — always a string, coercion belongs to pydantic (the same reason D1 stores strings).
|
||||||
|
- `AdminSettingsCleared`: `cleared: int`.
|
||||||
|
Stringify helpers live in the router (`_display(field, value)`), not in the schemas: `bool` → `"true"/"false"`, `list[str]` → `json.dumps(...)`, everything else `str(value)`.
|
||||||
|
2. `app/api/admin_settings.py` (new) — `router = APIRouter(prefix="/admin/settings", tags=["admin-settings"], dependencies=[Depends(require_admin)])` (the `app/api/ui_settings.py` pattern: anonymous AND token users 403 on every route). Module docstring lists the routes and the three invariants (validate-before-store, masked secrets, never write `.env`).
|
||||||
|
- `GET ""` → `AdminSettingsOut`: read the rows once (`SELECT key, value FROM app_settings`), resolve the effective `Settings` through `get_settings()`, then emit one `AdminSettingItem` per catalog entry **in `GROUPS` order, catalog order within the group** (a stable order the E2E pins against). `source`: `db` when an *allowed* row is in effect, `env` when the process env has the `BOR_*` var set (check `os.environ`), else `default`. A masked field emits `masked=True`, `value=None`, `db_value=None` and `has_value=bool(effective)`.
|
||||||
|
- `PUT "/{key}"` (`AdminSettingIn` → `AdminSettingItem`, 200):
|
||||||
|
- unknown key → 404 `f"Unknown setting {key}"`;
|
||||||
|
- `overridable=False` → 422 `f"{env_var} is not settable from the database"` + the reason in the docstring (A3: the engine is built from `database_url`; `chunks.embedding` is fixed at `embedding_dim` by A6);
|
||||||
|
- `secret=True` and the value is empty/whitespace → 422 `f"{env_var} must not be blank — use Reset to fall back to the environment"` (an empty API key or session secret is a self-lockout; `DELETE` is the explicit clear);
|
||||||
|
- **candidate-set validation** (A5/D5): the stored rows for the allowed keys + this key/value, run through the same `app.core.settings_overlay.merge()` path — a `ValidationError` (bad shape OR a cross-field rule like `lexical_support_floor` above a DB-overridden `relevance_threshold`) is a 422 whose detail names the field, and **nothing is written**;
|
||||||
|
- write: `INSERT … ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = now()` (`sqlalchemy.dialects.postgresql.insert`), commit, then `invalidate_settings()` (task 03 makes it live) and return the refreshed item.
|
||||||
|
- `DELETE "/{key}"` → `AdminSettingItem`: delete the row (idempotent — deleting an absent key is a success, not a 404), `invalidate_settings()`, return the item as `source="env"|"default"`.
|
||||||
|
- `DELETE ""` → `AdminSettingsCleared`: `DELETE FROM app_settings`, `invalidate_settings()`, `{"cleared": rowcount}` (the panel's "Clear all overrides").
|
||||||
|
- **Log one line per write, never the value** (AGENTS rule 10, A4): `settings action=set key=… status=200`, `settings action=clear key=…`, `settings action=clear_all count=N` on the module logger `app.api.admin_settings`. GETs log nothing here — the phase-167 `ResponseBudgetMiddleware` already writes the `api method=… bytes=…` line, and the whole catalogue must stay under `BOR_LOG_LARGE_RESPONSE_BYTES`' INFO bar (75 items ≈ 20 KB; verified in task 06 with the audit script).
|
||||||
|
- Never opens `.env`, never shells out, never `eval`s a value — the module contains no file writes (unit-pinned).
|
||||||
|
3. `app/main.py` — `from app.api.admin_settings import router as admin_settings_router` in the router-import block and `app.include_router(admin_settings_router, prefix="/api")` right after the `ui_settings_router` line, with the one-line comment the neighbours carry ("admin-only (router-wide require_admin; a token user stays 403)").
|
||||||
|
4. `tests/integration/test_admin_settings_api.py` (new; the `db`-fixture + skip idiom of `tests/integration/test_ui_settings_api.py`, admin client per that file's convention). Set `BOR_SETTINGS_DB=1` + `invalidate_settings()` in the fixture, and in teardown `DELETE FROM app_settings` + `invalidate_settings()` + `get_settings.cache_clear()` — **no row may survive this module** (the same discipline `test_admin_theme_tab.py:286` applies to `ui_settings`). Pin:
|
||||||
|
- **auth matrix:** anonymous `403`, token user `403` (issue a token the way `test_tokens_api.py` does), admin `200` — on all four routes;
|
||||||
|
- **shape:** `items` length == `len(Settings.model_fields)`, groups in `GROUPS` order, `totals.count`/`overridden` correct, `overlay_enabled` true; on an empty table a field whose `BOR_*` var the test env sets reports `source="env"`, an untouched one `source="default"`, and none reports `"db"`;
|
||||||
|
- **write → live:** `PUT {"value": "0.5"}` for `relevance_threshold` → 200 with `source="db"`, `get_settings().relevance_threshold == 0.5`, `totals.overridden == 1`, and the row present with a non-null `updated_at`;
|
||||||
|
- **visible-through-the-app proof:** `PUT input_placeholder` then `GET /api/config` (no auth) shows the DB value — the DB-over-env rule reaches the public config endpoint through the same overlay;
|
||||||
|
- **errors:** unknown key → 404; `database_url` / `embedding_dim` → 422 naming the env var; `"abc"` for `relevance_threshold` → 422 naming `relevance_threshold` and the table stays empty; a cross-field pair that only breaks in combination (store `relevance_threshold=0.2`, then `lexical_support_floor=0.9`) → the second `PUT` 422s and the first row is unchanged;
|
||||||
|
- **masking:** store `llm_api_key` directly, `GET` reports `masked=True`, `value is None`, `has_value=True`, and the plaintext appears NOWHERE in `response.text` (assert on the raw text, not the parsed model); blank `PUT` for it → 422; `DELETE` clears it;
|
||||||
|
- **reset:** `DELETE /{key}` → the item is `source="env"` (with the env var set in the fixture) and `get_settings()` is back to the env value; `DELETE ""` with 3 rows → `{"cleared": 3}` and `overridden == 0`;
|
||||||
|
- **kill switch:** with `BOR_SETTINGS_DB=0` (monkeypatch + `invalidate_settings()`) `overlay_enabled` is `false` while a `PUT` still stores the row and `get_settings()` still reports the env value — the switch disables the mechanism, not the panel;
|
||||||
|
- **byte budget:** `len(response.content)` for the full `GET` is under 64 KB (the phase-167 INFO bar) — so the catalogue never floods the log.
|
||||||
|
5. `README.md` — under **Configuration reference** (L1071): a short lead paragraph stating the precedence order (`app_settings` row → `BOR_*` environment → code default), that the panel never writes `.env`, that `BOR_SETTINGS_DB=0` disables DB overrides, and that the values marked "restart required" apply on the next `podman compose restart app`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `uv run pytest tests/integration/test_admin_settings_api.py -v` (DB up), then `uv run pytest tests/integration -q` twice with a `app_settings` row left in on purpose ONCE inside the run only — the module's teardown must clean it (proven by a second clean run).
|
||||||
|
- Unit: the module's masking + `_display` helpers (bool/JSON/scalar stringification), the `.env`-never-written and no-shell static pins.
|
||||||
|
- Coverage: **>90%** on `app/api/admin_settings.py` (every 404/422 branch + both `DELETE`s + the masked branch).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/admin/settings` returns all 75 settings grouped, each with its effective value and a `db`/`env`/`default` badge; secrets report `masked`.
|
||||||
|
- [ ] `PUT` stores, validates first (nothing written on failure, 422 names the field) and makes the value live through `get_settings()` in the same process.
|
||||||
|
- [ ] `DELETE /{key}` restores the env value; `DELETE ""` clears everything; both log a value-free line (`grep " :: app.api.admin_settings " <logs>`).
|
||||||
|
- [ ] Anonymous and token-user callers get 403 on every route; the byte audit keeps this endpoint under the 256 KB budget and, pinned, under 64 KB.
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Task 05 — The Settings view: the shell's eighth view, `#view-settings`
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "I want an admin panel that can edit anything in .env … The change should take effect on clicking 'save' on that page."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An admin opens **Settings**, sees every setting grouped with the value in effect and where it came from, edits a few, presses **Save settings**, and sees the badges flip to `DB` — while the anonymous visitor gets the sign-in gate and a token user never sees the nav link. Same wiring as the phase-91 Theme view, same table discipline as the phase-156 catalogue tables.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html`
|
||||||
|
- nav: after `#nav-theme` (L62) add `<a href="/settings.html" class="nav-link" id="nav-settings" hidden>Settings</a>` (ship-hidden, revealed for admin only — the phase-34 one-bar contract; the same one line goes into `document.html` (~L74), `login.html` (~L67) and `shared.html` (~L68) after their Theme links).
|
||||||
|
- view: after `#view-theme` add `<section class="view" id="view-settings" hidden inert aria-label="Settings" tabindex="-1">` following the `#view-tokens` skeleton exactly:
|
||||||
|
- `<div class="container settings-shell">` (PLAN §7.1's 72rem centered column — no new width token),
|
||||||
|
- `.page-head` with `<h1>Settings</h1>` and a `.page-sub` that states the rule in the owner's terms: values saved here are stored in the database and win over the server's environment variables; clearing one hands control back to the environment,
|
||||||
|
- `<section class="sources-gate" id="settings-gate" aria-labelledby="settings-gate-title" hidden>` — the `#tokens-gate` shape, heading "Sign in to change settings", sub-copy naming what is admin-only, `<a class="sources-gate-link" href="/login.html?next=/settings.html">Sign in</a>`,
|
||||||
|
- `<div id="settings-content" hidden>` holding, in order:
|
||||||
|
- `<div class="kb-banner is-error" id="settings-disabled" role="alert" hidden>` — shown when `overlay_enabled` is `false` ("DB overrides are switched off on the server (`BOR_SETTINGS_DB=0`) — saving stores the value but it will not take effect."),
|
||||||
|
- `<span class="settings-status" id="settings-status" role="status" aria-live="polite"></span>` (the ONE status region, the `#tokens-status` precedent) and `<p class="settings-error" id="settings-error" role="alert" hidden></p>` (the `#theme-error` precedent),
|
||||||
|
- the search row: `<label for="settings-filter">Search settings</label>` + `<input id="settings-filter" type="search" autocomplete="off">` (client-side filter, no fetch — the phase-101 D4 contract),
|
||||||
|
- the scroll card: `<div class="table-wrap"><table class="settings-table"><caption class="visually-hidden">Application settings — the value in effect and where it comes from</caption><thead><tr><th scope="col">Setting</th><th scope="col">Value</th><th scope="col">Source</th><th scope="col"><span class="visually-hidden">Actions</span></th></tr></thead></table></div>` — the `tbody` elements are created by `settings.js` (one per group, each opening with a `<tr class="settings-group-row"><th colspan="4" scope="colgroup">Group</th></tr>`), and the `.table-wrap` + sticky-`th` rules at `styles.css:2825/2841` are already the bounded-scroll + sticky-header contract (AGENTS rule 5) — `.visually-hidden` inside the header needs the `.table-wrap` `position: relative` that phase 86 added,
|
||||||
|
- the action row: `<button type="button" class="btn primary" id="settings-save" disabled>Save settings</button>` + `<button type="button" class="btn ghost" id="settings-clear-all">Clear all overrides</button>` + `<span class="settings-dirty" id="settings-dirty" aria-hidden="true"></span>` (an "N unsaved changes" hint, `textContent` only).
|
||||||
|
- No `<script>` tag for the new module: the router lazy-imports it (rule 6 — everything local).
|
||||||
|
2. `frontend/assets/router.js` — three entries in the phase-91 style: `"/settings.html": "settings"` in `VIEW`, `settings: "/settings.html"` in `VIEW_PATH`, `settings: () => import("./settings.js")` in `VIEW_MODULES` (a STATIC specifier so the Containerfile's `esbuild --bundle` of `router.js` inlines it — `settings.js` needs no new esbuild line because no page references it directly; `tests/integration/test_containerfile_assets.py` proves it).
|
||||||
|
3. `app/main.py` — add `"/settings.html"` to the `_shell_routes(...)` tuple (after `"/theme.html"`) with the usual comment.
|
||||||
|
4. `app/core/caching.py` — add `"/settings.html"` to `HTML_PAGES` (L169 block) with the `# phase 168: the admin settings view (shell route)` comment, so it gets the `no-cache` + `?v=` contract (A19).
|
||||||
|
5. `frontend/assets/header.js` — after the Theme reveal block (~L225) add the null-safe `#nav-settings` block, admin-only, with the comment naming the contract it copies (a token user never sees it).
|
||||||
|
6. `frontend/assets/settings.js` (new) — `export async function mount(root)`, the `theme.js` shape (gate first via `fetchIsAdmin()`/`fetchWhoami()` from `header.js`; anonymous shows `#settings-gate` and fetches nothing; then unhide `#settings-content`, load, bind; listen for `bor:view-refresh` to re-load on a re-show):
|
||||||
|
- `GET /api/admin/settings` → render the groups in the server's order (`createElement` + `textContent` only, never HTML strings — PLAN §7 frontend rule). One row per setting:
|
||||||
|
- **Setting cell:** the `label` (`.cell-clip` ellipsis when it truncates, the phase-156 contract with the full text in the control's `aria-label`), the `BOR_*` name in a `<code class="settings-env">`, and — when `restart_required` — a `.settings-flag` with the literal text `restart required` (text, never color alone, B5) plus `title`/`aria-describedby` copy explaining when it applies;
|
||||||
|
- **Value cell:** the control chosen by `type` — `int`/`float` → `<input type="number">` (`step="any"` for float), `bool` → a two-option `<select>` (`true`/`false`), `json` → `<textarea spellcheck="false">`, `str` → `<input type="text">` (a `<textarea rows="2">` for the CSV/path fields whose value exceeds 40 chars), `secret` → `<input type="password" autocomplete="new-password" placeholder="•••••• (stored)">` with `aria-label` naming the env var. Every control gets an `id` (`set-<key>`) and a `<label for>` (visually the Setting cell's label, the house pattern for table-hosted inputs). For a masked field an empty input means "leave unchanged" (never an accidental wipe);
|
||||||
|
- **Source cell:** `<span class="settings-src is-db|is-env|is-default">DB</span>` / `env` / `default` — literal text;
|
||||||
|
- **Actions cell:** a `Reset` button, `disabled` when `!has_override` (nothing to reset) or `!overridable` (with `title` naming the reason).
|
||||||
|
- dirty tracking in module state (`Map<key, value>`): an `input`/`change` marks the row (`.is-dirty` + the `N unsaved changes` hint) and enables `#settings-save`; a row that returns to its served value un-marks.
|
||||||
|
- `#settings-save` → for each dirty row in DOM order `PUT /api/admin/settings/<key>` with `{ value }`; **stop at the first failure** (K4), show its server detail in `#settings-error` (role=alert), keep the remaining edits in the form, re-`GET` so the badges show the truth, keep focus on the offending input. All good → one `#settings-status` line: `Saved 3 settings.` (then `role=status`, not alert) + a re-`GET` so every badge/value is the settled server state. While in flight the button is `disabled` with `aria-busy="true"` (§7.4: every control state defined; a failure path always re-enables).
|
||||||
|
- `Reset` → `DELETE /api/admin/settings/<key>` → status line `BOR_X is back to the environment value.` + re-`GET`. `#settings-clear-all` → the house two-step inline confirm (the `history-confirm-*` classes, focus moves to the confirm button, Escape/`No` cancels) → `DELETE /api/admin/settings` → status line `Cleared N overrides.` + re-`GET`.
|
||||||
|
- `#settings-filter` → case-insensitive substring over the label + `BOR_*` text cells, applied on `input` with no fetch, re-applied after every re-`GET` (the query survives re-shows, phase-77); a group with no visible rows hides its group row; zero matches with a non-empty query → one no-match row `No settings match "<query>"` (`textContent`).
|
||||||
|
- A `restart_required` row that saves successfully adds "— applies on the next restart" to the status line (so the owner is never told "saved" and then sees nothing change).
|
||||||
|
7. `frontend/assets/styles.css` — a `---------- Settings view (phase 168) ----------` block: `.settings-shell` (no new container), `.settings-table { width: 100%; }` with `.git-sources-table`-style column proportions (never a skinny list — PLAN §7.1), `.settings-group-row th` (a `--surface` band, `--ink` text ≥4.5:1), `.settings-env` (mono, `--ink-soft`), `.settings-src` + the three `is-*` variants (solid `--brand-soft` / `--surface` / `--line` backgrounds with the existing ink tokens — no new palette variable, the phase-156 B3 rule; the badge is TEXT, the color is decoration), `.settings-flag` (`--accent-bg`/`--accent-ink`, the state-family pair), `.settings-row.is-dirty` (a `--brand` left border on the row + the `N unsaved changes` hint, so "unsaved" survives grayscale), `.settings-status` (the `.tokens-status` mirror), `.settings-error` (the `.theme-error` mirror), `input`/`select`/`textarea` sizing with `:focus-visible` inherited from the global rule, and ≥44px touch targets for the buttons/inputs at `≤640px` (§7.1 mobile). Nothing animates, so `prefers-reduced-motion` needs no new rule.
|
||||||
|
8. `tests/unit/test_settings_frontend_view.py` (new — the house static-source-assertion style, cf. `tests/unit/test_thinking_scroll.py` / `test_frontend_router.py`): `index.html` contains the `#view-settings` section, the `#settings-gate`, `#settings-content`, `#settings-status role="status"`, `#settings-error role="alert"`, `#settings-save`, `#settings-clear-all`, and `#nav-settings` with `hidden`; `router.js` maps `/settings.html` in all three tables; `main.py`'s shell tuple and `caching.HTML_PAGES` both list `/settings.html`; `header.js` reveals `#nav-settings` and keys it off `admin` (not `signedIn`); `settings.js` contains no `innerHTML =` with a template literal and no `http(s)://` reference (rule 6 + the no-HTML-strings rule); `styles.css` declares `.settings-table` and the three `.settings-src.is-*` variants and does NOT add a `:root` variable (B3); the four pages that carry the Theme link now carry the Settings link too.
|
||||||
|
9. `README.md` — a **Settings (`/settings.html`)** subsection under "Pages at a glance" (after "Access tokens", L205): what it is, the precedence rule, "restart required", and that it is admin-only.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new file, plus `uv run pytest tests/unit/test_frontend_router.py tests/unit/test_caching.py -v` (both enumerate the views/pages and must be *extended*, not broken) and `uv run pytest tests/integration/test_containerfile_assets.py -v` (the new asset is reachable through the bundle).
|
||||||
|
- Integration: `uv run pytest tests/integration/test_caching_revalidation.py -v` — `/settings.html` must serve `no-cache` with rewritten `?v=` asset refs.
|
||||||
|
- The E2E for this view is task 06's file; here the view must be reachable by hand (`uv run uvicorn app.main:app --reload` → sign in → Settings).
|
||||||
|
- Coverage: **>90%** on the `app/` diff of this task (the shell route + `HTML_PAGES`), untouched elsewhere.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `/settings.html` deep-links to the Settings view (the shell document, `router.js` picks the view), anonymous sees the gate and no data, a token user never sees the nav link.
|
||||||
|
- [ ] The table lists every group with a bounded scroll region and a sticky header at 72rem and at 375px; long labels clip with the ellipsis and stay reachable through the control's `aria-label`.
|
||||||
|
- [ ] Editing marks the row dirty, **Save settings** reports one status line and every badge/value re-reads from the server; a rejected value shows the server's detail in the `role="alert"` region and keeps the other edits.
|
||||||
|
- [ ] Keyboard-only operation works end to end (filter → edit → Save → Reset → two-step Clear-all), `:focus-visible` visible on every control, and the page introduces no CDN reference and no HTML-string DOM building.
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# Task 06 — `tests/e2e/test_admin_settings_panel.py`: the live-on-Save contract, proven in a browser
|
||||||
|
|
||||||
|
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…The change should take effect on clicking 'save' on that page."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's dedicated Playwright suite (AGENTS rule 4/9): a real admin signs in, changes a setting through the panel, presses **Save**, and the running app behaves differently — no restart, no `.env` edit — plus the gate, the masking, the rejection path, and the reset. Then the full gates and the one atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_admin_settings_panel.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_admin_theme_tab.py:205-270` verbatim (mock LLM, `BOR_ENVIRONMENT=e2e`, `BOR_STATIC_DIR`, the E2E admin password/secret, the docs-repo/suggestions/app-name leak guards) and change exactly these lines:
|
||||||
|
- `env["BOR_SETTINGS_DB"] = "1"` — the module opts back in over `tests/e2e/conftest.py`'s global `0` pin (task 02), the `test_pictorial_pdf.py:339` opt-in pattern;
|
||||||
|
- `env["BOR_INPUT_PLACEHOLDER"] = "Env placeholder for panel E2E"` and `env["BOR_FOOTER_TEXT"] = "Env footer for panel E2E"` — a distinctive env base, so "the DB row now wins" is a visible DOM change and not a no-op against the default;
|
||||||
|
- `env["BOR_RELEVANCE_THRESHOLD"] = "0.30"` (the E2E calibration, so a test that saves `0.31` proves the retrieval path read the new number, not a broken gate).
|
||||||
|
Add the module docstring in the house style: what the suite proves, the DB-isolation rule, and the Source line (`TODO.md:3`).
|
||||||
|
2. **DB isolation** (module autouse fixture, the `test_admin_theme_tab.py:286` `TRUNCATE ui_settings` idiom): `TRUNCATE app_settings` before AND after every test, plus `invalidate_settings()`-equivalent on the server side is unnecessary (the app runs in its own subprocess and re-reads on the save's own invalidation). Teardown must leave the shared dev E2E DB with an empty `app_settings` — a leftover row is the cross-suite poison this phase's D6 exists to prevent.
|
||||||
|
3. Tests (names pinned; helpers for `login(page, app_url)` from `tests/e2e/auth_helpers.py` and the `httpx` + cookies pattern those suites already use):
|
||||||
|
- `test_anonymous_sees_gate_and_no_data` — `/settings.html` serves the shell with `#view-settings` visible, `#settings-gate` visible, `#settings-content` hidden, `#nav-settings` hidden; `GET /api/admin/settings` unauthenticated is 403; no setting label or value appears in `page.content()`.
|
||||||
|
- `test_token_user_never_sees_the_panel` — a token user (issue one as admin via the API the way `test_api_tokens.py` does, then drive the token gate with `login_with_token`) sees `#nav-settings` hidden and gets 403 from the API.
|
||||||
|
- `test_admin_sees_groups_and_source_badges` — after `login()`, `#nav-settings` reveals, the view mounts, rows render, `table.settings-table` is inside `.table-wrap` (bounded scroll, sticky `thead` computed style), at least the `Retrieval & honesty` and `LLM (aipi)` group rows appear, `input_placeholder`'s row shows the served env value and an `env` badge, an untouched knob (e.g. `top_n_docs`) shows its code default with a `default` badge, `llm_api_key` renders a password input whose `value` is `""` and whose placeholder is the "stored" hint, and `database_url` / `embedding_dim` render read-only with no Reset and no input.
|
||||||
|
- `test_save_takes_effect_without_restart` — **the TODO's own acceptance test.** Filter to "placeholder", set the value to `DB placeholder — panel saved this`, press **Save settings**, expect: the `#settings-status` line (`Saved 1 setting`), the badge flips to `DB`, then `goto("/")` (a fresh page load, chat view) and expect the composer's `placeholder` attribute to be `DB placeholder — panel saved this` — the same browser, the same server process, no `.env` touched. Assert `footer_text` is still the env value (only one row was saved) and that the DB value survives a reload of `/settings.html`.
|
||||||
|
- `test_saved_value_reaches_a_real_turn` — set `BOR_RELEVANCE_THRESHOLD` from the panel to `0.31`, POST `/api/chat` (the `test_admin_theme_tab.py`/`test_honest_deflection.py` HTTP helper style) and assert the turn's honesty classification follows the new bar for a fixture question whose cosine sits between `0.30` and `0.31` (if the fixture corpus has no such question — verify with `uv run python -m scripts.eval_retrieval "<question>"` — pin the strongest available proxy: `query_log.top_score` is unchanged while the deflected/grounded decision flips when the saved bar moves above/below it, which proves the live read). Then **Reset** it and assert the pre-save classification returns.
|
||||||
|
- `test_invalid_value_is_refused_and_nothing_changes` — set `relevance_threshold` to `abc`, Save → the `#settings-error` (`role="alert"`) region is visible and names `relevance_threshold`, the badge is still `env`, the row keeps its typed value (K4), `GET /api/admin/settings` still reports `source=env`, and the composer placeholder (a second dirty row that was NOT saved because the batch stopped) is unchanged server-side.
|
||||||
|
- `test_cross_field_pair_is_refused` — save `relevance_threshold = 0.2`, then try `lexical_support_floor = 0.9` → the alert names the field and the stored pair never becomes effective (`GET /api/admin/settings` still shows the first saved value only).
|
||||||
|
- `test_secret_never_reaches_the_browser` — set `llm_api_key` to a sentinel (`sk-e2e-secret-…`) via the API (not through the DOM), reload `/settings.html`, and assert the sentinel is absent from `page.content()` while the row's password input shows the "stored" placeholder and `has_override` renders a live Reset button; then set it from the panel's password input, Save, and assert the sentinel still never appears in the served HTML or the response body.
|
||||||
|
- `test_reset_hands_control_back_to_the_env` — after a DB override on `footer_text`, press its Reset → status line, badge back to `env`, and the footer on `/` reads the env value again.
|
||||||
|
- `test_clear_all_overrides` — save three overrides, `#settings-clear-all` → the two-step confirm (Escape cancels and the overrides survive), then confirm → `Cleared 3 overrides`, every badge back to `env`/`default`.
|
||||||
|
- `test_disabled_switch_is_announced` (subprocess-level, its own module fixture or an `httpx`-only variant): a second app subprocess with `BOR_SETTINGS_DB=0` → the panel shows `#settings-disabled` (`role="alert"`) and a save still stores without changing the app. If a second app fixture is too heavy for the file, pin the `overlay_enabled=false` render through `page.route` interception of `GET /api/admin/settings` instead — state which was done in the commit body.
|
||||||
|
- `test_view_layout_and_accessibility` — no horizontal page overflow at 375px and at 1440px (§7.1: the table is ≥80% of the container, not a skinny list), keyboard tab order reaches filter → first input → Save → Reset (a `page.keyboard` walk asserting focus lands on the real controls), the `.settings-flag` text `restart required` present on `db_pool_size`'s row, and zero external requests during the whole test (`page.on("request")` — rule 6).
|
||||||
|
4. **Regression set, all in isolation** (`--no-cov`): `test_admin_theme_tab.py` (the Theme view's gate/save contract that this view copies), `test_api_tokens.py`, `test_configurable_brand.py` (`/api/config`'s strings — now DB-over-env-capable), `test_global_tuning.py`, `test_admin_auth.py`, `test_cache_busting.py`, `test_wide_desktop_column.py` (the 72rem container contract, new view included).
|
||||||
|
5. **Full gates** (AGENTS rule 9, all green before the commit):
|
||||||
|
```bash
|
||||||
|
uv run pytest
|
||||||
|
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
|
||||||
|
uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov
|
||||||
|
uv run ruff check . && uv run pyright
|
||||||
|
```
|
||||||
|
Plus the operational proof of AGENTS rule 10: with the app running, save a setting, then `grep ' :: app.api.admin_settings ' <app log>` shows the key-only line and `grep ' :: api ' <app log>` shows the `GET /api/admin/settings` byte line; `uv run python -m scripts.api_payload_audit` lists the endpoint under the 256 KB budget.
|
||||||
|
6. **Docs & close-out:** confirm `README.md` carries both additions from tasks 04/05 (the Settings page section + the Configuration-reference precedence note); move the phase directory:
|
||||||
|
then move the phase directory and commit (AGENTS rule 8, `--no-gpg-sign`):
|
||||||
|
```bash
|
||||||
|
git mv .agents/phases/todo/168_admin_settings_panel .agents/phases/complete/168_admin_settings_panel
|
||||||
|
```
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The suite passes **in isolation** (`-v --no-cov`) and the app it drives writes no `.env` change (`git status --porcelain .env` empty after the run — `.env` is gitignored, so assert by `stat` mtime instead if the file is untracked).
|
||||||
|
- Unit + integration already exist from tasks 02/03/04 — this file adds the browser contract only; if it needs a behavior the API lacks, fix the API in this task and say so in the commit body.
|
||||||
|
- Coverage: **>90%** on `app/` after everything (`--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `test_save_takes_effect_without_restart` is green: a value saved through the panel shows up in the running app in the same browser session with no restart.
|
||||||
|
- [ ] Gate, masking, invalid-value, cross-field, reset, clear-all and mobile/keyboard coverage all green in isolation.
|
||||||
|
- [ ] `app_settings` is empty after the suite (`SELECT count(*) FROM app_settings` → 0) and the rest of the E2E suite still passes with that pin in place.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the byte audit clean.
|
||||||
|
- [ ] Phase directory moved to `.agents/phases/complete/168_admin_settings_panel/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Phase 170 — Edit bubble: the editor lives inside the user bubble, "Send" re-asks
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "the edit bubble is tiny compared to the message bubble. When you click "edit" it collapses down to this super small text field. It should stay the same size as the user message bubble. Also the save button should read "send" rather than "save" because you're not saving it you're re-sending the message."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
|
||||||
|
**Context:** `frontend/assets/app.js` — `openEditor(wrap, toggleBtn)` (L1210–1357: label + `textarea[rows=3][maxlength=4000]` prefilled through `.value`, `editSaveBtn` "Save" L1238 + cancel, `closeEditor()` L1252 (form.remove + `.is-editing` remove + focus), the submit handler L1265–1345 (in-flight guard, empty guard, replace-in-place, tail truncation + DOM drop, `lastBrainWrap = null`, `bubble.innerHTML = renderMarkdown(text)` + the `.msg-image` re-prepend L1322–1325, `saveConversation()` + `persistConversation()`, `markLastRetryable()`, `closeEditor()` L1337, `runTurn(text, { reask: true })`), the open-tail L1355–1357 (`classList.add("is-editing")` + `body.appendChild(form)` + `ta.focus()`)), `appendEditButton` / `markLastEditable` (L793–871, the phase-159 single-target discipline), `tests/unit/test_frontend_edit_message.py` (house source pins: `editSaveBtn.textContent = "Save"` L348; the `.is-editing` add L365 / remove L387 / ordering L372; the `body.appendChild(form)` pin near L699), `tests/e2e/test_edit_and_resend.py` (the phase-159 suite: `page.click(".edit-form .edit-save")` L337 — class-based, so the label rename touches only its docstrings; the composer's `#send-label` "Send" pins L312/L533 are a different control), `frontend/assets/styles.css` — `.msg.user .bubble.is-editing { display: none; }` (L1411), `.edit-form` (L1412–1420 — its own `--surface` box + border + padding), `.edit-form label` (L1421–1425), `.edit-form textarea` (L1426–1436, `resize: vertical`, `min-height: 2.6rem`), `.edit-form-actions` (L1437), `.edit-save` (L1438–1455, incl. `:disabled { opacity: 0.6; cursor: wait; }` L1454), `.edit-cancel` (L1456–1469), `.edit-error` (L1470–1479), `.bubble` (L501–508), `.msg.user .bubble` (L592–596 — the `--brand` fill + `--bg` text).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Clicking **Edit** no longer swaps the user bubble for a shrunken form: the editor lives *inside* the bubble itself (inheriting its box, background, and padding, so it keeps the bubble's dimensions) and the submit button reads **Send** — the action re-sends the question, it does not save a note. Everything else of the phase-159 flow (prefill, empty guard, tail truncation, the `reask` re-ask, one-editor-at-a-time, image survival, the in-flight guards) stays behavior-identical.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The form IS the bubble's content (LOCKED A1).** `openEditor` stops appending the form as a sibling of `.bubble` in `.msg-body` and stops hiding the bubble. It keeps the `.msg-image` child in a closure variable, keeps `bubble.classList.add("is-editing")` (L1355 — the state class survives, the unit pins keep it), and replaces `body.appendChild(form)` (L1356) with the form appended **into** the bubble, the rendered markdown cleared first. The bubble keeps its own `--brand` fill, border, padding, and width — "the same size as the user message bubble" holds by construction (the box IS the box), with zero measurement/matching code. `.msg.user .bubble.is-editing { display: none; }` (styles.css L1411) is **deleted** — the bubble stays visible while the editor is open.
|
||||||
|
- **The form is transparent.** `.edit-form` drops its own `background` / `border` / `padding` (the bubble provides them) and keeps `display:flex; flex-direction:column; gap: 0.5rem`. The label and the action text take the bubble's `--bg`-family ink (contrast documented in a comment — the B5 text-state rule); the textarea keeps its own readable `--surface` fill + `--ink` text (an edit field on a `--brand` bubble) and gains `width: 100%`. `rows=3` and `maxlength=4000` are untouched — the *outer* box is the bubble's, the textarea is the inner field.
|
||||||
|
- **Cancel = restore the markdown.** `closeEditor()` keeps exactly three jobs: `form.remove()`, `bubble.classList.remove("is-editing")`, `toggleBtn.focus()` — no content change. The cancel branch (Escape + the Cancel button) gains a restore step: `bubble.innerHTML = renderMarkdown(conversation[editIdx].text)` + re-prepend the saved image (the mirror of the submit re-render L1322–1325), then `closeEditor()`.
|
||||||
|
- **Submit ordering.** The submit handler currently re-renders (`bubble.innerHTML = renderMarkdown(text)` L1322) *before* `closeEditor()` (L1337). With the form inside the bubble, `closeEditor()` (which detaches the form) must run **before** the innerHTML re-render; the rest of the handler — truncation, `lastBrainWrap = null` / `leavePartialIndex` reset, persistence, `markLastRetryable()`, the in-flight re-check, `runTurn(text, { reask: true })` — is byte-untouched.
|
||||||
|
- **Label (LOCKED A2).** `editSaveBtn.textContent = "Send"` (app.js L1238) — matching the composer's `#send-label` "Send" and the "Cancel" sibling. The phase-168 "Save settings" button is a different control and keeps its name.
|
||||||
|
- **NOT touched:** `appendEditButton` / `markLastEditable`, `runTurn`, `retryLastTurn`, `handleSend`, `rememberBrainTurn`, the record-truncation + persistence path, `MAX_QUESTION_CHARS` / `updateCharCount`, the composer (the unsent draft survives the edit), the SSE contract (A15), the shared-chat page, `app/` server code (A10 stateless holds).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `159_edit_and_resend_message` (complete) — the inline editor + re-ask flow this phase re-houses and re-labels.
|
||||||
|
- `166_hide_empty_answer_bubble` (complete) — `.msg .bubble:empty { display: none }` (styles.css L524): while the editor is open the bubble holds the form (never `:empty`), and on cancel the re-rendered markdown is never empty (`min_length=1` pinned client- and server-side) — no interaction, the pin stays green.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_editor_inside_bubble.md` — `openEditor` builds the form **inside** `.bubble` (transparent form, `display:none` deleted), the cancel re-render, the submit ordering; the unit pins updated.
|
||||||
|
2. `02_save_label_send.md` — the submit label "Save" → "Send" (app.js L1238), the unit pin, the phase-159 E2E docstring alignment.
|
||||||
|
3. `03_e2e_edit_bubble.md` — `tests/e2e/test_edit_bubble_sizing.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_frontend_edit_message.py` (house source pins, updated in tasks 01/02): the form is appended to the bubble (not `.msg-body`), the `display:none` rule's absence pinned in the CSS, `.is-editing` still added (L365) / removed (L387), the cancel restore (markdown re-render + image re-prepend) and the submit ordering (form detached before the re-render), `editSaveBtn.textContent = "Send"`, and `runTurn` / `retryLastTurn` / `handleSend` byte-unchanged.
|
||||||
|
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_edit_bubble_sizing.py`, run **in isolation** — the size/label contract plus the regression flows.
|
||||||
|
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Clicking Edit on the last user bubble shows the editor **inside** the bubble: the bubble is visible (computed `display` ≠ `none`), keeps its `--brand` fill, and its box width is unchanged by the edit (the E2E bounding-box pin).
|
||||||
|
- [ ] The submit button reads "Send"; Cancel / `Escape` restore the rendered question; trim + Send re-asks with the truncated shape and the "Question too long" driver case still resolves.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_edit_and_resend.py`, `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
|
||||||
|
- **A1 — the editor lives inside the bubble:** "stay the same size as the user message bubble" = the editor occupies the bubble element itself (inherits its box/styling), not a pixel-matched sibling form. `rows=3` + `maxlength=4000` stay.
|
||||||
|
- **A2 — the label is "Send"** (capital S, matching the composer's `#send-label` "Send" and "Cancel"); only the edit-form label changes — the phase-168 "Save settings" button keeps its name.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): edit the last question inside its bubble and re-send it with Send"
|
||||||
|
```
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Task 01 — the editor renders inside the user bubble (same box, no shrink)
|
||||||
|
|
||||||
|
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "the edit bubble is tiny compared to the message bubble. When you click "edit" it collapses down to this super small text field. It should stay the same size as the user message bubble."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`openEditor` renders its form **inside** `.bubble` instead of as a hidden-bubble + small-sibling-form: the editor inherits the bubble's own box, background, and padding, so it keeps the bubble's dimensions by construction. The open / cancel / submit behavior of the phase-159 flow is otherwise unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `openEditor` (L1210–1357), three surgical changes:
|
||||||
|
- **Open:** keep `if (bubble) bubble.classList.add("is-editing");` (L1355); replace `body.appendChild(form);` (L1356) with: grab the `.msg-image` child into a closure variable, clear the bubble's rendered markdown (`bubble.innerHTML = "";` or `bubble.replaceChildren(form)`), append the form **into** the bubble; `ta.focus()` (L1357) stays.
|
||||||
|
- **Cancel:** `closeEditor()` (L1252–1257) keeps exactly `form.remove()` + `bubble.classList.remove("is-editing")` + `toggleBtn.focus()` — no content change. The cancel paths (the `Escape` keydown branch L1258–1261 and the Cancel-button listener) gain a restore step BEFORE `closeEditor()`: `bubble.innerHTML = renderMarkdown(conversation[editIdx].text)` + re-prepend the saved image (the exact mirror of the submit re-render L1322–1325).
|
||||||
|
- **Submit:** reorder — move `closeEditor()` (L1337) to **before** the `if (bubble) { … bubble.innerHTML = renderMarkdown(text); … }` re-render block (L1322–1325), so the form is detached cleanly first. Everything else in the handler — the in-flight guard, the empty guard, `conversation[editIdx].text = text`, the tail truncation + DOM drop, `lastBrainWrap = null` / `leavePartialIndex` reset, `saveConversation()` + `persistConversation()`, `markLastRetryable()`, the second in-flight re-check, `runTurn(text, { reask: true })` — is byte-untouched.
|
||||||
|
- Update the `openEditor` comment block (L1154–1195): the "While open the rendered .bubble is hidden (.is-editing — the question appears ONCE)" sentence becomes the new contract (the form renders inside the bubble; the bubble stays visible; `.is-editing` is the state class); keep the rest of the block.
|
||||||
|
2. `frontend/assets/styles.css`:
|
||||||
|
- **Delete** `.msg.user .bubble.is-editing { display: none; }` (L1411) — the bubble stays visible while the editor is open.
|
||||||
|
- `.edit-form` (L1412–1420): drop the own `background: var(--surface);`, `border: 1px solid var(--brand-soft);`, and `padding: 0.75rem 0.85rem;` (the bubble provides them); keep `display: flex; flex-direction: column; gap: 0.5rem;`.
|
||||||
|
- `.edit-form label` (L1421–1425): the color becomes the bubble's `--bg`-family ink — on the `--brand` fill, `--bg` is the documented 1.0-against-brand pairing inverted (the user-bubble text pairing, L592–596); document the ratio in the comment (B5: text states it, never color alone); keep `font-size: 0.85rem; font-weight: 600;`.
|
||||||
|
- `.edit-form textarea` (L1426–1436): add `width: 100%;` (the field fills the bubble's width — the "not tiny" fix); keep `font-size: 0.9rem`, the `--surface` fill + `--ink` text (a readable edit field on the `--brand` bubble), `border`, `resize: vertical`, `min-height: 2.6rem`.
|
||||||
|
- `.edit-error` (L1470–1479): the `--err-bg` / `--err-ink` pair is documented against `--surface`; on the `--brand` bubble it needs a check — if the pair fails ≥4.5:1 there, scope a `.msg.user .bubble .edit-error` variant on the bubble's own ink pairing (`--bg` text on a `--brand`-darkened fill) and document the ratio; the `role="alert"` behavior is untouched.
|
||||||
|
- The `:disabled` wait-cursor on `.edit-save` (L1454) is the phase-159 in-flight dim — leave it (task 03 of THIS phase does not touch it; the owner's cursor complaint is the settings page, phase 171).
|
||||||
|
3. `tests/unit/test_frontend_edit_message.py` — update the pins (house source-pin style):
|
||||||
|
- the form is appended to the bubble: replace the `body.appendChild(form)` pin (near L699) with the bubble-append pin (e.g. `bubble.replaceChildren(form)` or the innerHTML-clear + append, whichever lands in app.js);
|
||||||
|
- the `.is-editing` add (L365) / remove (L387) / ordering (L372) pins **stay** (the class is still the state marker);
|
||||||
|
- new pins: the cancel restore (`renderMarkdown(conversation[editIdx].text)` on cancel + the image re-prepend) and the submit ordering (`closeEditor()` before the `bubble.innerHTML = renderMarkdown(text)` re-render);
|
||||||
|
- a CSS pin: `.msg.user .bubble.is-editing { display: none; }` is **absent** from `styles.css`;
|
||||||
|
- `runTurn` / `retryLastTurn` / `handleSend` pins byte-unchanged.
|
||||||
|
|
||||||
|
- ASSUMPTION: A1 (phase-locked) — "stay the same size as the user message bubble" is implemented by the editor occupying the bubble element itself (it inherits the bubble's box/styling), not by pixel-matching a sibling form; `rows=3` + `maxlength=4000` stay.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_edit_message.py` green (`uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright` (the JS is not linted by ruff — the unit pins are the JS gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] In a live browser (or the E2E from task 03): clicking Edit shows the form inside the visible bubble — no `display:none`, no shrunken sibling box.
|
||||||
|
- [ ] Cancel / `Escape` restore the rendered markdown + the attachment; the record is unchanged.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` green; `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] No behavior change in `runTurn` / `retryLastTurn` / `handleSend` (byte-unchanged pins green).
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 02 — the edit-form submit label reads "Send", not "Save"
|
||||||
|
|
||||||
|
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "Also the save button should read "send" rather than "save" because you're not saving it you're re-sending the message."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The edit form's submit button reads **Send** — the action re-sends the question (it never persists a standalone note, A10 stateless holds), so "Save" was a lie. The label matches the composer's `#send-label` "Send" and the "Cancel" sibling; nothing else in the flow changes.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` L1238 — `editSaveBtn.textContent = "Save";` → `editSaveBtn.textContent = "Send";`. The `edit-save` class, the `type="submit"` wiring, the Ctrl/Cmd+Enter submit, and the whole submit handler are untouched.
|
||||||
|
2. `tests/unit/test_frontend_edit_message.py` L348 — the pin `assert 'editSaveBtn.textContent = "Save";' in body` → `"Send"`; keep the surrounding pin block (the `editSaveBtn.type` / `editSaveBtn.className` pins) and align the module docstring's "Save/Cancel" phrasing (L15) to "Send/Cancel" where it names the **edit form** (the docstring lines that name the composer's send flow stay — that control was always "Send").
|
||||||
|
3. `tests/e2e/test_edit_and_resend.py` — the flow drives the edit submit by class (`page.click(".edit-form .edit-save")` L337) — **no code change**. Align the module docstring / comments that name the edit form's "Save" action (L29, L32, L51, L79, L255, L341, L383, L510) to "Send"; the composer's `#send-label` "Send" assertions (L312, L533) are a different control and stay byte-identical.
|
||||||
|
|
||||||
|
- ASSUMPTION: A2 (phase-locked) — the label is "Send" (capital S, matching the composer's `#send-label` and "Cancel"); only the edit-form label changes — the phase-168 "Save settings" button (a different control) keeps its name.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_edit_message.py` green.
|
||||||
|
- E2E (regression, in isolation): `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` — the phase-159 suite now runs the flow through the "Send" label (the click is class-based, so the suite is code-identical; the docstrings are aligned).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The edit form's submit button reads "Send" in the live UI; "Save" appears nowhere in the edit form.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` green; `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] The phase-168 "Save settings" button (settings view) is byte-untouched.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Task 03 — `tests/e2e/test_edit_bubble_sizing.py`: the in-bubble editor, proven in a browser
|
||||||
|
|
||||||
|
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "the edit bubble is tiny compared to the message bubble … It should stay the same size as the user message bubble. Also the save button should read "send" rather than "save" …"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's dedicated Playwright suite (AGENTS rule 4/9): the editor opens **inside** the user bubble with the bubble's dimensions, the submit label reads "Send", and the whole phase-159 flow (trim + re-ask, cancel restore, the "Question too long" driver case) works through the new housing. Then the regression set, the full gates, and the one atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_edit_bubble_sizing.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_edit_and_resend.py` verbatim (mock LLM, `BOR_ENVIRONMENT=e2e`, the E2E admin/secret pins) and change exactly: the module docstring (what the suite proves + the Source line `TODO.md:3` — the "in-bubble editor + Send label" contract) and the test list below. DB isolation: this suite writes no `app_settings` / theme rows; the `saved_chats` rows it creates follow the phase-159 suite's convention (no extra teardown needed beyond what that fixture already does — if the fixture lacks a `saved_chats` cleanup, add the same one it would need, mirroring `test_chat_persistence.py`).
|
||||||
|
2. Tests (names pinned; helpers `login(page, app_url)` from `tests/e2e/auth_helpers.py` where a gate is crossed — the chat is public, so most tests need no login, exactly like the phase-159 suite):
|
||||||
|
- `test_editor_renders_inside_the_bubble` — send a fixture question, wait for the grounded answer (the mock-LLM settle convention), capture the user bubble's `bounding_box()` + computed `background-color`, click Edit: the form is a **child of the bubble** (`bubble.evaluate("el => el.querySelector('.edit-form') !== null")`), the bubble's computed `display` is not `none`, its `background-color` is unchanged (the `--brand` fill), the textarea prefills the EXACT question, and `.is-editing` is on the bubble.
|
||||||
|
- `test_editor_box_keeps_the_bubble_dimensions` — the pre-edit bubble `bounding_box()` vs the open-editor bubble: width is ≥ 95 % of the pre-edit width (the "no longer tiny" pin — the old shrunken sibling form failed this by construction) and height is ≥ the pre-edit height (the box grows, never collapses); at 375 px the bubble has no horizontal overflow (§7.1 reading column).
|
||||||
|
- `test_submit_label_reads_send` — `.edit-save`'s text is "Send" (and not "Save"); `.edit-cancel`'s is "Cancel".
|
||||||
|
- `test_trim_and_send_reasks_in_place` — the phase-159 flow through the new label: append " in one line", click `.edit-form .edit-save` (the "Send" button): the truncated shape `[user(edited), brain]` (the `bor.chat.v1` localStorage pin, the phase-159 convention), the re-ask streams the grounded answer, no duplicated question, the UNSSENT composer draft survives.
|
||||||
|
- `test_cancel_restores_the_bubble` — open the editor, Cancel: the rendered markdown is back (the question text is visible in the bubble, the form is gone, `.is-editing` off), the `conversation` record is byte-identical; a second pass with `Escape` (a fresh turn) pins the same restore through the keydown path.
|
||||||
|
- `test_failed_turn_still_editable` — the phase-114 driver case (the 4000+ char question → the "Question too long — trim it and re-ask" error frame, the phase-120 failed record): the editor opens inside the bubble, trimming the question to the cap + "Send" streams the grounded answer (the failed record is replaced).
|
||||||
|
3. **Regression set, all in isolation** (`--no-cov`): `test_edit_and_resend.py` (the phase-159 suite — now running against the in-bubble editor + the "Send" label), `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
|
||||||
|
4. **Full gates** (AGENTS rule 9, all green before the commit):
|
||||||
|
```bash
|
||||||
|
uv run pytest
|
||||||
|
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
|
||||||
|
uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov
|
||||||
|
uv run ruff check . && uv run pyright
|
||||||
|
```
|
||||||
|
5. **Docs & close-out:** confirm `README.md`'s edit-the-last-question note (if any names the "Save" label or the hidden-bubble behavior — phase 159's README section, grep `edit` there) is aligned to the in-bubble editor + "Send"; move the phase directory:
|
||||||
|
```bash
|
||||||
|
git mv .agents/phases/todo/170_edit_bubble_sizing .agents/phases/complete/170_edit_bubble_sizing
|
||||||
|
```
|
||||||
|
and commit (AGENTS rule 8, `--no-gpg-sign`):
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): edit the last question inside its bubble and re-send it with Send"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The suite passes **in isolation** (`-v --no-cov`).
|
||||||
|
- Unit pins already exist from tasks 01/02 — this file adds the browser contract only; if it needs a behavior the UI lacks, fix the UI in this task and say so in the commit body.
|
||||||
|
- Coverage: **>90%** on `app/` after everything (the `--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `test_editor_box_keeps_the_bubble_dimensions` is green: the open editor's bubble keeps the pre-edit width (≥ 95 %) and never collapses.
|
||||||
|
- [ ] The "Send" label, the in-bubble housing, the cancel restore, and the failed-turn driver case are all green in isolation.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] Phase directory moved to `.agents/phases/complete/170_edit_bubble_sizing/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Phase 171 — Admin settings page: row rhythm, distinct group bands, a properly disabled Save
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 — "In the admin settings page: The search settings text box overlaps settings table. The settings category headers are not differentiated from the settings themselves, making them confusing at a glance. The save settings and clear all overrides are smushed right up against the settings table. The save settings shows a loading cursor when moused over rather than just being disabled."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
|
||||||
|
**Context:** `frontend/index.html` — `#view-settings` (L1499), `#settings-content` (L1535, a plain block — its children `#settings-disabled`, `#settings-status`, `#settings-error`, `.settings-search` (L1562), the `.table-wrap` card (L1576, `#settings-tbody` created by settings.js), and `.settings-actions` (L1607: `#settings-save` disabled by default + `#settings-clear-all` + `#settings-dirty`) stack with **zero vertical rhythm** — the root of the "overlap" and the "smushed" complaints), `frontend/assets/settings.js` (637 lines — the group-row render L319–335: `tr.settings-group-row` + `th[colspan=4][scope=colgroup]` with the group name, the client-side search filter, the §7.4 Save lifecycle), `frontend/assets/styles.css` — `.settings-shell` (L4660–4665: `display:flex; flex-direction:column; gap: 1.25rem;`), `.settings-group-row th` (L4684–4690: `background: var(--surface)` — the **same** fill as the `.table-wrap` card (L2825–2831) → the band is effectively invisible, the "not differentiated" complaint), the sticky-th tint rule `.table-wrap table thead th { background: var(--brand-soft); }` (L2841–2848, the solid-tint precedent), `.settings-search` (L4780–4786), `.settings-actions` (L4804–4809), `#settings-save` (L4810–4824, incl. `:disabled { opacity: 0.6; cursor: wait; }` L4824 — the loading-cursor complaint), `.settings-reset:disabled { opacity: 0.5; cursor: default; }` (L4775 — the house idiom), the mobile media queries (L6396–6400: `.settings-search` / `.settings-actions` `flex-direction: column`, `#settings-save` `width: 100%`), `tests/unit/test_settings_frontend_view.py` (the phase-168 static pins, incl. the L561 media-query pin), `tests/e2e/test_admin_settings_panel.py` (the phase-168 suite — the regression set).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The four layout complaints on the Settings page — the search box overlapping the table card, the group headers indistinguishable from the rows, the Save / Clear row smushed against the table, and the disabled Save showing a loading cursor — are fixed with **CSS only** (plus the pin updates): one rhythm rule, one band rule, one cursor value. The phase-168 contract (the gate, the catalogue, the §7.4 Save lifecycle, the API, the bounded scroll + sticky header) stays behavior-identical — no `app/` code, no JS logic change.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the rhythm (task 01).** `#settings-content` becomes `display: flex; flex-direction: column; gap: 1.25rem;` — the **same** token `.settings-shell` already uses (L4664). The children (the disabled banner, the status line, the error line, the search row, the `.table-wrap` card, the `.settings-actions` row) then stack with one uniform gap: the search box clears the table card (the "overlap" dies) and the Save / Clear row is separated from the table (the "smushed" dies) in the same change. The `min-height` rules on `#settings-status` / `#settings-error` (L4849–4862) already reserve their lines — the gap is additive and uniform, no per-element margins are introduced.
|
||||||
|
- **D2 — the band (task 02).** `.settings-group-row th` gets a band that is distinct from the row cells at a glance: `background: var(--brand-soft)` — the **same solid tint** the sticky `thead` rule already uses (L2847; B3: no new palette variable), `font-weight: 700` (the 0.82rem / uppercase / letter-spacing stay), and `border-top: 1px solid var(--line)` so a band never merges with the card's top edge or a previous band. The row cells stay on the card's `--surface`. The group text stays `--ink` on `--brand-soft` — the pairing the sticky-th rule already documents. Static, text-only, no animation (nothing for `prefers-reduced-motion` to still).
|
||||||
|
- **D3 — the cursor (task 03).** `#settings-save:disabled` → `cursor: default` — the `.settings-reset:disabled` idiom (L4775); "just disabled", per the owner. The §7.4 in-flight state is the **dimmed** `:disabled` (`opacity: 0.6` stays) + `aria-busy` — the `cursor: wait` was never part of the contract; it was the "loading cursor on hover" the owner saw.
|
||||||
|
- **NOT touched:** `settings.js` (the render / filter / save / reset / clear-all logic), `app/api/admin_settings.py`, the sign-in gate, the nav reveal, the `.table-wrap` bounded-scroll + sticky-th contract (L2825–2848), the `#settings-status` / `#settings-error` regions, the `restart required` flag, the mobile media queries (L6396–6400 — they compose with the new gap and stay byte-identical), `app/` server code.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `168_admin_settings_panel` (in flight — the code + the E2E suite are committed; only its directory move + final commit are pending) — the Settings view this phase polishes. Note: the abandoned `169` settings status card never landed — there is nothing to coordinate with.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_content_vertical_rhythm.md` — `#settings-content` becomes a flex column with the shell's `1.25rem` gap (fixes the search/table overlap **and** the smushed action row); the pin added.
|
||||||
|
2. `02_group_band_distinction.md` — `.settings-group-row th` gets the `--brand-soft` band + 700 weight + top hairline, so groups read at a glance.
|
||||||
|
3. `03_disabled_save_cursor.md` — `#settings-save:disabled` → `cursor: default` (the house idiom), killing the loading cursor.
|
||||||
|
4. `04_e2e_settings_layout.md` — `tests/e2e/test_admin_settings_layout.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_settings_frontend_view.py` (the phase-168 static pin set, extended in tasks 01/02/03): the `#settings-content` flex-column + gap rule, the band's `--brand-soft` background + weight + top border, `#settings-save:disabled` `cursor: default` (and `cursor: wait` absent from the settings rules), plus the existing pins (the no-CDN pin, the `router.js` mapping, the media-query pin L561) staying green.
|
||||||
|
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_admin_settings_layout.py`, run **in isolation** — the four layout contracts, proven in a browser.
|
||||||
|
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The search box and the table card do not intersect; the Save / Clear row is visibly separated from the table (the E2E bounding-box pins, at 1440 px and 375 px).
|
||||||
|
- [ ] A settings group band is visually distinct from its rows (computed background ≠ the row cells' background; weight 700) and no band merges with the card edge.
|
||||||
|
- [ ] A disabled **Save settings** shows `cursor: default` (not `wait`); the §7.4 dimmed-`aria-busy` in-flight state is intact.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` (phase 168) + `test_wide_desktop_column.py` green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
|
||||||
|
- **A3 — the band tint is `--brand-soft`** (the solid tint the sticky `thead` already uses — B3: no new palette variable) + a `--line` top hairline + weight 700; static, no animation.
|
||||||
|
- **A4 — the rhythm reuses the shell's existing `gap: 1.25rem`** — no new spacing value; the existing mobile media queries (L6396–6400) keep working unchanged.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): settings page layout — row rhythm, distinct group bands, a properly disabled Save"
|
||||||
|
```
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 01 — `#settings-content` gets the shell's vertical rhythm (overlap + smushing die)
|
||||||
|
|
||||||
|
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The search settings text box overlaps settings table. … The save settings and clear all overrides are smushed right up against the settings table."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The `#settings-content` block becomes a flex column with the shell's existing `gap: 1.25rem`, so its children — the disabled banner, the status line, the error line, the search row, the `.table-wrap` card, the `.settings-actions` row — stack with one uniform gap. One change kills both complaints: the search box clears the table card (the "overlap") and the Save / Clear row separates from the table (the "smushed").
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — add ONE rule (place it directly after the `.settings-shell` rule, L4660–4665, with a comment citing the phase-171 task + TODO L4):
|
||||||
|
```css
|
||||||
|
#settings-content {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 1.25rem;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
No per-element margins anywhere; the `min-height` rules on `#settings-status` / `#settings-error` (L4849–4862) stay (they reserve the lines; the gap is additive and uniform). Verify by reading, not by adding: `.settings-search` (L4780–4786) and `.settings-actions` (L4804–4809) get **no** new margins — the gap is the only spacing source.
|
||||||
|
2. `tests/unit/test_settings_frontend_view.py` — add the house-style CSS pin: the `#settings-content` rule exists with `display: flex`, `flex-direction: column`, and `gap: 1.25rem` (the same pin family as the L561 media-query pin — parse `styles.css`, assert the three declarations in one rule block).
|
||||||
|
|
||||||
|
- ASSUMPTION: A4 (phase-locked) — the rhythm reuses the shell's existing `gap: 1.25rem` token; no new spacing value; the existing mobile media queries (styles.css L6396–6400: `.settings-search` / `.settings-actions` `flex-direction: column; align-items: stretch;` + `#settings-save { width: 100%; }`) keep working unchanged.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_frontend_view.py` green (`uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] In the live UI (or the task-04 E2E): the search input's box and the `.table-wrap` card's box do not touch — a visible, uniform gap between them, and the same gap above the `.settings-actions` row.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] No per-element margin was added (the diff touches exactly one CSS rule + the pin); the mobile stacking (L6396–6400) is byte-unchanged.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — the group bands are distinct from the rows at a glance
|
||||||
|
|
||||||
|
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The settings category headers are not differentiated from the settings themselves, making them confusing at a glance."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`.settings-group-row th` — today `background: var(--surface)`, the **same** fill as the `.table-wrap` card, i.e. invisible — becomes a band that reads as a category: the `--brand-soft` tint (the solid tint the sticky `thead` rule already uses), weight 700, and a `--line` top hairline so a band never merges with the card edge or a previous band.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — `.settings-group-row th` (L4684–4690), three value changes in the existing rule (no new selector):
|
||||||
|
- `background: var(--surface);` → `background: var(--brand-soft);` (B3: no new palette variable — the exact tint `.table-wrap table thead th` already documents at L2841–2848);
|
||||||
|
- add `font-weight: 700;` (the `font-size: 0.82rem; text-transform: uppercase; letter-spacing: 0.04em;` stay);
|
||||||
|
- add `border-top: 1px solid var(--line);` (the rule's existing `border-bottom` from the `.settings-table th` family, L4676–4681, stays — a band is delimited above AND below, and can never merge with the card's top edge or the band before it);
|
||||||
|
- update the contrast comment (L4683): the text stays `--ink` on `--brand-soft` — the pairing the sticky-th rule already documents (the existing "15.3:1, AA" comment applies to `--surface`; recompute / restate the ratio on `--brand-soft` in the comment — the `:root` `--brand-soft` token carries its own documented pairing).
|
||||||
|
2. `tests/unit/test_settings_frontend_view.py` — add the pin: the `.settings-group-row th` rule carries `background: var(--brand-soft)`, `font-weight: 700`, and `border-top: 1px solid var(--line)` (same pin family as task 01's).
|
||||||
|
|
||||||
|
- ASSUMPTION: A3 (phase-locked) — the band = `--brand-soft` (the sticky-th tint, no new palette variable) + `--line` top hairline + weight 700; static, text-only, no animation (nothing for `prefers-reduced-motion` to still).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_frontend_view.py` green.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] In the live UI (or the task-04 E2E): each group band is visibly tinted (`--brand-soft`), bold, and hairline-delimited above; the row cells stay on the card's `--surface` — the two are distinguishable at a glance.
|
||||||
|
- [ ] No band merges with the card's top edge or the band above it (the top hairline is painted).
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
|
||||||
|
- [ ] `settings.js` is byte-unchanged (the band is pure CSS — the group-row render L319–335 is untouched).
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 03 — a disabled Save settings is just disabled (no loading cursor)
|
||||||
|
|
||||||
|
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The save settings shows a loading cursor when moused over rather than just being disabled."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`#settings-save:disabled` stops painting `cursor: wait` (the loading cursor the owner saw on hover) and takes `cursor: default` — the house idiom `.settings-reset:disabled` already uses. "Just disabled": the §7.4 in-flight state stays the **dimmed** `:disabled` (`opacity: 0.6`) + `aria-busy`; the cursor was never part of the contract.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` L4824 — `#settings-save:disabled { opacity: 0.6; cursor: wait; }` → `#settings-save:disabled { opacity: 0.6; cursor: default; }`. Keep `opacity: 0.6` (the §7.4 dim) and the `#settings-save:hover:not(:disabled)` rule (L4823) byte-unchanged. Update the adjacent comment block (L4809–4810, "the §7.4 in-flight state is the dimmed :disabled + aria-busy") to state the cursor contract explicitly: `cursor: default` when disabled (the `.settings-reset:disabled` L4775 idiom) — the wait cursor was never part of §7.4.
|
||||||
|
2. `tests/unit/test_settings_frontend_view.py` — add the pin: `#settings-save:disabled` carries `cursor: default`, and `cursor: wait` is **absent** from the settings-page rules (a negative grep over the `.settings*` / `#settings-*` rule blocks — the house negative-pin idiom). Note: `.edit-save:disabled` (styles.css L1454) keeps its own `cursor: wait` — it is the chat edit form's in-flight state (phase 159/170), a different control; the negative pin is scoped to the settings rules only, and the test must not fail on the `.edit-save` rule.
|
||||||
|
|
||||||
|
- ASSUMPTION: none — the owner's words fix the target ("just being disabled"); the idiom choice (`default`, mirroring L4775) is the house convention, not a guess.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_frontend_view.py` green.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] In the live UI (or the task-04 E2E): an untouched **Save settings** (no dirty rows) is `disabled` and its computed `cursor` is `default` — hovering shows no wait/loading cursor; after a row is edited the button enables and its cursor is `pointer` again (the §7.4 lifecycle, phase 168, intact).
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
|
||||||
|
- [ ] The diff is exactly one CSS value + one comment + the pin; `settings.js` and `app/` are byte-unchanged.
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 04 — `tests/e2e/test_admin_settings_layout.py`: the four layout contracts, proven in a browser
|
||||||
|
|
||||||
|
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "In the admin settings page: The search settings text box overlaps settings table. The settings category headers are not differentiated … The save settings and clear all overrides are smushed right up against the settings table. The save settings shows a loading cursor when moused over rather than just being disabled."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's dedicated Playwright suite (AGENTS rule 4/9): the admin signs in, and the four layout contracts hold in a real browser — the search row clears the table card, the action row is gapped from the table, the group bands are distinct, and a disabled Save is just disabled. Then the regression set, the full gates, and the one atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_admin_settings_layout.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_admin_settings_panel.py` verbatim (the mock LLM, `BOR_ENVIRONMENT=e2e`, `BOR_STATIC_DIR`, the E2E admin password/secret, the leak guards, the module's `BOR_SETTINGS_DB=1` opt-in + the `TRUNCATE app_settings` before/after — this suite never writes overrides, so the truncation is belt-and-braces, the same idiom) and change exactly: the module docstring (what the suite proves + the Source line `TODO.md:4` — the four layout contracts) and the test list below.
|
||||||
|
2. Tests (names pinned; `login(page, app_url)` from `tests/e2e/auth_helpers.py`; the suite drives the admin gate exactly like the phase-168 suite):
|
||||||
|
- `test_search_row_is_clear_of_the_table` — at 1440 px and 375 px: `#settings-filter`'s `bounding_box()` and the `.table-wrap` card's box do not intersect (the vertical gap between the search row's bottom and the card's top is > 0), and the card keeps its bounded scroll + sticky `thead` (the computed `position: sticky` on the `thead th` — the phase-156/168 contract intact under the new rhythm).
|
||||||
|
- `test_actions_row_is_gapped_from_the_table` — at 1440 px and 375 px: `.settings-actions`'s box does not intersect the `.table-wrap` card's box (the "smushed" pin — the gap between the card's bottom and the action row's top is > 0); `#settings-save` and `#settings-clear-all` are both reachable (no clipping at either viewport).
|
||||||
|
- `test_group_bands_are_distinct_from_the_rows` — after `login()`: at least the `Retrieval & honesty` and `LLM (aipi)` `tr.settings-group-row` bands render; a band's computed `background-color` ≠ the `.table-wrap` card's computed background (the `--surface` row-cell fill) — the "not differentiated" pin — and the band's computed `font-weight` is ≥ 700; a band's computed `border-top-width` is ≥ 1 px (the hairline — no merge with the card edge or the previous band).
|
||||||
|
- `test_disabled_save_is_just_disabled` — before any edit: `#settings-save` is `disabled` and its computed `cursor` is `default` (NOT `wait` — the loading-cursor pin); edit one row (a `type=search` filter + a value change, the phase-168 dirty-row flow): the button enables (computed `cursor` `pointer`) and dims back to `default` after a **Save settings** settles (the §7.4 lifecycle, phase 168, intact under the new cursor).
|
||||||
|
- `test_settings_page_layout_and_accessibility` — no horizontal page overflow at 375 px and 1440 px (§7.1: the table is ≥80 % of the container, not a skinny list); keyboard tab order still reaches filter → first row input → **Save settings** → **Clear all overrides** (a `page.keyboard` walk asserting focus lands on the real controls — the new rhythm must not have broken the phase-168 keyboard contract); zero external requests during the whole test (`page.on("request")` — rule 6).
|
||||||
|
3. **Regression set, all in isolation** (`--no-cov`): `test_admin_settings_panel.py` (the phase-168 suite — the gate, the save-live contract, the masking, the reset / clear-all, now under the new layout), `test_wide_desktop_column.py` (the 72rem container contract, the Settings view included).
|
||||||
|
4. **Full gates** (AGENTS rule 9, all green before the commit):
|
||||||
|
```bash
|
||||||
|
uv run pytest
|
||||||
|
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
|
||||||
|
uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov
|
||||||
|
uv run ruff check . && uv run pyright
|
||||||
|
```
|
||||||
|
5. **Docs & close-out:** confirm `README.md`'s Settings page section (phase 168's addition) needs no layout wording change (it describes the contract, not the pixels — grep `Settings` there); move the phase directory:
|
||||||
|
```bash
|
||||||
|
git mv .agents/phases/todo/171_admin_settings_layout .agents/phases/complete/171_admin_settings_layout
|
||||||
|
```
|
||||||
|
and commit (AGENTS rule 8, `--no-gpg-sign`):
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ README.md .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): settings page layout — row rhythm, distinct group bands, a properly disabled Save"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The suite passes **in isolation** (`-v --no-cov`) and leaves `app_settings` empty (the `TRUNCATE` idiom — a leftover row is the cross-suite poison phase-168 D6 exists to prevent).
|
||||||
|
- The pins already exist from tasks 01/02/03 — this file adds the browser contract only; if it needs a behavior the CSS lacks, fix the CSS in this task and say so in the commit body.
|
||||||
|
- Coverage: **>90%** on `app/` after everything (the `--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All four layout pins are green in isolation: search clear of the table, action row gapped, bands distinct (tint + weight + hairline), disabled Save `cursor: default`.
|
||||||
|
- [ ] `app_settings` is empty after the suite and the phase-168 regression suite is green in isolation.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] Phase directory moved to `.agents/phases/complete/171_admin_settings_layout/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Phase 172 — Thinking level: a pi-style, operator-tunable LLM reasoning knob
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Brain of Reese needs to support thinking levels like pi does. should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
|
||||||
|
**Context:** `app/config.py` — the LLM section (L97–125: `llm_base_url`, `llm_api_key`, `llm_chat_model` L101, `llm_embed_model`, `llm_summary_model` L107, `stream_thinking` L112, `llm_retries` / `llm_retry_delay` / `llm_timeout` L122 — the house pattern this setting joins), `.env.example` L57–65 (the LLM block — `BOR_LLM_CHAT_MODEL=turbo` L60), `app/core/settings_catalog.py` L137–161 (the `LLM (aipi)` catalogue group; the completeness pin in `tests/unit/test_settings_catalog.py` — `set(CATALOG) == set(Settings.model_fields)` — makes the new field a build failure until it has a `SettingMeta`), `app/core/settings_overlay.py` (the generic DB-over-env overlay — a plain `str` field rides it with zero extra code: `PUT` stores the raw string, `invalidate_settings()` re-reads), `app/rag/llm.py` — `LLMClient.chat()` (L446, the one-shot `chat.completions.create` L416, `temperature=0.2`, 2048 tokens) and `LLMClient.chat_stream()` (L511, the kwargs dict L592–600: `model` / `messages` / `temperature=0.4` / `max_tokens` / `stream`), `app/api/chat.py` — the per-turn log line (L1268–1291: `…thinking_chars=%d tool_calls=%d…`), `frontend/assets/settings.js` (the phase-168 generated panel — **no code change here**: a new `str` catalogue entry renders as a free-form text row automatically, exactly like `llm_chat_model`), `tests/unit/test_config.py`, `tests/unit/test_llm_client.py` (the mocked-SDK-client family), `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (the set/reset/live-apply contract), `tests/e2e/test_admin_settings_panel.py` (the phase-168 suite — the pattern to copy).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The operator can pick the model's thinking level — pi-style, free-form text, default `low` — from `.env` (`BOR_LLM_THINKING_LEVEL`) or the admin Settings panel, and the value rides every chat-completions request to aipi as the OpenAI-standard `reasoning_effort` parameter, so the same knob works as the model's reasoning support evolves. The empty value is the off switch (the parameter is not sent at all).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — one `Settings` field (task 01).** `llm_thinking_level: str = "low"` in the LLM section of `app/config.py` (placed directly after the `llm_summary_model` block, before `stream_thinking` — model-identity settings stay together). Free-form, exactly like `llm_chat_model` (A4: **no allowlist, no validator** — any non-empty text is accepted; the endpoint is the source of truth). The docstring names the env var (`BOR_LLM_THINKING_LEVEL`), the default, the wire mapping (D2) and the empty-off rule (D3). `.env.example` gains `BOR_LLM_THINKING_LEVEL=low` directly under `BOR_LLM_CHAT_MODEL=turbo` (L60) with a one-line comment (the house style of L61–63).
|
||||||
|
- **D2 — the wire (task 03).** The value is sent as a **top-level chat-completions parameter** `reasoning_effort` via the OpenAI SDK's `extra_body` (`extra_body={"reasoning_effort": "<value>"}` — the SDK merges `extra_body` into the request JSON body). Applied in BOTH `LLMClient.chat()` (the one-shot: document summaries, KB overview) and `LLMClient.chat_stream()` (the chat turn) — A3. The embeddings path (`_post_embeddings` / `embed` / `embed_one`) is **untouched**. `ASSUMPTION: A1 — aipi accepts the OpenAI-standard `reasoning_effort` top-level parameter (pi's `reasoningEffort`); if aipi exposes a different key (e.g. `chat_template_kwargs`), only task 03's key changes — the rest of the phase is key-agnostic by design.**
|
||||||
|
- **D3 — the off switch (tasks 03/04).** The **trimmed** value decides: non-empty → the parameter is sent with the trimmed value; empty/whitespace → `extra_body` is absent from the request (byte-identical to today's payload). Nothing is sent as `"off"` or `null` — absence is the off.
|
||||||
|
- **D4 — the admin row (task 02).** One `SettingMeta("llm_thinking_level", "LLM (aipi)", "Thinking level")` entry in `CATALOG` (placed after `llm_summary_model` L150–151, before `llm_retries` L156). Nothing else: the panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes, and the DB-over-env overlay are all **generic over the catalogue** — the row, its free-form text input, its save/reset and its live-apply fall out with zero extra code (A5). The completeness pin is the enforcement: a missing entry fails the build.
|
||||||
|
- **D5 — the per-turn line (task 04).** The per-turn log line (PLAN §9, `app/api/chat.py` L1268) gains `thinking=<value>` immediately before the existing `thinking_chars=%d` — the effective (overlay-resolved) value, trimmed, or `-` when empty, so "which level did this turn run at" stays one `grep` away (AGENTS rule 10). No `query_log` schema change (the table is retrieval-scoped; the line is the home of turn knobs).
|
||||||
|
- **NOT touched:** `frontend/assets/settings.js` (generated from the catalogue), `frontend/assets/styles.css`, the `app_settings` table shape, `app/core/settings_overlay.py` (generic), the embeddings path, `stream_thinking` (the phase-17 kill-switch for `thinking` SSE events — different concern, stays as-is), every other setting.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
— (none: the app is fully functional at the head of `complete/`; this phase builds on the committed phase-168/171 settings machinery)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_setting_and_env.md` — `Settings.llm_thinking_level: str = "low"` + the `CATALOG` row (the completeness pin keeps the build green) + `BOR_LLM_THINKING_LEVEL=low` in `.env.example`; the config/catalog unit tests.
|
||||||
|
2. `02_admin_row_contract.md` — the integration set / reset / live-apply contract for the new key (stored raw, live without restart, empty storable as the off switch).
|
||||||
|
3. `03_llm_extra_body_passthrough.md` — `extra_body={"reasoning_effort": …}` on `chat()` + `chat_stream()` when trimmed-non-empty; absent when empty; the mocked-client unit tests.
|
||||||
|
4. `04_per_turn_log_field.md` — `thinking=<value>` (or `-`) on the per-turn log line; the log-line test.
|
||||||
|
5. `05_e2e_admin_settings_row.md` — `tests/e2e/test_thinking_level.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_config.py` — the default is `low`, the env override works, an empty env value yields an empty string (no coercion, no allowlist — D3/A4). `tests/unit/test_settings_catalog.py` — the completeness pin stays green (the new field has its `SettingMeta`; group/label/flags correct). `tests/unit/test_llm_client.py` — `chat()` and `chat_stream()` send `extra_body={"reasoning_effort": <trimmed value>}` when the setting is non-empty, send **no** `extra_body` when it is empty/whitespace, and the embeddings path is unaffected (the mocked-SDK-client family).
|
||||||
|
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the row appears in `GET /api/admin/settings` (group, `str`, effective `low`); `PUT` stores the raw string, the empty value is storable (the off switch), the overlay resolves live without a restart; `DELETE` resets to env.
|
||||||
|
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_thinking_level.py`, run **in isolation** — the row is visible in the `LLM (aipi)` group with default `low`, is a free-form text input (type `high`, save, survives reload), reset-to-env restores `low`, and the `test_admin_settings_panel.py` regression set stays green.
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest -k "config and thinking" -v` shows the default `low` + the env override; `.env.example` contains `BOR_LLM_THINKING_LEVEL=low` under `BOR_LLM_CHAT_MODEL`.
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists `llm_thinking_level` in the `LLM (aipi)` group (`str`, default `low`); `PUT` / `DELETE` round-trip per the overlay contract.
|
||||||
|
- [ ] With the setting non-empty, a `chat_stream()` / `chat()` request carries `reasoning_effort` (unit-asserted via the mocked client); with it empty, the request kwargs contain no `extra_body` (byte-identical to today).
|
||||||
|
- [ ] The per-turn log line carries `thinking=<value>` (`-` when empty) next to `thinking_chars`.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 171) green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(llm): operator-tunable thinking level (BOR_LLM_THINKING_LEVEL, default low) sent as reasoning_effort`); the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
|
||||||
|
- **A1 — the wire key is `reasoning_effort`** (the OpenAI-standard top-level chat-completions parameter, pi's `reasoningEffort`), sent via the SDK's `extra_body`. If aipi exposes a different key, only task 03 changes.
|
||||||
|
- **A2 — the off switch is absence:** empty/whitespace value → the parameter is not sent at all; the default `low` is always sent.
|
||||||
|
- **A3 — scope:** the parameter rides BOTH `chat_stream()` (the chat turn) and `chat()` (the one-shot summaries / KB overview); the embeddings path is untouched.
|
||||||
|
- **A4 — free-form, no allowlist:** any non-empty text is accepted (like `llm_chat_model`); no validator, no value coercion.
|
||||||
|
- **A5 — no frontend code change:** the panel is generated from the catalogue; the row appears automatically as a free-form text input.
|
||||||
|
- **A6 — the per-turn line** gains `thinking=<trimmed value>` (`-` when empty) immediately before `thinking_chars`; no `query_log` schema change.
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Task 01 — The `llm_thinking_level` setting + the catalogue row + `.env.example`
|
||||||
|
|
||||||
|
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The new setting exists in `app/config.py` (default `low`, free-form like `llm_chat_model`), in the admin catalogue (so the build's completeness pin stays green — a field without a `SettingMeta` fails the build, and the per-task gate runs the full suite), and in `.env.example`, so `BOR_LLM_THINKING_LEVEL` is a first-class, documented knob before anything consumes it.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — in the LLM section, **directly after** the `llm_summary_model` block (L103–107) and **before** `stream_thinking` (L108), add ONE field, house docstring style (env var named, default, wire mapping, off rule):
|
||||||
|
```python
|
||||||
|
#: The model's thinking level (phase 172, ``BOR_LLM_THINKING_LEVEL``) —
|
||||||
|
#: pi-style reasoning effort, free-form text exactly like
|
||||||
|
#: ``llm_chat_model`` (no allowlist — the endpoint is the source of
|
||||||
|
#: truth). Default ``low``. Rides every chat-completions request to
|
||||||
|
#: aipi as the top-level ``reasoning_effort`` parameter
|
||||||
|
#: (``app/rag/llm.py``); the EMPTY value is the off switch — the
|
||||||
|
#: parameter is not sent at all.
|
||||||
|
llm_thinking_level: str = "low"
|
||||||
|
```
|
||||||
|
No validator, no coercion — a plain `str` (ASSUMPTION A4, phase-locked). Verify by reading: `llm_chat_model` (L101) keeps its shape byte-identical.
|
||||||
|
2. `.env.example` — in the LLM block (L57–65), directly under `BOR_LLM_CHAT_MODEL=turbo` (L60), add:
|
||||||
|
```
|
||||||
|
BOR_LLM_THINKING_LEVEL=low # the model's thinking level (pi-style; free-form, rides reasoning_effort); empty = the parameter is not sent
|
||||||
|
```
|
||||||
|
(one-line comment, the house style of L61–63; `#`-prefixed optional vars stay commented — this one is set, like L60).
|
||||||
|
3. `app/core/settings_catalog.py` — in the `LLM (aipi)` catalogue group, **directly after** the `llm_summary_model` entry (L150–151) and **before** `llm_retries` (L156), add ONE entry:
|
||||||
|
```python
|
||||||
|
"llm_thinking_level": SettingMeta(
|
||||||
|
"llm_thinking_level", "LLM (aipi)", "Thinking level"
|
||||||
|
),
|
||||||
|
```
|
||||||
|
Nothing else — `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` are DERIVED from `CATALOG` (module docstring), so no other list changes. The row is plain: not `secret`, not `restart_required` (the LLM client reads the setting per request via `get_settings()` — live), overridable (the overlay is generic over `str` fields, `app/core/settings_overlay.py`).
|
||||||
|
4. `tests/unit/test_settings_catalog.py` — the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green with the new pair; add the house-style assertion that `CATALOG["llm_thinking_level"]` sits in group `LLM (aipi)` with label `Thinking level` and is **not** in `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` (the derived sets).
|
||||||
|
5. `tests/unit/test_config.py` — add the house-style cases: (a) the DEFAULT is `"low"` (no env set); (b) the env override — `BOR_LLM_THINKING_LEVEL=high` resolves to `"high"`; (c) the empty env value (`BOR_LLM_THINKING_LEVEL=`) resolves to `""` (no coercion to a default, no allowlist — the off switch stays expressible).
|
||||||
|
|
||||||
|
- ASSUMPTION: A4 (phase-locked) — free-form text, no allowlist/validator; A2 (phase-locked) — empty = off switch, so the setting must NOT strip/normalize away the empty value at load (it is trimmed only at the wire, task 03); A5 (phase-locked) — the panel row falls out of the catalogue with zero `settings.js` code.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_config.py` + `tests/unit/test_settings_catalog.py` green (`uv run pytest tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest -k "thinking" tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov` green: default `low`, env override `high`, empty stays `""`, the completeness pin holds with the new pair.
|
||||||
|
- [ ] `grep -n "BOR_LLM_THINKING_LEVEL" .env.example` shows the line under `BOR_LLM_CHAT_MODEL=turbo`.
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists `llm_thinking_level` in the `LLM (aipi)` group (`str`, effective `low`) — the pin + a quick curl, or the task-02 integration test.
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (no other `Settings` field touched; `settings.js` untouched).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 02 — The admin row's API contract (set / reset / live-apply for `llm_thinking_level`)
|
||||||
|
|
||||||
|
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "… should be a free-form text entry like chat model. … Update .env and admin settings."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove, at the API/overlay level, that the new key behaves exactly like `llm_chat_model`: it appears in the catalogue, `PUT` stores the raw string and the overlay resolves it live (no restart), `DELETE` resets to env, and the effective value the LLM layer will read follows the overlay.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/integration/test_admin_settings_api.py` — extend the existing key-parameterized patterns (read how the phase-168 suite picks a `str` key, e.g. `llm_chat_model`) with the house-style cases for `llm_thinking_level`: (a) `GET /api/admin/settings` → the item is present, group `LLM (aipi)`, `type str`, effective value `low` (no row in `app_settings`), `env_var` `BOR_LLM_THINKING_LEVEL`; (b) `PUT` with `{"value": "high"}` → 200, the row is stored **raw** in `app_settings` (env-native string form, D1 of the overlay), the response reports the new effective value; (c) `PUT` with an empty string `""` → 200 (the off switch is storable — the API stores as-is, no allowlist); (d) `DELETE /api/admin/settings/llm_thinking_level` → the row is gone, effective value back to the env default `low`; (e) the non-admin 403 gate still applies (the phase-168 router dependency — assert once, mirroring the existing suite).
|
||||||
|
2. `tests/integration/test_admin_settings_live.py` — add the live-apply case in the house pattern: `PUT llm_thinking_level=high` → `get_settings().llm_thinking_level` is `"high"` **without** a restart (the `invalidate_settings()` path); `DELETE` → back to `"low"`. (This is the proof the LLM layer — which reads `get_settings()` per request, task 03 — picks the override up live.)
|
||||||
|
3. If the phase-168 suites parameterize over `CATALOG` keys generically (check `tests/integration/test_admin_settings_api.py` first — read it, do not assume): the new key may already be covered by the parameterization; in that case add only the empty-string (c) case and the live-apply (2) case, and note it in the test's comment.
|
||||||
|
|
||||||
|
- ASSUMPTION: A4 (phase-locked) — the empty string is storable (the off switch must be settable from the panel, not only via `.env`); the API "stores as-is" per the phase-168 module docstring — no value validation to add.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` green (`uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged `app/` code — the gate must stay green, not drop).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists the row (`LLM (aipi)`, `str`, `low`); `PUT high` → live effective `high`; `PUT ""` → live effective `""`; `DELETE` → back to `low` (the integration cases green).
|
||||||
|
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v --no-cov` green; full suite green.
|
||||||
|
- [ ] ruff + pyright clean; no `app/` code changed by this task.
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Task 03 — The thinking level rides every chat-completions request (`reasoning_effort` via `extra_body`)
|
||||||
|
|
||||||
|
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. …"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`LLMClient` sends the operator's thinking level to aipi on every chat-completions request — `chat()` (the one-shot) and `chat_stream()` (the chat turn) — as the top-level `reasoning_effort` parameter via the SDK's `extra_body`; when the (trimmed) value is empty the request is byte-identical to today's (no `extra_body` at all).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/llm.py` — ONE small helper (module-level, near the top, after the imports):
|
||||||
|
```python
|
||||||
|
def _thinking_extra_body(settings: Settings) -> dict[str, str] | None:
|
||||||
|
"""The phase-172 thinking level as the chat-completions extra body.
|
||||||
|
|
||||||
|
Non-empty (trimmed) value → ``{"reasoning_effort": <value>}`` (the
|
||||||
|
OpenAI-standard top-level parameter, A1); empty/whitespace → ``None``
|
||||||
|
(the off switch — the parameter is ABSENT from the request, A2).
|
||||||
|
"""
|
||||||
|
level = settings.llm_thinking_level.strip()
|
||||||
|
return {"reasoning_effort": level} if level else None
|
||||||
|
```
|
||||||
|
Then in **BOTH** request sites, add the parameter only when present:
|
||||||
|
- `chat()` (L446, the `await self._client.chat.completions.create(...)` at L416–421): compute `extra = _thinking_extra_body(self.settings)` and pass the kwarg **only when present** — `**({"extra_body": extra} if extra is not None else {})` spread into the `create(...)` call, so `extra_body` is ABSENT (never `None`) when off.
|
||||||
|
- `chat_stream()` (L511, the kwargs dict L592–600): same pattern — after the dict is built (and after the `if tools is not None:` block, L601–602), `if (extra := _thinking_extra_body(self.settings)) is not None: kwargs["extra_body"] = extra`.
|
||||||
|
The value is the **trimmed** setting (`.strip()` in the helper) — the raw stored string is never sent with surrounding whitespace. The embeddings path (`_post_embeddings` L256, `embed` L346, `embed_one` L400) is **untouched** (A3) — verify by reading: no `extra_body` appears anywhere in the embedding code.
|
||||||
|
2. `app/rag/llm.py` — update the module docstring (L1–L30) with ONE sentence: the thinking level (phase 172, `BOR_LLM_THINKING_LEVEL`) rides chat completions as `reasoning_effort`; empty = absent. (The house style: wire facts live in the module docstring, cf. the `reasoning_content` note at L10–12.)
|
||||||
|
3. `tests/unit/test_llm_client.py` — extend the mocked-SDK-client family (read the existing `chat` / `chat_stream` cases first and mirror their mocking):
|
||||||
|
- `chat()` with `llm_thinking_level="low"` → `create` is awaited with `extra_body={"reasoning_effort": "low"}` (assert on the call kwargs).
|
||||||
|
- `chat()` with `llm_thinking_level=" "` (whitespace) → `extra_body` is **absent from the call kwargs** (assert `"extra_body" not in kwargs`).
|
||||||
|
- `chat_stream()` with `"high"` → same `extra_body` assertion on the `create` call.
|
||||||
|
- `chat_stream()` with `""` → `extra_body` absent (the off switch — the kwargs dict is byte-identical in shape to pre-phase-172).
|
||||||
|
- one `embed()` / `embed_one()` case with the level set → the HTTP POST body (the mocked transport) contains **no** `reasoning_effort` (the embeddings path is untouched, A3).
|
||||||
|
- the TRIM case: `llm_thinking_level="low "` (trailing spaces — reachable via a hand-edited `.env`) → the sent value is `"low"`.
|
||||||
|
|
||||||
|
- ASSUMPTION: A1 (phase-locked) — the wire key is the OpenAI-standard top-level `reasoning_effort` via `extra_body`; if aipi exposes a different key, ONLY this task's helper changes (the setting, the row, the log, the E2E are key-agnostic). A2 (phase-locked) — absence is the off switch (never `extra_body=None`, never `"off"`). A3 (phase-locked) — `chat()` + `chat_stream()` only; embeddings untouched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_llm_client.py` green (`uv run pytest tests/unit/test_llm_client.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (the new helper is fully covered by the cases above).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] With the setting non-empty, a `chat()` / `chat_stream()` request carries `extra_body={"reasoning_effort": <trimmed value>}` (the unit cases green).
|
||||||
|
- [ ] With it empty/whitespace, `extra_body` is absent from the request kwargs (byte-identical shape to today); the embeddings POST body never contains `reasoning_effort`.
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (temperature / max_tokens / tools / stream kwargs untouched; the `stream_thinking` kill-switch untouched).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 04 — The per-turn log line records the effective thinking level
|
||||||
|
|
||||||
|
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. …"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The per-turn log line (PLAN §9, AGENTS rule 10) records the effective thinking level of the turn, so "which level did this turn run at" is one `grep` away, next to the `thinking_chars` it governs.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/chat.py` — the per-turn `logger.info(...)` (L1268–1291): insert `thinking=%s` **immediately before** the existing `thinking_chars=%d` in the format string, with the argument `settings.llm_thinking_level.strip() or "-"` (the effective overlay-resolved value — `settings` is already in scope, the line already logs `settings.relevance_threshold` L1280; trimmed, `-` when empty, per A6). No reordering of the existing fields; the `query_log` row (L1250–1260) is **untouched** (A6: the table is retrieval-scoped, the line is the home of turn knobs).
|
||||||
|
2. `app/api/chat.py` — the module docstring: the existing per-turn-line notes (L14, L46, L64, L78, L141, L161) set the house pattern — add ONE line where the log line's fields are described: the per-turn log line records `thinking=<level>` (the effective `BOR_LLM_THINKING_LEVEL`, `-` when empty) before `thinking_chars` (phase 172).
|
||||||
|
3. The existing per-turn-line test (find it: `grep -rn "thinking_chars" tests/` — the house pattern pins the line's fields in one place; extend that test, do not invent a new file): assert the new `thinking=` field is present, carries the effective value in the non-empty case, and is `-` when the setting is empty.
|
||||||
|
|
||||||
|
- ASSUMPTION: A6 (phase-locked) — `thinking=<trimmed value>` or `-`, placed immediately before `thinking_chars`; no `query_log` schema change, no new `Settings` read (the per-turn `settings` object is already resolved through the overlay).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the per-turn-line test (above) green; `uv run pytest` green (the SSE turn machinery is untouched — the change is one format field + one argument).
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green, not drop).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A test turn's per-turn log line (the test's captured log) contains `thinking=low` (default) and `-` when the setting is empty; `thinking_chars` stays in its position after it.
|
||||||
|
- [ ] `grep -rn "thinking=" app/api/chat.py` shows the field exactly once in the format string.
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean; no `query_log` schema change.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 05 — The phase's dedicated Playwright suite (AGENTS rule 4) + the full gates + the atomic commit
|
||||||
|
|
||||||
|
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "… should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`tests/e2e/test_thinking_level.py` proves the owner's acceptance in a real browser — the Thinking level row exists in the admin Settings panel as a **free-form text** entry (like the chat model's), defaults to `low`, saves and persists, and resets to env — run **in isolation**; then the full gates, then ONE atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_thinking_level.py` — the phase's dedicated suite, run in isolation (DB must be up: `podman compose up -d db`; `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov`). **Copy the harness from `tests/e2e/test_admin_settings_panel.py`** (read it first, then mirror: the `_boot_app`/`app_server`/`app_url` fixtures with the mock LLM, the `_clean` autouse fixture (its `app_settings` truncation pattern — the new key's rows must be wiped between tests), the `_cookies`/`_admin_client` helpers, the settings-page nav + Save/Reset drives at L561ff). Three tests:
|
||||||
|
- `test_row_renders_free_form_with_default_low` — admin opens `/settings.html`; the `LLM (aipi)` group band is visible (the phase-171 band contract); a row labelled **Thinking level** (`BOR_LLM_THINKING_LEVEL` env-var display, the catalogue pattern) is present, its input is a **free-form text** input (`type="text"` — NOT `number`/`select`, the A4 contract), shows `low` with the `env` badge. Assert the `llm_chat_model` row's input is the same `type="text"` control (the "like chat model" reference, in one assertion pair).
|
||||||
|
- `test_saved_level_persists_without_restart` — type `high` into the row (free-form: also type a custom value `think-soft` once, to prove no allowlist — A4), Save (the phase-168 §7.4 Save lifecycle: dirty → save → `DB` badge), the effective value reads `think-soft`; reload `/settings.html` → still `think-soft` with the `DB` badge (no restart, the phase-168 contract).
|
||||||
|
- `test_reset_returns_to_env_default` — the row's Reset (per-row `DELETE`) → the value reads `low` again with the `env` badge (the overlay reset, task-02's contract, in the browser).
|
||||||
|
2. Run the **full gates** in order, all from the repo root:
|
||||||
|
- `uv run pytest -v` (unit + integration) and `uv run pytest --cov=app --cov-report=term-missing` → `app/` coverage **>90%**.
|
||||||
|
- `uv run ruff check . && uv run pyright`.
|
||||||
|
- `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` → green **in isolation**.
|
||||||
|
- Regression, in isolation: `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168) and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171) → both green.
|
||||||
|
3. The **atomic commit** (AGENTS rule 8 — one commit per phase, Conventional Commits, `--no-gpg-sign`): the whole phase — `app/config.py`, `app/core/settings_catalog.py`, `app/rag/llm.py`, `app/api/chat.py`, `.env.example`, the four test files, and `.agents/phases/todo/172_thinking_level/` → moved to `.agents/phases/complete/172_thinking_level/` in the SAME commit. Message: `feat(llm): operator-tunable thinking level (BOR_LLM_THINKING_LEVEL, default low) sent as reasoning_effort` + the `LLM-Generated: true` trailer. The pre-existing dirty `TODO.md` (the now-converted entry, cleared in Phase 4 of this conversion) rides this commit ONLY if the conversion commit has not already committed it — check `git status` first; the conversion commit is the one that clears TODO.md (see the skill's Phase 4), so the phase commit should be the phase files + code only.
|
||||||
|
|
||||||
|
- ASSUMPTION: A5 (phase-locked) — no `settings.js` / `styles.css` change; the row renders from the catalogue. A6 (phase-locked) — the per-turn-line assertion lives in the unit/integration layer (task 04), not the E2E (the mock LLM turn does not exercise the wire; the E2E proves the settings surface only).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `tests/e2e/test_thinking_level.py` green **in isolation**; the phase-168 + phase-171 suites green in isolation (regression).
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` green in isolation (the three contracts).
|
||||||
|
- [ ] `uv run pytest` green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] `test_admin_settings_panel.py` + `test_admin_settings_layout.py` green in isolation (no regression in the settings machinery).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit with the `LLM-Generated: true` trailer; the phase dir moved to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Phase 173 — Docs-push credentials: split the username/token out of `BOR_DOCS_REPO`
|
||||||
|
|
||||||
|
**Source:** owner audit of the production admin Settings panel (2026-10-10): `BOR_DOCS_REPO` is a plain-text row whose production value is a git URL with an embedded username + Gitea token (`https://user:token@host/…`) — a live credential leak on the settings page, the raw HTML, and the settings API.
|
||||||
|
**Story:** n/a (owner-directed, chat-confirmed 2026-10-10)
|
||||||
|
**Context:** `app/config.py` — the Docs push section (`docs_repo` L539, `docs_branch` L545, `docs_base_branch` L549, `docs_work_dir` L554; the branch validator L831–846 requires non-empty branches only while `docs_repo` is set; `docs_configured` L902–906 is the 409 inert-switch on `docs_repo` alone — **untouched**), `.env.example` L229–234 (the Docs push block — commented optional vars), `app/core/settings_catalog.py` L303–308 (the `Docs push` group: `docs_repo` L303, `docs_branch` L304, `docs_base_branch` L305–307, `docs_work_dir` L308) and the existing secret mechanism (`llm_api_key` L142 `secret=True`; `SECRETS` is DERIVED from `CATALOG` — a `secret=True` entry is write-only end-to-end: the API reports `is_set` + `value=null` (`app/api/admin_settings.py` L147), a blank PUT 422s (L273), and the phase-168/171 panel renders the row masked as `(stored)` with zero `settings.js` code), `scripts/git_sync.py` L320 (`run_git(argv, cwd)` — the ONLY git invocation site, A11; `subprocess.run(argv, cwd=cwd, capture_output=True, text=True)` L331 — no env today), `app/core/docs_push.py` (`push_document` — clone L104–116 / fetch / checkout / add / commit / push, every step via `run_git`; `DocsPushError` carries the full git stderr for logs), `app/api/doc_drafts.py` L299–316 (the push call site; phase 84 `sanitize_error` already strips echoed credentials from the 502 response — defense-in-depth, stays), `app/core/settings_overlay.py` (`load_overrides()` L93 — the raw `app_settings` DB map; `resolve_settings()` L239; `reset_state_for_tests()` L274), `app/main.py` L98–245 (`create_app()` the composition root; `app = create_app()` L245), `tests/unit/test_config.py`, `tests/unit/test_settings_catalog.py`, `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (the generic set/reset/live-apply contract), `tests/e2e/test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 168/171 — the regression set), `tests/e2e/test_response_to_docs.py` (phase 59 — the docs-push E2E).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The docs-push remote's credentials live in their own settings — `BOR_DOCS_USER` (plain) and `BOR_DOCS_TOKEN` (write-only, masked like `BOR_LLM_API_KEY`) — while `BOR_DOCS_REPO` holds a clean URL. At push time the credentials travel as git's `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` environment variables (never in the command line / process list), a one-time startup migration splits any legacy `://user:pass@` value into the new settings (DB rows shadow the env, `.env` is never rewritten), and a PUT guard refuses a credentials-bearing repo URL so the leak cannot be re-created from the UI. Anonymous pushes (no token) stay byte-identical to today.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the three settings (task 01).** `docs_user: str = ""` and `docs_token: str = ""` in the Docs push section of `app/config.py` (directly after `docs_repo` L539, before `docs_branch` L545). Plain `str`, no validator (empty = "no credentials" is a valid state — the repo may be public/anonymous, exactly like today). `.env.example` L231–234 block gains `# BOR_DOCS_USER=` / `# BOR_DOCS_TOKEN=` (commented — optional, house style of the block). CATALOG: `docs_user` plain, `docs_token` **`secret=True`** (inserted directly after `docs_repo` L303, before `docs_branch` L304). The `SECRETS` derivation + the generic overlay + the phase-168/171 panel make `docs_token` a write-only, masked row with zero extra code (C1). No frontend code changes anywhere in this phase.
|
||||||
|
- **D2 — the PUT guard (task 02).** In `app/api/admin_settings.py`, the `PUT` route refuses `docs_repo` values that embed credentials — a `://<user>:<pass>@` segment (regex `://[^/@:]+:[^@/]+@`) — with a 422 whose detail names the replacement (use the Docs user / Docs token fields, keep the URL clean). Everything else about the row is unchanged (it stays overridable, plain). `docs_token`'s API behavior is the EXISTING generic secret contract (GET `is_set`/`value=null`, blank PUT 422, DELETE resets) — task 02 pins it with integration assertions, it needs no new code.
|
||||||
|
- **D3 — credentials on the wire (task 03).** `run_git` gains an optional `env: dict[str, str] | None = None` passed straight to `subprocess.run` — **the subprocess env-replacement trap**: when set, the dict REPLACES the parent environment, so `docs_push` must build it as `{**os.environ, …}` (a test asserts `PATH` survives). `push_document` gains `user: str = ""` / `token: str = ""`; when `token.strip()` is non-empty, EVERY git step of the sequence (clone, both fetches, checkout, add, commit, push) gets `env = {**os.environ, "GIT_CURL_USERNAME": user, "GIT_CURL_PASSWORD": token, "GIT_TERMINAL_PROMPT": "0"}` (git's documented HTTP credential env; `GIT_TERMINAL_PROMPT=0` fails fast instead of hanging a headless server on a prompt); when the token is empty, `env` stays `None` and the invocations are byte-identical to today (C5). The clean URL stays in the argv. `app/api/doc_drafts.py` passes `settings.docs_user` / `settings.docs_token`. `GIT_CURL_*` only takes effect for `http(s)` remotes; for `ssh://` / local-path remotes it is inert (accepted). Phase 84's `sanitize_error` on the 502 response is untouched (defense-in-depth).
|
||||||
|
- **D4 — the legacy migration (task 04).** New module `app/core/settings_migrate.py` with `migrate_legacy_docs_creds() -> bool`: read the `app_settings` overrides; take the EFFECTIVE `docs_repo` (DB override if present, else env — i.e. `resolve_settings().docs_repo`); if it matches `://<user>:<pass>@` (both non-empty) AND there is no `docs_token` override yet, write three DB rows — `docs_repo` = the credential-stripped URL, `docs_user` = user, `docs_token` = pass — and log one INFO line (`settings: migrated embedded docs-push credentials out of docs_repo into docs_user/docs_token`); any other shape → no-op. Idempotent (a `docs_token` row already present → never clobber owner-set values). Called ONCE from `create_app()` (the composition root) after the DB is available. `.env` is never read for writing or rewritten — the DB rows shadow the env (the settings model's standing rule). Unit-tested against the overlay module's state + `reset_state_for_tests()`.
|
||||||
|
- **D5 — the E2E (task 05).** `tests/e2e/test_docs_push_creds.py` — the phase's dedicated Playwright suite (AGENTS rule 4): the Docs push group shows a `Docs repo` text row, a `Docs user` text row, and a `Docs token` write-only row rendered masked (`(stored)`, not pre-filled, `type=password`); typing a token, saving, and reloading keeps the row masked and the typed token text is **absent from the entire page DOM** (the core assertion — the leak class this phase kills); resetting the token unsets it. The regression set (`test_admin_settings_panel.py`, `test_admin_settings_layout.py`, `test_response_to_docs.py`) stays green.
|
||||||
|
- **NOT touched:** `frontend/` (no file), the `app_settings` table shape, `settings_overlay.py` internals, `docs_configured` / the 409 inert-switch, the branch validator, `git_sync.clone_or_pull` (sync path — no credential settings exist for it; out of scope), `sanitize_error` (phase 84), every other setting.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
— (none: the app is fully functional at the head of `complete/`; this phase builds on the committed phase-59 docs-push, phase-168/171 settings machinery, and phase-84 error sanitization)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_config_and_catalog.md` — `Settings.docs_user` / `docs_token` (default `""`) + the two CATALOG rows (`docs_token` `secret=True`) + `.env.example`; the config/catalog unit tests.
|
||||||
|
2. `02_api_masking_and_guards.md` — integration pin of the `docs_token` write-only contract + the new `docs_repo` embedded-credentials PUT guard (422); the guard's unit/integration tests.
|
||||||
|
3. `03_run_git_env_and_push.md` — `run_git(env=None)` + `push_document(user, token)` credential env on every git step (merged over `os.environ`; `None` when the token is empty) + the `doc_drafts.py` call site; the mocked-git unit tests.
|
||||||
|
4. `04_legacy_creds_migration.md` — `app/core/settings_migrate.py` (idempotent, DB-shadowing, one log line) wired into `create_app()`; unit + integration tests.
|
||||||
|
5. `05_e2e_docs_creds_masking.md` — `tests/e2e/test_docs_push_creds.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_config.py` — `docs_user` / `docs_token` default `""`, env overrides, empty env stays `""` (no coercion). `tests/unit/test_settings_catalog.py` — the completeness pin holds with the two new fields; `docs_token` is in `SECRETS`, `docs_user` is in none of the derived sets. `tests/unit/test_git_sync.py` (or the existing `run_git` test home) — `env=None` leaves the subprocess call byte-identical; a passed env reaches `subprocess.run` verbatim. `tests/unit/test_docs_push.py` — token empty → every `run_git` call gets `env=None`; token set → every git step gets the 3-var env merged over `os.environ` (`PATH` present); `DocsPushError` propagation unchanged. `tests/unit/test_settings_migrate.py` — the 4 migration shapes (D4).
|
||||||
|
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — `docs_token` in `GET /api/admin/settings` (group `Docs push`, `secret=true`, `value=null`, `is_set` flips on PUT); PUT stores / DELETE resets / blank PUT 422 (the generic contract pinned for the new key); `docs_repo` PUT with `://user:pass@` → 422 naming the user/token fields, clean URLs pass. `tests/integration/test_docs_push_migrate.py` — seeded env `BOR_DOCS_REPO=https://u:pass@host/r.git`, no DB rows → after the migration runs, `resolve_settings().docs_repo` is `https://host/r.git`, `docs_user`/`docs_token` resolve from the DB; a second run is a no-op.
|
||||||
|
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_docs_push_creds.py`, run **in isolation** — per D5.
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists `docs_user` (plain) and `docs_token` (secret, `value=null`) in the `Docs push` group; the token value is NEVER in any response body.
|
||||||
|
- [ ] `PUT /api/admin/settings` for `docs_repo` with a `://user:pass@` value → 422 naming the Docs user / Docs token fields; a clean URL passes.
|
||||||
|
- [ ] With `docs_token` set, a docs push invokes git with `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` / `GIT_TERMINAL_PROMPT=0` in the environment and a clean URL in the argv (unit-asserted via mocked `run_git`); with it empty, the `run_git` calls are byte-identical to today (`env=None`).
|
||||||
|
- [ ] A legacy `://user:pass@` env value is split once at startup into `app_settings` rows (clean repo + user + token), `.env` untouched, a second run is a no-op; the INFO migration line is logged exactly once.
|
||||||
|
- [ ] `uv run pytest` green (100% — every test in the project, pre-existing included), `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` + `test_response_to_docs.py` green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(settings): split docs-push credentials (BOR_DOCS_USER/BOR_DOCS_TOKEN) out of the repo URL — masked, env-wired, legacy auto-migration`); the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
- [ ] `README.md` and `AGENTS.md` are accurate and consistent with the project as it will stand after the phase (the final pass reviews both in full and fixes every inconsistency found, including pre-existing ones).
|
||||||
|
|
||||||
|
## Documentation & Consistency (mandatory)
|
||||||
|
The final pass reviews `README.md` and `AGENTS.md` in their entirety against the project as it will stand after the phase and fixes every inconsistency found — including pre-existing ones not caused by the phase. Tasks update both files whenever their own changes are user-visible (the `.env.example` block is the user-visible docs surface for this phase; the README's docs-push section gains the two new vars + a line that credentials in the repo URL are no longer supported).
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-10, chat)
|
||||||
|
- **C1 — the split shape:** `docs_repo` (clean URL) + `docs_user` (plain `str`) + `docs_token` (`secret=True` in the CATALOG — reuses the existing write-only/masked mechanism end-to-end, `llm_api_key`-style). No new UI code: the panel renders both rows from the catalogue.
|
||||||
|
- **C2 — the credential transport:** `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` (+ `GIT_TERMINAL_PROMPT=0`) environment variables on the push sequence — never in the git argv / process list; applied only when `docs_token` is non-empty.
|
||||||
|
- **C3 — the migration:** one-time at startup (`create_app()`), effective-value based, DB rows shadow the env, `.env` never rewritten, idempotent, one INFO log line.
|
||||||
|
- **C4 — the guard:** `PUT docs_repo` with embedded credentials → 422 (the leak cannot be re-created via the UI).
|
||||||
|
- **C5 — no behavior change for anonymous pushes:** empty `docs_token` → the git invocations are byte-identical to today (`env=None`).
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Task 01 — The `docs_user` / `docs_token` settings + the catalogue rows + `.env.example`
|
||||||
|
|
||||||
|
**Phase:** `173_docs_push_creds_split` · **Source:** owner audit of the production admin Settings panel (2026-10-10): `BOR_DOCS_REPO` leaks its embedded Gitea token in plain text; the credentials get their own settings.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The two new settings exist in `app/config.py` (default `""`), in the admin catalogue (so the build's completeness pin stays green — a field without a `SettingMeta` fails the build, and the per-task gate runs the full suite), and in `.env.example`, so `BOR_DOCS_USER` / `BOR_DOCS_TOKEN` are first-class, documented knobs before anything consumes them. `docs_token` is `secret=True` — write-only, masked end-to-end via the EXISTING mechanism (C1); nothing else in the phase touches `frontend/`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — in the Docs push section, **directly after** the `docs_repo` block (L539) and **before** `docs_branch` (L545), add TWO fields, house docstring style (env var named, default, consumption point, off rule):
|
||||||
|
```python
|
||||||
|
#: Docs-push remote username (phase 173, ``BOR_DOCS_USER``) — paired
|
||||||
|
#: with ``docs_token``; travels to git as the ``GIT_CURL_USERNAME``
|
||||||
|
#: environment variable (``app/core/docs_push.py``, task 03) — never
|
||||||
|
#: in the command line. Empty (default) = no credentials: the push
|
||||||
|
#: is anonymous/URL-credential-free, exactly as before this phase.
|
||||||
|
docs_user: str = ""
|
||||||
|
#: Docs-push remote token (phase 173, ``BOR_DOCS_TOKEN``) — write-only
|
||||||
|
#: (catalogue ``secret=True``): the API reports ``is_set`` and never
|
||||||
|
#: the value; the admin panel renders it masked. Travels to git as
|
||||||
|
#: ``GIT_CURL_PASSWORD`` (``app/core/docs_push.py``, task 03).
|
||||||
|
#: Empty (default) = the credential env is not set at all.
|
||||||
|
docs_token: str = ""
|
||||||
|
```
|
||||||
|
No validator, no coercion — plain `str`; an empty value is a valid state (C1/C5). The L831–846 branch validator (branches required only while `docs_repo` is set) and `docs_configured` (L902–906) stay byte-identical.
|
||||||
|
2. `.env.example` — in the Docs push block (L229–234), directly under `# BOR_DOCS_REPO=/path/to/docs-repo` (L231), add:
|
||||||
|
```
|
||||||
|
# BOR_DOCS_USER=
|
||||||
|
# BOR_DOCS_TOKEN=
|
||||||
|
```
|
||||||
|
(commented — optional, the block's house style; add one line to the block's comment: credentials in the repo URL are no longer supported — use BOR_DOCS_USER/BOR_DOCS_TOKEN).
|
||||||
|
3. `app/core/settings_catalog.py` — in the `Docs push` group, **directly after** the `docs_repo` entry (L303) and **before** `docs_branch` (L304), add TWO entries:
|
||||||
|
```python
|
||||||
|
"docs_user": SettingMeta("docs_user", "Docs push", "Docs user"),
|
||||||
|
"docs_token": SettingMeta(
|
||||||
|
"docs_token", "Docs push", "Docs token", secret=True
|
||||||
|
),
|
||||||
|
```
|
||||||
|
Nothing else — `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` are DERIVED from `CATALOG` (module docstring): `docs_token` lands in `SECRETS` automatically, `docs_user` in none. Both are plain `str` → the overlay is generic (live, no restart). The panel row falls out of the catalogue with zero `settings.js` code (C1 — the phase-168/171 machinery renders a `secret=True` row masked as `(stored)`, `llm_api_key`-style).
|
||||||
|
4. `tests/unit/test_settings_catalog.py` — the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green with the two new fields; add the house-style assertions: `CATALOG["docs_user"]` sits in group `Docs push` with label `Docs user` and is in **none** of the derived sets; `CATALOG["docs_token"]` has label `Docs token`, `secret=True`, and **is** in `SECRETS`.
|
||||||
|
5. `tests/unit/test_config.py` — add the house-style cases: (a) the DEFAULTS are `""` / `""` (no env set); (b) the env override — `BOR_DOCS_USER=alice` / `BOR_DOCS_TOKEN=tok` resolve to `"alice"` / `"tok"`; (c) the empty env values stay `""` (no coercion, no normalization away from empty — C5's anonymous state stays expressible).
|
||||||
|
|
||||||
|
- ASSUMPTIONS: C1 (phase-locked) — the split shape; C5 (phase-locked) — empty token = no credential env, so the settings must NOT strip/coerce away the empty value at load.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_config.py` + `tests/unit/test_settings_catalog.py` green (`uv run pytest tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest -k "docs" tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov` green: defaults `""`, env overrides, empty stays `""`, the completeness pin holds, `docs_token` in `SECRETS`, `docs_user` in none.
|
||||||
|
- [ ] `grep -n "BOR_DOCS_USER" .env.example` shows the line under `# BOR_DOCS_REPO=…` (commented).
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists `docs_user` + `docs_token` in the `Docs push` group — `docs_token` as `secret=true` with `value=null` (a quick curl, or the task-02 integration test).
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (no other `Settings` field touched; `frontend/` untouched; `docs_configured` + the branch validator byte-identical).
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 02 — The `docs_token` write-only contract (pinned) + the `docs_repo` embedded-credentials guard
|
||||||
|
|
||||||
|
**Phase:** `173_docs_push_creds_split` · **Source:** C2/C4 (phase-locked) — the token is write-only end-to-end via the existing mechanism; a credentials-bearing repo URL is refused on the way in so the leak cannot be re-created from the UI.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The `docs_token` row's API behavior is the EXISTING generic secret contract — task 02 pins it with integration assertions for the new key (no new code expected). The ONE piece of new logic: the `PUT` route refuses `docs_repo` values with embedded credentials (C4), pointing the operator at the Docs user / Docs token fields.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/admin_settings.py` — in the `PUT` route, **before** the existing blank-secret 422 check (L273) or alongside it, add the `docs_repo` guard (a small module-level helper keeps the route readable — house style):
|
||||||
|
```python
|
||||||
|
#: A git URL with embedded credentials (``://<user>:<pass>@``) — the
|
||||||
|
#: phase-173 leak class. Refused for ``docs_repo`` on the way in:
|
||||||
|
#: the credentials belong in ``docs_user`` / ``docs_token`` (C4).
|
||||||
|
_EMBEDDED_CREDS = re.compile(r"://[^/@:]+:[^@/]+@")
|
||||||
|
|
||||||
|
def _refuse_embedded_credentials(key: str, value: str) -> None:
|
||||||
|
if key == "docs_repo" and _EMBEDDED_CREDS.search(value):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=422,
|
||||||
|
detail=(
|
||||||
|
"docs_repo must not contain embedded credentials — "
|
||||||
|
"keep the URL clean and set them in the Docs user / "
|
||||||
|
"Docs token fields instead"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
Call it at the top of the per-key validation in the PUT route (after the payload is parsed, before store). Only `docs_repo` is guarded — `ssh://` URLs, local paths, and clean `https://` URLs pass untouched. (A legacy `.env` value with embedded credentials is NOT a PUT — the task-04 startup migration handles that path; the guard covers only new writes.)
|
||||||
|
2. `tests/integration/test_admin_settings_api.py` (or `test_admin_settings_live.py` — match the file that owns the secret-row contract; the phase-172 pin is the template) — add the house-style cases for the new key:
|
||||||
|
- `GET /api/admin/settings` (admin): `docs_token` in group `Docs push`, `secret=true`, `value=null`, `is_set=false`; `docs_user` plain, `value=""`.
|
||||||
|
- `PUT docs_token` with a value → 200; `GET` → `is_set=true`, `value=null` (the value is **never** in the response body — assert the token text is absent from the raw response JSON).
|
||||||
|
- `PUT docs_token` with a blank value → 422 (the existing self-lockout guard — pinned for the new key).
|
||||||
|
- `DELETE docs_token` → `is_set=false` (reset to env — `""` by default).
|
||||||
|
3. `tests/integration/test_admin_settings_api.py` — the guard's integration cases: `PUT docs_repo` `https://u:pass@gitea.example.com/o/r.git` → 422, detail mentions the Docs user / Docs token fields; `PUT docs_repo` `https://gitea.example.com/o/r.git` → 200 (clean passes); `PUT docs_repo` a local path (`/srv/docs-repo`) → 200 (untouched); a `user:` segment that is NOT `://user:pass@` (e.g. `https://sub.domain.example.com/r.git`) → 200 (the regex must not false-positive on subdomains — the `@` requirement does that job).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the cases above green (`uv run pytest tests/integration/test_admin_settings_api.py -v --no-cov`).
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green — the guard helper must be covered by the 422 + pass cases).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py -v --no-cov` green, including: `docs_token` GET `is_set`/`value=null` (token text absent from the raw JSON), PUT store, blank PUT 422, DELETE reset; `docs_repo` embedded-creds 422 + clean/local/subdomain 200s.
|
||||||
|
- [ ] The 422 detail names the Docs user / Docs token fields (asserted in the test, not just present).
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (no other key's PUT path touched; the blank-secret guard at L273 byte-identical).
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Task 03 — Credentials on the wire: `run_git(env=…)` + `push_document(user, token)`
|
||||||
|
|
||||||
|
**Phase:** `173_docs_push_creds_split` · **Source:** C2/C5 (phase-locked) — credentials travel as `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` env (never in the argv / process list); an empty token leaves the invocations byte-identical to today.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The docs-push git sequence authenticates with the settings' user/token via git's documented HTTP credential environment, without the credential ever appearing in the command line, and without changing a single invocation when no token is set (C5).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `scripts/git_sync.py` — `run_git` (L320) gains one optional parameter:
|
||||||
|
```python
|
||||||
|
def run_git(argv: list[str], cwd: Path, env: dict[str, str] | None = None) -> str:
|
||||||
|
…
|
||||||
|
proc = subprocess.run(
|
||||||
|
argv, cwd=cwd, capture_output=True, text=True, env=env
|
||||||
|
)
|
||||||
|
```
|
||||||
|
`None` (the default, and the sync path's only usage) is byte-identical to today. **The subprocess env-replacement trap:** when `env` is a dict, it REPLACES the parent environment — the caller must build a complete environment (task 03's `docs_push` does the merge; a unit test pins `PATH` survival). Update the docstring: the `env` parameter is passed through verbatim; `None` = inherit.
|
||||||
|
2. `app/core/docs_push.py` — `push_document` gains `user: str = ""` / `token: str = ""` (positional order: appended AFTER `commit_message`, so every existing call site keeps working; update the docstring with the C2 transport + the C5 no-token rule). At the top of the sequence:
|
||||||
|
```python
|
||||||
|
git_env: dict[str, str] | None = None
|
||||||
|
if token.strip():
|
||||||
|
# C2: git's documented HTTP credential env — never in the argv.
|
||||||
|
# GIT_TERMINAL_PROMPT=0 fails fast instead of hanging the headless
|
||||||
|
# server on a credential prompt.
|
||||||
|
git_env = {
|
||||||
|
**os.environ,
|
||||||
|
"GIT_CURL_USERNAME": user,
|
||||||
|
"GIT_CURL_PASSWORD": token,
|
||||||
|
"GIT_TERMINAL_PROMPT": "0",
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Pass `env=git_env` to **every** `run_git` call in the sequence (clone, both fetches, `rev-parse` ×2, `branch --list`, checkout, add, commit, push — `rev-parse`/`branch` may take it too; uniformity over cleverness). The clean URL stays in the argv untouched. `DocsPushError` propagation is unchanged (the error still carries the full git stderr for logs; phase 84's `sanitize_error` on the API response stays as defense-in-depth). `GIT_CURL_*` is inert for `ssh://` / local-path remotes (accepted — the remote decides).
|
||||||
|
3. `app/api/doc_drafts.py` — the push call site (L306–314) passes `user=settings.docs_user, token=settings.docs_token`. Nothing else changes (the 409 inert-switch on `docs_configured` stays repo-based — an empty token is a valid anonymous-push state, C5).
|
||||||
|
4. `tests/unit/test_git_sync.py` (or the existing `run_git` test home) — (a) default: `run_git(["git", "version"], cwd)` → `subprocess.run` called with `env=None` (byte-identical to today); (b) an explicit `env={"A": "b"}` reaches `subprocess.run` verbatim.
|
||||||
|
5. `tests/unit/test_docs_push.py` — (a) **no token** (default args) → every mocked `run_git` call gets `env=None` (the C5 byte-identity pin); (b) **token set** → every git step of a successful push gets an env with `GIT_CURL_USERNAME=user`, `GIT_CURL_PASSWORD=token`, `GIT_TERMINAL_PROMPT="0"`, **and** `PATH` present (the merge-over-`os.environ` pin — run with a controlled `os.environ` in the test); (c) whitespace-only token (`" "`) → treated as empty (the `.strip()` rule) → `env=None`; (d) a failing step still raises `DocsPushError` with git's stderr — with and without credentials (the error path is credential-agnostic).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above green (`uv run pytest tests/unit/test_git_sync.py tests/unit/test_docs_push.py -v --no-cov` — adjust file names to the existing homes if they differ).
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green — the `token.strip()` branches are covered by 5a/5b/5c).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_docs_push.py -v --no-cov` green: no-token `env=None` (all steps), token-set 3-var env merged over `os.environ` (`PATH` present, all steps), whitespace-only token = empty, `DocsPushError` propagation unchanged.
|
||||||
|
- [ ] The sync path (`clone_or_pull`) is byte-identical (it never passes `env` — verify by reading: no `env=` at its `run_git` call sites).
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (the git argv of an anonymous push is byte-identical to pre-phase; `sanitize_error` untouched).
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 04 — The one-time startup migration of a legacy `://user:pass@` repo URL
|
||||||
|
|
||||||
|
**Phase:** `173_docs_push_creds_split` · **Source:** C3 (phase-locked) — a legacy credentials-bearing `docs_repo` value (env or DB) is split once at startup into `app_settings` rows; the DB rows shadow the env; `.env` is never rewritten; idempotent; one INFO line.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Production's existing `BOR_DOCS_REPO=https://ducoterra:<token>@gitea.…` keeps working (the push still authenticates) with zero operator action, while the effective value the settings page shows becomes the clean URL and the credentials live in `docs_user` / `docs_token` (masked). A second run is a no-op; owner-set values are never clobbered.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. New module `app/core/settings_migrate.py`:
|
||||||
|
```python
|
||||||
|
"""One-time settings migrations (phase 173).
|
||||||
|
|
||||||
|
``migrate_legacy_docs_creds`` splits a legacy ``://user:pass@``
|
||||||
|
``docs_repo`` value into ``docs_repo`` (clean) + ``docs_user`` +
|
||||||
|
``docs_token`` — DB rows that SHADOW the env (the settings model's
|
||||||
|
standing rule: the database wins over the environment). ``.env`` is
|
||||||
|
never read for writing or rewritten. Idempotent: it runs only while
|
||||||
|
no ``docs_token`` override exists, so owner-set values are never
|
||||||
|
clobbered. One INFO line when it acts (AGENTS rule 10).
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
- `migrate_legacy_docs_creds() -> bool` — (1) read the `app_settings` overrides (via the overlay module's state — `load_overrides()` L93 or its backing read; use `reset_state_for_tests()` in tests); (2) the target value is the **effective** `docs_repo` (an override if present, else the env value — i.e. what `resolve_settings().docs_repo` resolves to, L239); (3) match `://([^/@:]+):([^@/]+)@` with BOTH groups non-empty — otherwise return `False` (no-op); (4) if a `docs_token` override already exists → `False` (no-op — never clobber); (5) otherwise write three rows (the overlay's existing write path — the same one the `PUT` route uses, or a direct `app_settings` insert if the overlay exposes none): `docs_repo` = the value with the `user:pass@` segment removed (e.g. `https://u:pass@host/r.git` → `https://host/r.git`), `docs_user` = user, `docs_token` = pass; log ONE INFO line: `settings: migrated embedded docs-push credentials out of docs_repo into docs_user/docs_token`; return `True`. A malformed `user:pass@` (empty user or pass) → `False`, no rows, no log (the legacy value stays as-is — the pre-phase behavior; nothing breaks, nothing half-migrates).
|
||||||
|
2. `app/main.py` — `create_app()` (L98, the composition root): after the DB is available and BEFORE `return app`, call `settings_migrate.migrate_legacy_docs_creds()` ONCE (wrap in `try/except` so a migration hiccup logs a WARNING and never blocks startup — the app boots with the pre-phase behavior in that case; the migration is a convenience, not a dependency).
|
||||||
|
3. `tests/unit/test_settings_migrate.py` — the four shapes (with the overlay state controlled via `reset_state_for_tests()` + a fake/monkeypatched override store — house style):
|
||||||
|
- (a) clean env URL, no overrides → `False`, no rows written;
|
||||||
|
- (b) legacy env `https://u:pass@host/r.git`, no overrides → `True`, three rows written (clean repo / `u` / `pass`), `resolve_settings()` resolves the clean URL from the DB;
|
||||||
|
- (c) legacy env + a `docs_token` override already present → `False`, the pre-existing rows are untouched (no clobber);
|
||||||
|
- (d) malformed `https://:@host/r.git` (empty user) → `False`, no rows, no log.
|
||||||
|
Plus: (e) the INFO line is emitted exactly once in (b) (caplog / log-capture, house style).
|
||||||
|
4. `tests/integration/test_docs_push_migrate.py` — the end-to-end shape: seed env `BOR_DOCS_REPO=https://u:pass@host/r.git` (no DB rows) → run the migration (direct call, the `create_app()` wiring is covered by the unit seam) → `resolve_settings().docs_repo == "https://host/r.git"`, `docs_user == "u"`, `docs_token == "pass"` (all from DB overrides); a second call → `False` (idempotent); then `reset_state_for_tests()` for the next test.
|
||||||
|
5. `.env.example` — the block comment added in task 01 (already covers this: "credentials in the repo URL are no longer supported — use BOR_DOCS_USER/BOR_DOCS_TOKEN"; the migration note: a legacy value is auto-split on startup into the new settings). No other file touched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_migrate.py` green (the 5 cases above).
|
||||||
|
- Integration: `tests/integration/test_docs_push_migrate.py` green.
|
||||||
|
- Coverage: **>90%** on `app/` (the gate must stay green — the no-op/act/clobber/malformed branches are all covered by 3a–3d; the `create_app()` try/except seam is covered by a unit test that monkeypatches the migration to raise).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_migrate.py tests/integration/test_docs_push_migrate.py -v --no-cov` green: the four shapes + the idempotency + the one-INFO-line pin.
|
||||||
|
- [ ] `create_app()` calls the migration once, wrapped so a failure logs a WARNING and the app still boots (unit-asserted via the monkeypatched-raise seam).
|
||||||
|
- [ ] After a (b)-shape migration, the settings API reports the clean `docs_repo` + `is_set=true` for `docs_token` with `value=null` (a quick curl or an integration assertion) — the leak is gone from the page.
|
||||||
|
- [ ] `uv run pytest` green; ruff + pyright clean.
|
||||||
|
- [ ] no behavior change in completed work (no pre-migration behavior touched; the sync path untouched).
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Task 05 — The phase E2E suite + the full gates + the atomic commit
|
||||||
|
|
||||||
|
**Phase:** `173_docs_push_creds_split` · **Source:** AGENTS rule 4 (one dedicated Playwright E2E suite per phase, run in isolation) + the test-gate protocol (AGENTS rules 8/9).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`tests/e2e/test_docs_push_creds.py` proves the leak class is dead in the real UI — the `Docs token` row is write-only, masked, never pre-filled, and a saved token never appears anywhere in the page DOM — and the phase's full gate passes: 100% of the project suite, >90% `app/` coverage, ruff + pyright, the regression E2E set, one atomic `--no-gpg-sign` Conventional Commit, and the phase dir moves to `complete/`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_docs_push_creds.py` — the phase's dedicated Playwright suite (the phase-168 `test_admin_settings_panel.py` is the template for login + the settings panel; the mock-LLM default per A16 — the suite needs no LLM at all). Tests, in order:
|
||||||
|
- **rows render:** sign in as admin → Settings → the `Docs push` group shows a `Docs repo` text row (the clean URL, or the test-seed value), a `Docs user` text row, and a `Docs token` row that is write-only — its input is `type=password`, **not pre-filled** (value `""`, rendered `(stored)`-style per the phase-168 secret-row contract), and the token text is absent from the initial DOM.
|
||||||
|
- **save stays masked:** type a unique sentinel token (e.g. `e2e-sentinel-<uuid>` from the test) into the `Docs token` row → Save → the row re-renders masked; reload the page → the row is still masked, `is_set`-state visible, and the **sentinel text is absent from the entire page DOM** (the core assertion — `page.content()` must not contain the sentinel).
|
||||||
|
- **reset unsets:** Reset the `Docs token` row → the row returns to the not-set state (no `(stored)` marker).
|
||||||
|
- **the guard, UI-side:** type a credentials-bearing URL (`https://u:pass@e2e.invalid/o/r.git`) into `Docs repo` → Save → the 422 surfaces (the phase-168 error-display contract), the row keeps the previous value, and the page DOM never shows the typed URL's `pass` segment as a *saved* value (the input echoes what was typed before save — that is fine; the assertion is on the saved/persisted state after the failed save + reload).
|
||||||
|
- Seed/cleanup: the test manages its own `app_settings` rows via the admin API (or the test DB fixture the other settings E2Es use) — no state leaks between runs.
|
||||||
|
2. **Regression set (run in isolation, house order):** `uv run pytest tests/e2e/test_admin_settings_panel.py tests/e2e/test_admin_settings_layout.py tests/e2e/test_response_to_docs.py -v --no-cov` — phase 168/171 (the settings panel) and phase 59 (the docs push) stay green.
|
||||||
|
3. **Full gates:** `uv run pytest` (100% — every test in the project, pre-existing included) · `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`) · `uv run ruff check . && uv run pyright`.
|
||||||
|
4. **Docs & consistency pass (mandatory — AGENTS.md rule 5 / phase 00):** review `README.md` and `AGENTS.md` in their entirety against the project as it now stands; fix every inconsistency found — including pre-existing ones. At minimum: the README's docs-push section documents `BOR_DOCS_USER` / `BOR_DOCS_TOKEN`, states that credentials in the repo URL are no longer supported (the PUT 422 + the startup migration), and notes the token is masked (write-only) in the settings page/API.
|
||||||
|
5. **Commit (AGENTS rule 8):** ONE atomic Conventional Commit, `--no-gpg-sign`, covering `app/`, `scripts/`, `.env.example`, `tests/`, the README fix, and this phase dir's move to `.agents/phases/complete/173_docs_push_creds_split/`. Message shape:
|
||||||
|
```
|
||||||
|
feat(settings): split docs-push credentials (BOR_DOCS_USER/BOR_DOCS_TOKEN) out of the repo URL
|
||||||
|
|
||||||
|
- docs_token is write-only (masked in the page + API, llm_api_key-style); the
|
||||||
|
docs_repo PUT guard 422s on embedded credentials
|
||||||
|
- the push authenticates via GIT_CURL_USERNAME/GIT_CURL_PASSWORD env (never in
|
||||||
|
the argv); anonymous pushes are byte-identical to before
|
||||||
|
- a one-time startup migration splits a legacy ://user:pass@ env value into the
|
||||||
|
new settings (DB rows shadow the env; .env untouched; idempotent)
|
||||||
|
|
||||||
|
LLM-Generated: true
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green **in isolation** (the phase's dedicated suite, AGENTS rule 4).
|
||||||
|
- Full suite: `uv run pytest` 100% green; `app/` coverage **>90%**; ruff + pyright clean.
|
||||||
|
- Regression: the phase 168/171/59 E2E set green in isolation.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green in isolation: rows render, the token row is `type=password` + not pre-filled, save→reload keeps it masked with the sentinel **absent from the page DOM**, reset unsets, the 422 guard surfaces.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_admin_settings_panel.py tests/e2e/test_admin_settings_layout.py tests/e2e/test_response_to_docs.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app` >90% on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full and consistent (the docs-push section documents the new vars + the guard + the migration).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/173_docs_push_creds_split/`.
|
||||||
|
- [ ] **Owner follow-up (noted, not code):** the exposed production Gitea token (seen in the 2026-10-10 audit) is rotated out-of-band; the migration makes its replacement the `docs_token` value.
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Phase 174 — Sources (git sources) page: the card stack regains its 1.25rem rhythm
|
||||||
|
|
||||||
|
**Source:** Owner bug report 2026-10-21 (screenshot of `/git-sources.html` — "the bubbles on the sources page are really smooshed together": the "Git sources" header card, the "Add a git source" card, the "Upload a source archive" card, and the sources table card stack flush — only their 1px borders separate them).
|
||||||
|
**Story:** n/a (owner bug report, this chat 2026-10-21)
|
||||||
|
**Context:** `frontend/index.html` — `#view-git-sources` (L646): `.container.git-sources-shell` holds TWO children — the anonymous `.sources-gate` (`#git-sources-gate`, L651) and the single `#git-sources-content` wrapper (L672) that wraps **every** visible card: `.page-head`, the `#git-sources-load-error` alert (hidden), the `#git-source-env-note` (hidden), `#git-source-form`, `#archive-upload-form`, and the `.table-wrap#git-sources-table-wrap`; `frontend/assets/styles.css` — `.git-sources-shell` (L3193–3200: `display:flex; flex-direction:column; gap: 1.25rem; flex:1` — the gap only separates the gate from the wrapper), `#git-sources-content` (**no rule at all** — a plain div, so its children stack with zero vertical rhythm — the root of the "smooshed" look), the phase-93 `.page-head` card (L2423–2428), the phase-27/49 form cards `#git-source-form, #archive-upload-form` (L3207+), the ≤640px block (L6400–6403: the forms go column — they compose with the fix unchanged), `frontend/assets/git-sources.js` (L294, L1515–1519: the gate↔content reveal — `contentEl.hidden` flips only); `tests/e2e/auth_helpers.py::login` (the admin E2E sign-in), `tests/e2e/test_admin_settings_layout.py` (the phase-171 bounding-box rhythm E2E — the house pattern this suite copies), `tests/unit/test_git_source_branch_ui.py` (the static CSS/HTML/JS pin style this phase extends).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The five stacked cards on the Sources page (the navbar "Sources" link → `/git-sources.html`) get one uniform 1.25rem gap between them — CSS only, reusing the shell's **existing** `gap: 1.25rem` token (the phase-171 `#settings-content` idiom applied to the one remaining admin view whose cards live inside a wrapper div). Nothing else changes: no JS, no `app/` code, no new palette/spacing value, the gate + load-error + env-note reveal lifecycles byte-identical.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the rhythm (task 01).** `#git-sources-content` becomes `display: flex; flex-direction: column; gap: 1.25rem;` — the **same** token `.git-sources-shell` already uses (L3196) and the value phase 171 gave `#settings-content`. One rule, placed directly after the `.git-sources-shell` rule with a phase-174 comment explaining the wrapper-swallowed-the-shell-gap root cause. The children (page-head card, load-error alert, env note, the two form cards, the table card) then stack with one uniform gap at every viewport.
|
||||||
|
- **D2 — the hidden children (task 01, pinned in 02).** The two `hidden` children (`#git-sources-load-error`, `#git-source-env-note`) keep the UA `[hidden] → display:none` even inside a flex parent (the parent's `display` never overrides a child's own `display`), so `git-sources.js`'s reveal lifecycle (L1515–1519) is untouched; a revealed error/note simply joins the rhythm. The anonymous gate↔content flip (L1515–1519: `contentEl.hidden = false`) works on a flex element exactly as on a block.
|
||||||
|
- **D3 — no other changes.** `.git-sources-shell` (its `flex: 1` + gap stay — the gate↔content separation at anonymous), `git-sources.js`, `app/` server code, the form/table card rules, and the ≤640px media queries (L6400–6403) are all untouched; the fix is a single wrapper rule.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `171_admin_settings_layout` (complete) — the idiom this phase reuses (wrapper div → flex column + the existing 1.25rem token).
|
||||||
|
- `93_theme_semantic_completion` (complete) — the `.page-head` card language the rhythm spaces out (context only).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_content_card_rhythm.md` — `#git-sources-content` becomes the flex-column rhythm wrapper (the one CSS rule); the static pins.
|
||||||
|
2. `02_e2e_sources_card_rhythm.md` — `tests/e2e/test_git_sources_card_rhythm.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_git_sources_card_rhythm.py` (new module, the `test_git_source_branch_ui.py` pin style): the `#git-sources-content` rule in `styles.css` with `display: flex` / `flex-direction: column` / `gap: 1.25rem` (the exact token — a regression to a different value fails), the `.git-sources-shell` gap staying `1.25rem` (unchanged), the `#git-sources-content` div present in `index.html` inside `.git-sources-shell` with the gate as a sibling (the structural pin — if the cards ever move back to the shell's direct children the rule becomes dead weight and the pin must say so).
|
||||||
|
- **E2E (AGENTS rule 4, task 02):** `tests/e2e/test_git_sources_card_rhythm.py`, run **in isolation** — the rhythm contract proven in a browser at 1440px and 375px, plus the hidden-children pin and the standard a11y/no-CDN light pass.
|
||||||
|
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's) and the gates above are green.
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (this phase is a visual fix with no setup/command/config surface — expected: no doc edits).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] On `/git-sources.html` (admin), the five cards (page-head → add form → upload form → table, plus a revealed load-error when one is forced) stack with one uniform ≈20px gap — no two card edges touch (the E2E bounding-box pins, at 1440px and 375px).
|
||||||
|
- [ ] `#git-sources-content`'s hidden children (`#git-sources-load-error`, `#git-source-env-note`) stay hidden in the flex wrapper (E2E pin); the anonymous gate still shows and the content stays hidden for anonymous (regression).
|
||||||
|
- [ ] The ≤640px mobile layout is intact (the forms column-stack per L6400–6403; the table scrolls inside its card — the E2E mobile pin).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` green **in isolation**; `test_git_sources_admin.py` + `test_archive_upload_sources.py` + `test_git_source_dates.py` green in isolation (the regression set).
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(sources): Sources (git sources) page — the card stack regains its 1.25rem rhythm"
|
||||||
|
```
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 01 — `#git-sources-content` becomes the flex-column rhythm wrapper
|
||||||
|
|
||||||
|
**Phase:** `174_git_sources_card_rhythm` · **Story:** n/a (owner bug report 2026-10-21)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The one CSS rule that restores the vertical rhythm on the Sources page: `#git-sources-content` becomes a flex column with the shell's existing `1.25rem` gap — so the five stacked cards (page-head, load-error, env-note, add form, upload form, table card) stop stacking flush.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — directly after the `.git-sources-shell` rule (L3193–3200), add the `#git-sources-content` rule with a phase-174 comment block (the house comment style: root cause + the reused token + the hidden-children behavior):
|
||||||
|
```css
|
||||||
|
/* Phase 174 (owner bug report 2026-10-21): the shell's gap only
|
||||||
|
separates the gate from this wrapper — every visible card
|
||||||
|
(page-head, the load-error alert, the env note, the two form
|
||||||
|
cards, the table card) is a child of THIS div, which was a
|
||||||
|
plain block, so the cards stacked flush (1px borders touching).
|
||||||
|
The phase-171 #settings-content idiom: the wrapper itself is
|
||||||
|
the rhythm column, reusing the shell's own 1.25rem token — no
|
||||||
|
new spacing value. The hidden children (#git-sources-load-error,
|
||||||
|
#git-source-env-note) keep the UA [hidden] display:none inside
|
||||||
|
a flex parent, so git-sources.js's reveal lifecycle is
|
||||||
|
untouched; a revealed one joins the rhythm. */
|
||||||
|
#git-sources-content {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 1.25rem;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
No other CSS change. Do not touch `.git-sources-shell`, the form/table card rules, or the ≤640px block (L6400–6403 — the forms' column-stacks compose with the new wrapper unchanged).
|
||||||
|
2. `tests/unit/test_git_sources_card_rhythm.py` — new module (the `test_git_source_branch_ui.py` pin style: `FRONTEND`/`STYLES_CSS`/`SHELL_HTML` path constants, raw text assertions, module docstring citing the phase + the owner report):
|
||||||
|
- the `#git-sources-content` rule exists in `styles.css` and carries `display: flex`, `flex-direction: column`, and `gap: 1.25rem` **exactly** (extract the rule body with a small regex over the `#git-sources-content { … }` block — a regression to any other gap value or a missing flex line fails);
|
||||||
|
- the `.git-sources-shell` rule still carries `gap: 1.25rem` (the fix must not have "fixed" the shell — D3);
|
||||||
|
- the structural pin: in `index.html`, `#git-sources-content` sits inside `.container.git-sources-shell` and the `.sources-gate` (`#git-sources-gate`) is a sibling of it (so the rule is not dead weight — if a future refactor moves the cards to the shell's direct children, this pin names the dead rule);
|
||||||
|
- `#git-sources-load-error` and `#git-source-env-note` still carry `hidden` in `index.html` (the D2 premise the E2E pin proves in a browser).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; `uv run pytest tests/unit/test_git_sources_card_rhythm.py -v` green.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged by this task — the gate must stay green, not drop).
|
||||||
|
- No `app/` code change; no JS change.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_git_sources_card_rhythm.py -v` green.
|
||||||
|
- [ ] `uv run pytest` green (the full unit + integration suite — no pin broke).
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] README.md/AGENTS.md need no change for this task (visual-only; the phase's final pass re-checks both project-wide).
|
||||||
|
- [ ] no behavior change in completed work (the gate reveal, the load-error retry, the env note, the form lifecycles are untouched — the rule only adds inter-card spacing).
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 02 — the phase's dedicated Playwright E2E suite + gates + commit
|
||||||
|
|
||||||
|
**Phase:** `174_git_sources_card_rhythm` · **Story:** n/a (owner bug report 2026-10-21)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the rhythm in a browser — `tests/e2e/test_git_sources_card_rhythm.py` (the phase's dedicated Playwright suite, AGENTS rule 4): the cards on `/git-sources.html` stack with one uniform gap at 1440px and 375px, the hidden children stay hidden in the flex wrapper, and the anonymous gate is intact. Then the regression set, the full gates, and the atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_git_sources_card_rhythm.py` — new module, run **in isolation** (`uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov`; DB must be up: `podman compose up -d db`). House shape: module docstring (the owner report + the run command + the test→contract map), the session `mock_llm`/`app_server`/`app_url`/`db_ready`/`page` fixtures from `tests/e2e/conftest.py`, `tests/e2e/auth_helpers.py::login` for the admin sign-in, a per-test `_truncate_kb`-style reset so each test's table state is its own (the `test_git_sources_admin.py` pattern — this suite is UI-only: it needs NO sync, no git, no network; the app can boot plain, the cards render regardless of row count).
|
||||||
|
Tests (four):
|
||||||
|
- `test_cards_stack_with_uniform_rhythm_desktop` — 1440×900, admin (`login(page, app_url, next="/git-sources.html")`): `#git-sources-content`'s computed `display` is `flex` and its `gap` resolves to 20px; the bounding boxes of the four visible cards — `.page-head` → `#git-source-form` → `#archive-upload-form` → `#git-sources-table-wrap` — are **pairwise disjoint** and each successive gap is ≈20px (`box_{n+1}.top − box_n.bottom == 20 ± 2`); no two card edges touch (the pre-fix regression: the gaps were ~0 — the 1px borders).
|
||||||
|
- `test_cards_stack_with_uniform_rhythm_mobile` — 375×720, admin: the same four cards are pairwise disjoint with ≈20px gaps; the two form cards column-stack per the ≤640px block (L6400–6403 — assert `#git-source-form` computed `flex-direction: column`) and the table scrolls **inside** its card (the `#git-sources-table-wrap` horizontal overflow is contained — the card's right edge is left of the viewport, no page-level horizontal scroll: `document.scrollingElement.scrollWidth <= 375`).
|
||||||
|
- `test_hidden_children_stay_hidden_in_flex_wrapper` — admin: `#git-sources-load-error` and `#git-source-env-note` are not visible (the `[hidden]` UA rule survives the flex parent — `offsetParent is None` / `is_visible() == False`); then force the load-error's `hidden` off via the page (attribute removal) and assert it **joins the rhythm** (visible, and its box sits ≥18px below the element above it) — the D2 both-sides pin; anonymous: `#git-sources-gate` visible, `#git-sources-content` hidden, no `/api/git-sources` call (the `test_git_sources_admin.py` anonymous contract, copied — the gate↔content flip is unchanged by the fix).
|
||||||
|
- `test_sources_page_a11y_and_no_cdn_basics` — the standard light pass (AGENTS rule 5/6, the `test_git_sources_admin.py` a11y test shape): landmarks on the view, the labeled form controls, the 3px `:focus-visible` outline on a table-row control (keyboard Tab), same-origin assets only (no external `<script src>`/`<link href>`).
|
||||||
|
2. **Regression set (in isolation, in order):** `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`, then `test_archive_upload_sources.py`, then `test_git_source_dates.py` (the three suites that touch the git-sources view's gate/forms/table — all must be green with the wrapper now flex).
|
||||||
|
3. **Full gates (the `.agents/validate.sh` set):** `uv run pytest --cov=app --cov-report=term-missing` (the entire project suite 100% passing + `app/` coverage >90% — this phase changes no `app/` code, the floor must hold, not drop), `uv run ruff check .`, `uv run pyright`.
|
||||||
|
4. **Final documentation & consistency pass:** review `README.md` and `AGENTS.md` in full against the project as it stands after this phase; fix every inconsistency found, **including pre-existing ones** (expected for this phase: no doc edits — visual-only).
|
||||||
|
5. **Commit + move:** the atomic commit, then move the phase dir to `.agents/phases/complete/` (the `.agents/` tree is tracked — commit it with the phase per AGENTS rule 8).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the dedicated suite green **in isolation** + the three-suite regression set green in isolation.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged — the floor holds).
|
||||||
|
- Lint/types: ruff + pyright clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` green **in isolation** (DB up).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`, `test_archive_upload_sources.py`, `test_git_source_dates.py` — each green in isolation.
|
||||||
|
- [ ] `uv run pytest --cov=app --cov-report=term-missing`: the entire project suite 100% passing, `app/` coverage >90%.
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] README.md and AGENTS.md accurate and consistent with the whole project (verified in the final pass).
|
||||||
|
- [ ] one atomic `--no-gpg-sign` Conventional Commit:
|
||||||
|
`git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(sources): Sources (git sources) page — the card stack regains its 1.25rem rhythm"`
|
||||||
|
- [ ] the phase dir moved to `.agents/phases/complete/174_git_sources_card_rhythm/`.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# Phase 175 — History page: paginate the saved-chat list
|
||||||
|
|
||||||
|
**Source:** TODO.md L3 (owner 2026-10-10): "Need pagination on the history page - it gets long fast"
|
||||||
|
**Story:** n/a (owner request, this chat 2026-10-10)
|
||||||
|
**Context:** `app/api/chats.py` — the module docstring's route list (L60–66) and `list_chats` (L175–188): the admin-only `GET /api/chats` selects **all** `SavedChat` rows (`updated_at desc, id desc`), reads `current_sources_version` once per request, and returns a `SavedChatList` (the `chats` key only); `app/schemas.py` — `SavedChatRow` (L1699) and `SavedChatList` (L1725: `chats: list[SavedChatRow]`); `frontend/assets/history.js` — `loadChats()` (fetches bare `/api/chats`, re-entrant row replacement), the `#history-refresh` binding, the `showEmptyIfLast()` / `showEmptyState()` helpers, the `#history-status` live region; `frontend/index.html` — `#view-history` (L976+): the `.page-head` (h1 + the Refresh pill), `#history-gate`, `#history-status`, and `.table-wrap#history-table-wrap` (L1044) → `table.history-table` with `#history-tbody` (a hidden `#history-empty-row` ships in the tbody); `frontend/assets/styles.css` — the house `.table-wrap` scroll region (L2836–2844: `max-height: min(70vh, 64rem)` + the sticky thead), the `.history-refresh` pill language (L3995–4018, incl. the `:disabled` opacity + the svg sizing), the ≤640px full-width block (L6471), `.history-table` (L4050+); `tests/integration/test_chats_api.py` — the list pins (L518 `test_list_empty` asserts the body byte-exactly `{"chats": []}` — must absorb the new `total` key; L524 `test_list_orders_by_updated_at_desc` — the `ROW_KEYS` set is per-row and unaffected; L557 `test_list_reports_message_count`); `tests/e2e/test_chat_history.py` (the DB-isolation idiom: distinctive titles, never assert on absolute row counts, delete created rows in a `finally`), `tests/e2e/auth_helpers.py::login`, `tests/e2e/conftest.py` (the shared `app_server` / `mock_llm` / `browser` fixtures); `tests/unit/test_git_sources_card_rhythm.py` (the static-pin style task 02 extends).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The History page's saved-chat list is paginated **server-side** — `GET /api/chats` accepts `?limit=` (default 25, clamped 1–100; out-of-range 422) and `?offset=`, and reports the full `total` on every response; the view gains a Prev / "Page N of M" / Next pager under the table that walks the pages while the re-entrant load contract, the anonymous gate, the empty state, and the Refresh behavior stay intact. The full list is no longer fetched — a 30-row history ships 25 rows plus a count.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the query (task 01).** `list_chats` takes `limit: int = Query(25, ge=1, le=100)` and `offset: int = Query(0, ge=0)` (FastAPI validation → 422 on `limit=0`, `limit=101`, `offset=-1` — the house fixed-422 contract; no hand-rolled checks). The row fetch is the EXISTING select with `.offset(offset).limit(limit)` appended (the order is untouched — `updated_at desc, id desc`). `total`: one extra `select(func.count(SavedChat.id))` per request (two small queries; the once-per-request `current_sources_version` read is untouched). `SavedChatList` gains `total: int` — **required, no default** (the `stale` convention: the API must always compute it — there is no wire shape without the count).
|
||||||
|
- **D2 — wire compatibility (task 01, proven in 03).** The change is additive: the body gains one key (`total`), and the query string is optional (a bare `GET /api/chats` = the first 25 rows + `total`). The only consumers are `history.js` and the test suites. The ONE existing pin that breaks — `test_list_empty` (L518, asserts `{"chats": []}` byte-exactly) — is updated to `{"chats": [], "total": 0}` in task 01 (the only modification to an existing test in this phase). Per-row keys are untouched (`ROW_KEYS` stays green).
|
||||||
|
- **D3 — the pager (task 02).** A `.history-pager` div in `#view-history`, directly BELOW `.table-wrap#history-table-wrap` (inside the `.history-shell` container). Contents, in order: a Prev button (the house inline-SVG chevron `aria-hidden` + the visible "Prev" label + `aria-label="Previous page of saved chats"`), a `#history-pager-status` span ("Page N of M"), a Next button (mirrored, `aria-label="Next page of saved chats"`). The buttons reuse the `.history-refresh` pill language (solid brand pill, the `:disabled` opacity rule, the svg sizing — **no new palette variable, B3**, no new spacing token). The pager is **hidden unless `total > limit`** (A4: no dead "Page 1 of 1" control; it disappears with the empty state). At the edges: Prev is `disabled` on page 1, Next `disabled` on the last page (the real `disabled` attribute — the Refresh idiom).
|
||||||
|
- **D4 — the load contract (task 02).** `loadChats(offset)` fetches `/api/chats?limit=<PAGE_SIZE>&offset=<offset>` (the module keeps `PAGE_SIZE = 25` — matching the server default; A1: no size picker in the UI), and reads `total` from the body. Module state: the current `offset` (starts at 0). Prev/Next adjust the offset by `PAGE_SIZE` (the disabled edges make clamping trivial), re-run the re-entrant load (rows are **replaced**, never appended), and land `Showing <first>–<last> of <total>.` in `#history-status` (the never-stale contract — every page change is announced). **Refresh re-fetches the *current* page** (A5 — `refreshChats` passes the module's current `offset`; it never resets to 0). The empty-state helpers (`showEmptyIfLast` / `showEmptyState`) stay unchanged — they are DOM-local. A local Delete does NOT recompute the pager (the last page's rows can vanish locally while the pager's `total` is still the server's last truth; the next load — a page click or a Refresh — corrects it). A 0-row fetch (a deleted-behind-us race) shows the empty state and `updatePager` runs on the fresh `total`.
|
||||||
|
- **D5 — anonymous (task 02, pinned in 03).** The whoami gate is unchanged: anonymous sees `#history-gate`, the table wrap hides, the pager hides (its own `hidden` default — the JS never touches it on the anonymous branch), and NO `/api/chats` request is made (the E2E pins the request log).
|
||||||
|
- **D6 — mobile (task 02).** ≤640px: the pager buttons go full width (the `.history-refresh` ≤640px idiom, L6471) — one small addition to the existing media-query block, no new values.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `174_git_sources_card_rhythm` (todo, the preceding phase)
|
||||||
|
- `50_chat_history` (complete) — the list endpoint + the history view this phase paginates
|
||||||
|
- `156_table_ux_polish` (complete) — the `.table-wrap` scroll region + the pill button language the pager reuses (context only)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_api_pagination.md` — `GET /api/chats` takes `?limit=` / `?offset=` and `SavedChatList` gains `total`; the integration pins (incl. the `test_list_empty` wire-pin update).
|
||||||
|
2. `02_pager_ui.md` — the `.history-pager` control (`index.html` + `history.js` + `styles.css`, D3–D6); the static pins.
|
||||||
|
3. `03_e2e_history_pagination.md` — `tests/e2e/test_history_pagination.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Integration:** `tests/integration/test_chats_api.py` extended (task 01) — the default page + `total`, explicit limit/offset slices, order preserved across pages (the concatenation of the pages == the full-list order), 422 on out-of-range params, an offset beyond the end → empty `chats` with the honest `total`.
|
||||||
|
- **Unit:** `tests/unit/test_history_pagination.py` (new module, task 02, the `test_git_sources_card_rhythm.py` pin style): the `#history-pager` structure in `index.html` (below `#history-table-wrap`, the button `aria-label`s, the `aria-hidden` SVGs, the shipped `hidden`), the pager rules in `styles.css` (the flex row + the ≤640px block + the token reuse — every `var(--…)` the pager rules use is already declared in `:root`), and the `history.js` contract (the parameterized fetch, the `total` read, the hidden-when-single-page branch, the disabled edges, refresh-keeps-current-page, the `Showing …` status line).
|
||||||
|
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_history_pagination.py`, run **in isolation** — the 30-row fixture (via the public `POST /api/chats`, distinctive titles, `finally` cleanup — the `test_chat_history.py` DB-isolation idiom), the page-1→2→1 walk with row counts + pager state + status lines, refresh-on-page-2 stays on page 2, delete-to-empty on the last page, the single-page hide, the anonymous gate pin (no `/api/chats` request in the log), and the standard a11y / no-CDN light pass.
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's) and the gates above are green.
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the `?limit=` / `?offset=` params + `total` are a user-visible API surface — update `README.md` only if it documents the `GET /api/chats` shape; the pager is UI with no setup/command/config surface).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/chats` (bare) returns the first 25 rows (`updated_at desc, id desc`) + `total` (the full count); `?limit=10&offset=20` returns the correct slice; out-of-range `limit` / `offset` 422; an offset beyond the end returns `{"chats": [], "total": N}` (integration pins).
|
||||||
|
- [ ] On `/history.html` (admin, a 30-row fixture): the table shows 25 rows and the pager reads "Page 1 of 2" (Prev disabled, Next enabled); Next → 5 rows, "Page 2 of 2" (Next disabled, the page-1 rows gone — replaced, not appended); Prev → back to 25 (E2E pins); every page change lands a `Showing X–Y of Z.` line in the live region.
|
||||||
|
- [ ] Refresh on page 2 re-fetches page 2 (E2E pin — never resets to page 1); deleting the last page's rows shows the empty state locally; ≤1 page → the pager is fully hidden (no dead control; E2E pin).
|
||||||
|
- [ ] Anonymous: the gate shows, the table + pager are hidden, and NO `/api/chats` request appears in the log (E2E pin).
|
||||||
|
- [ ] The ≤640px pager layout is intact (full-width buttons — the E2E mobile pin).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` green **in isolation**; `test_chat_history.py` + `test_history_page_width.py` + `test_share_chat.py` + `test_stale_saved_chats.py` + `test_history_copy.py` green in isolation (the regression set).
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(history): paginate the saved-chat list — server-side ?limit/?offset with total + the view's pager"
|
||||||
|
```
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 01 — `GET /api/chats` is paginated server-side (`?limit=`, `?offset=`, `total`)
|
||||||
|
|
||||||
|
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The list endpoint ships a page instead of the full list: 25 rows by default, a `?limit=` (1–100) / `?offset=` the client can walk, an honest `total` on every response, and a FastAPI 422 on out-of-range params — the wire change is additive everywhere (a bare `GET` = the first page + the count).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/chats.py` — `list_chats` (L175–188):
|
||||||
|
- imports: add `Query` to the `fastapi` import line; add `func` to the `sqlalchemy` import line (`from sqlalchemy import func, select, text`).
|
||||||
|
- signature: `limit: int = Query(25, ge=1, le=100)` and `offset: int = Query(0, ge=0)` (declared before the `db` dependency; FastAPI serves the house field-message 422 for `limit=0` / `limit=101` / `offset=-1` — no hand-rolled validation).
|
||||||
|
- body: the existing select gains `.offset(offset).limit(limit)` appended (the order is untouched — `SavedChat.updated_at.desc(), SavedChat.id.desc()`); `total = db.scalar(select(func.count(SavedChat.id)))` (one small count query per request — the once-per-request `current_sources_version` read is untouched).
|
||||||
|
- return `SavedChatList(chats=[_to_row(row, current_version) for row in rows], total=total)`.
|
||||||
|
- docstrings: `list_chats`' docstring + the module docstring's route list (L60–66, the "``GET`` (list, …)" entry) — describe the two params + `total` in the house thoroughness (the 422 contract, the default, the once-per-request count read).
|
||||||
|
- NO other route in this file changes (detail / create / re-Save / delete / share / unshare / the public read + page route are untouched).
|
||||||
|
2. `app/schemas.py` — `SavedChatList` (L1725):
|
||||||
|
- add `total: int` — **required, no default** (the `stale` convention: the API must always compute it — there is no wire shape without the count); update the class docstring (`chats` + `total` described).
|
||||||
|
- `SavedChatRow` and its per-row key set (the `ROW_KEYS` in the integration suite) are untouched.
|
||||||
|
3. `tests/integration/test_chats_api.py` —
|
||||||
|
- **update the ONE existing pin that breaks:** `test_list_empty` (L518): `assert r.json() == {"chats": []}` becomes `assert r.json() == {"chats": [], "total": 0}` (the D2 wire-pin update — the body is now the two-key shape, byte-exact).
|
||||||
|
- new pins (the file's existing idiom: the `admin_client` fixture, direct `SavedChat` ORM adds with staggered `updated_at` — the L524 idiom — or `POST /api/chats` creates; distinctive titles):
|
||||||
|
- `test_list_default_page_is_25_and_reports_total` — 30 rows with distinct, staggered `updated_at` → bare `GET /api/chats`: exactly 25 rows, `total == 30`, and the first page's `id` order == the first 25 of the full ordered list (`updated_at desc`);
|
||||||
|
- `test_list_limit_offset_slice` — the same 30 rows → `GET /api/chats?limit=10&offset=20`: exactly 10 rows (the last 10 of the ordered list — pinned by title), `total == 30`;
|
||||||
|
- `test_list_order_preserved_across_pages` — `?limit=10` at offsets 0 / 10 / 20, the three pages concatenated == the full ordered `id` sequence (no duplicates, no gaps);
|
||||||
|
- `test_list_rejects_out_of_range_params` — `?limit=0`, `?limit=101`, `?offset=-1` each 422 (the house fixed-422 contract; one request per case, status asserted);
|
||||||
|
- `test_list_offset_beyond_total_returns_empty_with_total` — `?offset=100` on the 30-row fixture → `{"chats": [], "total": 30}` (an honest count, not an error).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the new pins + the updated `test_list_empty`; `uv run pytest tests/integration/test_chats_api.py -v` green.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The bare `GET /api/chats` returns 25 rows + `total`; the explicit `limit` / `offset` slices are correct; out-of-range params 422; an offset beyond the end is honest.
|
||||||
|
- [ ] `test_list_empty` updated to the two-key body (and green); every other existing pin green **unchanged** (per-row shape byte-exact — `ROW_KEYS`).
|
||||||
|
- [ ] `uv run pytest` green (the full unit + integration suite); `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] README.md: update the endpoint's shape only if `README.md` documents `GET /api/chats` (check first; if absent, no edit) — AGENTS.md needs no change (no setup/command/config surface).
|
||||||
|
- [ ] No behavior change in completed work (create / detail / re-Save / delete / share / unshare / the public read + page route untouched — the list route's only diff is the two query params + the `total` key + the select's `.offset()/.limit()`).
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Task 02 — the History view's pager (Prev / "Page N of M" / Next)
|
||||||
|
|
||||||
|
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The view walks the paginated list: the pager control under the table (D3), the parameterized re-entrant load (D4), hidden-when-single-page / hidden-when-empty (A4), refresh-keeps-current-page (A5), the untouched anonymous gate (D5), the ≤640px block (D6) — all in existing tokens and house idioms.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — in `#view-history` (L976+), **directly after** the `.table-wrap#history-table-wrap` div (L1044 — the table card is the view's last child today; the pager lands below it, inside the same `.history-shell` container), with a phase-175 comment (the house comment style — the reason + the idioms it reuses):
|
||||||
|
```html
|
||||||
|
<!-- Phase 175 (TODO.md L3): the list pager — visible only when
|
||||||
|
there is more than one page (total > the 25-row page size —
|
||||||
|
history.js toggles `hidden`), so a one-page list never shows a
|
||||||
|
dead control. The buttons reuse the .history-refresh pill
|
||||||
|
language (the house inline-SVG glyph + visible label +
|
||||||
|
aria-label; the ≤640px full-width block joins
|
||||||
|
.history-refresh). #history-pager-status carries "Page N of M";
|
||||||
|
page-change outcome lines land in #history-status (the existing
|
||||||
|
live region). -->
|
||||||
|
<div class="history-pager" id="history-pager" hidden>
|
||||||
|
<button type="button" class="history-pager-btn" id="history-pager-prev" aria-label="Previous page of saved chats">
|
||||||
|
<svg class="history-pager-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M15 18l-6-6 6-6"/></svg>
|
||||||
|
<span class="history-pager-label">Prev</span>
|
||||||
|
</button>
|
||||||
|
<span class="history-pager-status" id="history-pager-status"></span>
|
||||||
|
<button type="button" class="history-pager-btn" id="history-pager-next" aria-label="Next page of saved chats">
|
||||||
|
<span class="history-pager-label">Next</span>
|
||||||
|
<svg class="history-pager-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M9 18l6-6-6-6"/></svg>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
```
|
||||||
|
The anonymous gate / page-head / `#history-status` span / table markup are all untouched.
|
||||||
|
2. `frontend/assets/history.js` —
|
||||||
|
- module header comment: a phase-175 block (the house comment style — the parameterized fetch + `total` + the pager lifecycle).
|
||||||
|
- element lookups (scoped to `root`, the house style): `#history-pager`, `#history-pager-prev`, `#history-pager-next`, `#history-pager-status`.
|
||||||
|
- state: a `const PAGE_SIZE = 25;` (matches the server default — A1, no size picker in the UI; the comment says so, and the E2E suite catches drift) + `let offset = 0;` (the current page's offset).
|
||||||
|
- `loadChats(pageOffset)`: the fetch becomes `/api/chats?limit=` + `PAGE_SIZE` + `&offset=` + `pageOffset` (manual string join — the house style in this file); on 200, `const { chats, total } = await r.json();` then `updatePager(total)`. The 0-row / non-2xx / network-error branches keep their exact current behavior (empty state + the house error lines; a failed load leaves the pager as it was — the previous state is the last known truth).
|
||||||
|
- `updatePager(total)`: `const pages = Math.ceil(total / PAGE_SIZE);` — if `total <= PAGE_SIZE` → the pager is `hidden` (A4: no "Page 1 of 1" dead control); otherwise the pager shows, `#history-pager-status` gets `Page ` + (`offset / PAGE_SIZE + 1`) + ` of ` + `pages`, `#history-pager-prev.disabled = (offset === 0)`, `#history-pager-next.disabled = (offset + PAGE_SIZE >= total)`.
|
||||||
|
- Prev/Next binding (ADMIN branch only — the same gate as Refresh; the anonymous branch never binds them, D5): click → `offset += / -= PAGE_SIZE` (the disabled edges make clamping trivial) → `loadChats(offset)` → on success land `Showing ` + (`offset + 1`) + `–` + `Math.min(offset + PAGE_SIZE, total)` + ` of ` + `total` + `.` in `#history-status` (the existing live region — the never-stale contract).
|
||||||
|
- `refreshChats()`: pass the module's current `offset` to `loadChats` (A5 — re-fetch the *current* page; never reset to 0).
|
||||||
|
- boot: the initial `loadChats(0)` — the view opens on page 1.
|
||||||
|
- a local Delete does NOT touch the pager or `offset` (D4 — the next load corrects it).
|
||||||
|
3. `frontend/assets/styles.css` —
|
||||||
|
- the `.history-pager` rule (placed near the `.history-refresh` block, L3995–4018, with a phase-175 comment — root cause + token reuse): `display: flex; align-items: center; gap: 0.75rem;` (the existing `0.75rem` gap the page already uses — no new spacing token). The buttons: add `.history-pager-btn` to the existing `.history-refresh` declaration (a selector union on the pill rule, the `:disabled` rule, and the `svg` sizing rule — the executor's choice of union vs duplication, but **no new palette variable (B3)** and no value not already in the file); `#history-pager-status` needs no rule beyond the inherited text (if a min-width or text-center is added, it uses existing tokens only).
|
||||||
|
- the ≤640px media-query block (L6471): add `.history-pager-btn` to the existing `.history-refresh` full-width line (the house idiom).
|
||||||
|
- NOTHING else in styles.css changes (the `.table-wrap` scroll region, the sticky thead, the `.history-table` rules, the `.history-status` rule are all untouched).
|
||||||
|
4. `tests/unit/test_history_pagination.py` — new module (the `test_git_sources_card_rhythm.py` pin style: `FRONTEND` / `STYLES_CSS` / `SHELL_HTML` path constants, raw text assertions, a module docstring citing the phase + the TODO line):
|
||||||
|
- `index.html`: `#history-pager` sits in `#view-history` and AFTER `#history-table-wrap` (the structural pin — the pager is under the table card, not in the page-head); both buttons carry their exact `aria-label`s; both SVGs carry `aria-hidden="true"`; the div ships `hidden` (the hidden-when-single-page default the JS flips);
|
||||||
|
- `styles.css`: the `.history-pager` rule with `display: flex` and `gap: 0.75rem`; `.history-pager-btn` is covered by the pill / `:disabled` / svg-sizing declarations (extract with the small-regex house style — pinned by presence, not by a particular CSS authoring choice); the ≤640px block carries `.history-pager-btn` full width; **no new token** — every `var(--x)` used in any pager rule is declared in the file's `:root` block (the B3 pin);
|
||||||
|
- `history.js`: the parameterized fetch URL (`/api/chats?limit=` + the `&offset=` join), the `total` read from the JSON body, `PAGE_SIZE = 25`, the hidden-when-single-page branch (`total <= PAGE_SIZE`), the two `disabled` edge lines, the `Showing ` status-line format, the refresh path passing the current `offset`, and the pager lookups scoped to `root`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; `uv run pytest tests/unit/test_history_pagination.py -v` green.
|
||||||
|
- Coverage: **>90%** on `app/` (this task changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- The E2E proof is task 03 (the browser-level pins); this task's pins keep the static contract honest between runs.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_history_pagination.py -v` green.
|
||||||
|
- [ ] `uv run pytest` green (the full unit + integration suite — no pin broke).
|
||||||
|
- [ ] ruff + pyright clean; no new CSS token (B3 pin green).
|
||||||
|
- [ ] README.md/AGENTS.md need no change for this task (UI with no setup/command/config surface; the phase's final pass re-checks both project-wide).
|
||||||
|
- [ ] No behavior change in completed work (the anonymous gate, the Refresh lifecycle, the empty-state helpers, the Delete/Share row actions are untouched — the load only gains two query params + `total` + the pager state).
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Task 03 — E2E (Playwright): the History page's pager
|
||||||
|
|
||||||
|
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's dedicated E2E suite (AGENTS rule 4 — one file per phase, run in isolation): `tests/e2e/test_history_pagination.py` proves the pager contract in a real browser against the real app + DB, then the regression set, the full gates, and the atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_history_pagination.py` — new file (the `test_chat_history.py` house pattern: a module docstring citing the phase + the TODO line + the run-in-isolation instructions `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` (DB up: `podman compose up -d db`); the `conftest` fixtures — `app_server` / `page`; `auth_helpers.login` for the admin cookie). The suite touches ONLY `saved_chats` (no KB reset idiom needed — it never drives the chat page):
|
||||||
|
- **fixture helper** (the `test_chat_history.py` DB-isolation idiom — distinctive titles, never assert on absolute row counts, delete the rows it creates in a `finally` via admin `DELETE /api/chats/<id>`): `make_chats(n)` — POST `/api/chats` (the public write surface) `n` conversations, each with a distinctive auto-title (e.g. `Pagination probe <i> of <n> — what is the capital of France?`) and a minimal `messages` payload (the house `_simple_conversation` shape); then, to make the `updated_at desc` order fully deterministic, re-Save (`PUT /api/chats/<id>` — public, bumps `updated_at` via `onupdate`) the rows in a controlled loop, so the last re-Saved chat is row 1. Pin the order by the first row of page 1 (the last re-Saved) and the first row of the last page (the earliest re-Saved) — never by the full middle order.
|
||||||
|
- `test_pager_shows_page_1_of_2_for_30_rows` — admin, `/history.html`: the table holds 25 rows, the pager is visible, `#history-pager-status` reads "Page 1 of 2", `#history-pager-prev` is `disabled`, `#history-pager-next` is enabled, `#history-status` carries "Showing 1–25 of 30.";
|
||||||
|
- `test_next_walks_to_page_2` — click Next: the table holds 5 rows (pinned by the fixture's earliest-re-Saved title first — the order), "Page 2 of 2", Next disabled, Prev enabled, the live region reads "Showing 26–30 of 30."; the 25 page-1 rows are GONE (replaced, never appended — the row count is exactly 5);
|
||||||
|
- `test_prev_walks_back_to_page_1` — from page 2, click Prev: 25 rows again, "Page 1 of 2", Prev disabled, "Showing 1–25 of 30.";
|
||||||
|
- `test_refresh_on_page_2_stays_on_page_2` — walk to page 2, click `#history-refresh` (wait for its re-enable), still 5 rows + "Page 2 of 2" (A5 — never reset to page 1), and the "Saved chats refreshed." line lands;
|
||||||
|
- `test_delete_all_rows_of_page_2_then_prev_single_page_hides_pager` — on page 2 (5 rows), use the inline two-step confirm (the `Delete? [Yes] [No]` pair appears — the owner-locked no-native-dialog pin) to delete all five: `#history-empty-row` becomes visible (the local delete contract — the pager is NOT recomputed on a local delete, D4); then click Prev: 25 rows, and the pager is now HIDDEN (total 25 → single page, A4 — the fresh `total` from the page-1 load corrects it);
|
||||||
|
- `test_single_page_hides_the_pager` — a 10-row fixture: the table lists all 10, the pager is fully hidden (A4 — no dead control);
|
||||||
|
- `test_anonymous_sees_gate_and_never_fetches` — no login: `#history-gate` visible, `#history-table-wrap` + `#history-pager` hidden, and NO `GET /api/chats` in the request log (the house log-pin idiom);
|
||||||
|
- `test_pager_mobile_375px` — viewport 375px: the pager buttons are full width (bounding-box pin — the phase-171 rhythm-suite idiom) and the table still scrolls horizontally inside its card;
|
||||||
|
- the standard a11y light pass (the house pattern from the phase-171/167 suites): the pager buttons' `:focus-visible` outline, the `aria-label`s exposed in the accessibility tree, the pager text contrast via the existing tokens (computed-style spot check), and the no-CDN pin (no external-origin `script`/`link` — the house integration check on the index page).
|
||||||
|
2. the regression set (green in isolation, UNCHANGED — task 01's `test_list_empty` pin update is the only touch to an existing test in this phase):
|
||||||
|
- `uv run pytest tests/e2e/test_chat_history.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_history_page_width.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_share_chat.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_stale_saved_chats.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_history_copy.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/integration/test_chats_api.py -v`
|
||||||
|
3. the full gates:
|
||||||
|
- `uv run pytest` (the entire unit + integration suite — 100% green)
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` (app/ >90%)
|
||||||
|
- `uv run ruff check . && uv run pyright`
|
||||||
|
- the Documentation & Consistency final pass (README.md + AGENTS.md against the project as it stands — fix every inconsistency found, including pre-existing ones).
|
||||||
|
4. the atomic commit (the phase dir moves to `.agents/phases/complete/` **after** the commit):
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(history): paginate the saved-chat list — server-side ?limit/?offset with total + the view's pager"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the suite above green **in isolation**; the regression set green in isolation; the full gates green.
|
||||||
|
- The phase is "Complete" only when every test in the project passes 100% and the gates are green (the 00_phase.md contract).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] The five-file E2E regression set + the integration `test_chats_api.py` green (unchanged apart from task 01's wire-pin update).
|
||||||
|
- [ ] `uv run pytest` 100% green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] README.md + AGENTS.md verified consistent (the final pass).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Phase 176 — Admin settings page: a usable Value-column width on mobile
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields." (the TODO's `image-1.png` screenshot: at a 375 px phone viewport the Value-column inputs render only 2–3 characters wide — "Br", "0.", "pr", "Wr", "Pc", "IN" — unreadable and unusable)
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-12)
|
||||||
|
**Context:** `frontend/index.html` — `#view-settings` (L1519), the `.table-wrap` card (L1596) holding `table.settings-table` with the static `thead` (L1599–1606: the Setting / Value / Source / Actions `th`s — the Value `th` at L1602 carries no class), `frontend/assets/settings.js` (the row render — `makeRow` builds `td.settings-name-cell` + `td.settings-value-cell` (the `.settings-control` control) + the Source badge cell + the Reset cell; the controls are type-correct per item and carry `width: 100%; min-width: 0`), `frontend/assets/styles.css` — `.settings-table` (L4742–4746: `width: 100%; min-width: 560px;` — the ONLY width floor, no per-column widths — the root cause: at 375 px the table renders at its 560 px floor and the Setting cell's long `white-space: nowrap` labels (L4754: `td { white-space: nowrap }`) devour it, collapsing the `min-width: 0` Value cell), the `.table-wrap` bounded-scroll card (L2836–2842: `overflow: auto; max-height: min(70vh, 64rem)` — horizontal scroll is the established contract; the ≤640px block's own comment: "the table's own horizontal scroll covers the columns"), the ≤640px mobile block (L6180–6529) with the existing phase-168 settings squeeze rules (L6484–6488), `tests/unit/test_settings_frontend_view.py` (the phase-168/171 static pin set, incl. `test_styles_css_settings_block_uses_existing_tokens_only` L516), `tests/e2e/test_admin_settings_layout.py` (phase 171 — the "no horizontal page overflow at 375px / 1440px" pin L482–486) and `tests/e2e/test_admin_settings_panel.py` (phase 168 — the regression set), PLAN.md §7.1 (≥44px touch targets on mobile; lists/tables ≥80–90% of container; the table scrolls INSIDE its card).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
On a phone, the Settings table's Value inputs are impossibly small (2–3 chars). The owner's direction: the table is already horizontally scrollable, so the inputs should be **slightly wider** — i.e. give the Value column a usable minimum width (and raise the table's scroll floor to match) so every `.settings-control` is readable and editable on a 375 px phone, absorbed by the card's existing horizontal scroll. The fix is **CSS-only, mobile-scoped** (the ≤640px block) — the desktop layout stays byte-identical, no `app/` code, no JS logic change.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the Value-column floor (task 01, owner-locked A2).** Inside the existing `@media (max-width: 640px)` block (L6180, appended beside the phase-168 settings squeeze rules L6484–6488), two rules:
|
||||||
|
- `.settings-table { min-width: 720px; }` — the mobile scroll floor (was 560 px, which the nowrap Setting labels exhaust). At a 375 px viewport the table renders at 720 px and scrolls horizontally INSIDE the `.table-wrap` card (overflow: auto) — the page itself never overflows (the phase-171 pin holds: the scroll is the card's, not the page's).
|
||||||
|
- `.settings-value-cell, .settings-value-th { min-width: 13rem; }` — the Value column (and its control, `width: 100%`) gets a 13 rem (≈208 px ≈ 14 mono chars at the control's 0.85 rem) floor at the phone. `13rem` is a plain length value — NO new CSS custom property (the house token rule B3 is about palette variables; a bare length literal in the mobile block is the same idiom as the existing `#settings-save { width: 100% }` rules there).
|
||||||
|
- To make the column floor deterministic (auto table layout can otherwise leave the `th` unconstrained), the Value `th` in `frontend/index.html` L1602 gains the class: `<th scope="col" class="settings-value-th">Value</th>` — static HTML (the no-HTML-string house rule binds the JS, not the templates; `settings.js` is untouched).
|
||||||
|
- **D2 — mobile-only scope (owner-locked A1).** Every new rule lives INSIDE the `@media (max-width: 640px)` block only. At ≥641 px (the 1440 px E2E desktop pin, and the 72 rem container ≈ 1152 px) the media query is inert: the table renders at card width as today, the Value column takes its natural auto share, and the desktop bytes of `styles.css`/`index.html` behavior are unchanged.
|
||||||
|
- **D3 — no other change (task 01).** `settings.js` (render / filter / save / reset / clear-all), the `.table-wrap` bounded-scroll + sticky-th contract (L2836–2848), the phase-171 `#settings-content` rhythm, the group bands, the Source badges, the mobile action-row squeeze rules (L6484–6488), `app/` server code — all byte-identical. Nothing animates (nothing for `prefers-reduced-motion` to still).
|
||||||
|
- **NOT touched:** `app/`, `settings.js`, the ≥641 px rules in `styles.css`, the other catalogue tables.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `171_admin_settings_layout` (complete) — the Settings page rhythm/band/cursor layout this phase extends, and the ≤640px mobile block's existing settings squeeze rules (L6484–6488) the new rules are appended beside.
|
||||||
|
- `168_admin_settings_panel` (complete) — the panel contract (the gate, the catalogue, the `.table-wrap` scroll card, the §7.4 Save lifecycle) that must stay behavior-identical.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_mobile_value_column_width.md` — the Value-column `min-width: 13rem` + the table `min-width: 720px` floor, both inside the ≤640px block; the `settings-value-th` class on the Value `th`.
|
||||||
|
2. `02_unit_pins.md` — `tests/unit/test_settings_frontend_view.py`: the static pins for the two mobile rules (present ONLY inside the ≤640px block) + the `settings-value-th` class pin on the static thead; the existing token-only pin stays green.
|
||||||
|
3. `03_e2e_mobile_value_width.md` — `tests/e2e/test_settings_mobile_value_width.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_settings_frontend_view.py` (the phase-168/171 static pin set, extended in task 02): the two new rules parse out of the ≤640px block with the exact values, the `settings-value-th` class is present on the settings `thead`'s Value `th`, and `test_styles_css_settings_block_uses_existing_tokens_only` stays green (no new palette variable — a bare length is not a token).
|
||||||
|
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_settings_mobile_value_width.py`, run **in isolation** — the width contract, proven in a browser at 375 px and 1440 px.
|
||||||
|
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] At a 375 px viewport, every Settings `.settings-control` (the Value inputs) is ≥ 12 rem (≈192 px, the A3 pin threshold) wide — readable and editable (the E2E width pin).
|
||||||
|
- [ ] At 375 px the `.table-wrap` card scrolls horizontally (`scrollWidth > clientWidth`) while the PAGE does not overflow horizontally (the phase-171 pin at 375 px and 1440 px stays green).
|
||||||
|
- [ ] At a 1440 px viewport the Value input width is unchanged vs. pre-phase (the mobile rules are inert — no desktop regression; the phase-171 E2E suite stays green in isolation).
|
||||||
|
- [ ] The keyboard tab order still reaches filter → first row input → **Save settings** → **Clear all overrides** at 375 px (the phase-168/171 contract under the new column floor); the whole test issues ZERO external requests (rule 6).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` green **in isolation**; `test_admin_settings_layout.py` (phase 171) + `test_admin_settings_panel.py` (phase 168) green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-12, roadmap confirmation)
|
||||||
|
- **A1 — mobile-only scope.** The new width rules live ONLY in the existing `@media (max-width: 640px)` block; the desktop (≥641 px) layout stays byte-identical — the TODO is mobile-scoped.
|
||||||
|
- **A2 — the width values.** Table floor **720 px** (was 560 px — the phone scrolls the card horizontally, the established contract); Value column **min-width 13rem** → the input ≈ 208 px ≈ 14 mono characters, i.e. "slightly wider input fields". No new CSS custom property, no JS change.
|
||||||
|
- **A3 — the E2E pin threshold is 12 rem** (192 px) — a deliberate margin below the 13 rem (208 px) rule so the pin doesn't become brittle to sub-pixel rounding.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): mobile settings table — a usable Value-column width (the card's horizontal scroll absorbs it)"
|
||||||
|
```
|
||||||
+28
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — the mobile Value column gets a usable width (the card's horizontal scroll absorbs it)
|
||||||
|
|
||||||
|
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
At a 375 px phone viewport the Settings table's Value inputs render 2–3 characters wide (the Setting cell's long `white-space: nowrap` labels exhaust the table's 560 px floor). Give the Value column a **13 rem floor** and raise the table's mobile scroll floor to **720 px** — both inside the existing `@media (max-width: 640px)` block, so the phone scrolls the card horizontally (the established contract) while every `.settings-control` is readable and editable. CSS only, plus the one `th` class that makes the column floor deterministic; the desktop (≥641 px) stays byte-identical (owner-locked A1).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` L1602 — the static `thead`'s Value cell: `<th scope="col">Value</th>` → `<th scope="col" class="settings-value-th">Value</th>`. Static HTML only (the no-HTML-string house rule binds the JS — `settings.js` stays byte-unchanged); the class exists so the column floor constrains the header cell, not just the `td`s, under auto table layout.
|
||||||
|
2. `frontend/assets/styles.css` — inside the existing `@media (max-width: 640px)` block (opens L6180), appended directly after the phase-168 settings squeeze rules (L6484–6488: `.settings-search` / `.settings-actions` stack + `#settings-save` / `#settings-clear-all` full width + `.settings-actions-cell` wrap), add ONE comment + two rules (the phase-176 header comment, stating the TODO L3 contract: the Value column's 13 rem floor makes the input ≈208 px ≈ 14 mono chars at the phone; the 720 px table floor + the card's `overflow: auto` (L2836) mean the phone scrolls the CARD, never the page):
|
||||||
|
- `.settings-table { min-width: 720px; }`
|
||||||
|
- `.settings-value-cell, .settings-value-th { min-width: 13rem; }`
|
||||||
|
No other rule moves. `13rem` / `720px` are bare length literals — NO new CSS custom property (B3 binds palette variables; a length in the mobile block is the idiom the neighboring `width: 100%` rules already use).
|
||||||
|
3. `frontend/assets/styles.css` L4742–4746 — update the `.settings-table` comment to note the mobile block (≤640px) raises the floor to 720 px for the Value column (one line; the rule itself stays here at 560 px for ≥641 px viewports).
|
||||||
|
|
||||||
|
- ASSUMPTION: A1 (mobile-only scope — the rules live ONLY in the ≤640px block; desktop byte-identical), A2 (the width values — 720 px table floor, 13rem Value floor) — both owner-locked 2026-10-12 in the phase-176 roadmap confirmation. No runtime guesses.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_frontend_view.py` — the existing pins (incl. `test_styles_css_settings_block_uses_existing_tokens_only` L516) must stay green; the task-02 pins cover the new rules.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green, not drop).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] In the ≤640px block of `styles.css`, the two new rules are present with the exact values (720 px / 13 rem) and NO `settings-value` rule appears outside that block (grep: the selectors occur only once, inside `@media (max-width: 640px) … }`).
|
||||||
|
- [ ] `frontend/index.html` L1602 carries `class="settings-value-th"`; `settings.js` is byte-unchanged (`git diff --stat` shows only `frontend/index.html` + `frontend/assets/styles.css` + the test pin).
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
|
||||||
|
- [ ] No `app/` change; the ≥641 px rules are byte-unchanged.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — unit pins: the two mobile width rules + the `settings-value-th` class
|
||||||
|
|
||||||
|
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Pin task 01's CSS in `tests/unit/test_settings_frontend_view.py` the house way (the phase-168/171 static-pin idiom — `re.search` the rule body, assert the declarations): the two new rules exist with their exact values **inside the ≤640px block only**, and the `settings-value-th` class is on the settings `thead`'s Value `th`. The existing pins — incl. `test_styles_css_settings_block_uses_existing_tokens_only` (L516) — stay green: the new rules add a bare length, no `:root` custom property (B3 holds).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_settings_frontend_view.py` — a new section `# ---------- phase 176 (task 01): the mobile Value-column floor ----------` after the phase-171 section (after `test_settings_save_disabled_is_just_disabled_no_wait_cursor`, L714), with ONE test function `test_settings_mobile_value_column_floor` (the house one-test-per-contract idiom; the function docstring cites `TODO.md` L3 + owner-locked A1/A2, the way the phase-171 docstrings cite their TODO line):
|
||||||
|
- Extract the ≤640px block: `mobile = re.search(r"@media \(max-width: 640px\) \{([\s\S]*?)\n\}\n\n/\* Phase 46", css)` is too brittle — instead pin the rules by SELECTOR SCOPING: assert `css.count(".settings-value-cell, .settings-value-th")` is **exactly 1** and that the occurrence sits INSIDE the ≤640px block (locate the `@media (max-width: 640px)` opening at L6180 and the block's closing `}` at the end of the media region — the test may slice `css[mobile_open:media_close]` with the open/close markers; the phase-171 pin `test_settings_content_is_a_flex_column_with_the_shell_gap` (L589) already sets the rule-extraction idiom to follow).
|
||||||
|
- In the mobile block: the `.settings-table` rule declares `min-width: 720px` (the mobile floor — the base L4742 rule keeps `min-width: 560px` for ≥641 px; pin BOTH: the base `.settings-table` rule body contains `min-width: 560px` and NO `720px`, the mobile-block `.settings-table` rule body contains `min-width: 720px`).
|
||||||
|
- In the mobile block: the Value rule declares `min-width: 13rem` on both `.settings-value-cell` and `.settings-value-th`.
|
||||||
|
- `index.html` (via the file read the settings-view pins already use, `_read(FRONTEND / "index.html")`): the settings `thead` (the `table.settings-table` region — the `#settings-tbody` anchor) contains `<th scope="col" class="settings-value-th">Value</th>`.
|
||||||
|
2. `tests/unit/test_settings_frontend_view.py` — verify `test_styles_css_settings_block_uses_existing_tokens_only` (L516) still passes UNCHANGED (the new rules are inside the media block, not the settings block's `:root` scan scope; if its no-new-custom-property scan is a global grep for `--settings-` or similar, confirm it does not match `13rem`/`720px` — it cannot, they are bare lengths; run it and record the pass).
|
||||||
|
|
||||||
|
- ASSUMPTION: the media-block extraction uses open/closing markers (not a hardcoded line range) so the pin survives the two-line insertion above it — the phase-171 docstring's "the 640px mobile stacking composes with it" contract is the same robustness bar. (Not an owner decision: test-internal mechanics.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_settings_frontend_view.py` — the new pin + all existing settings pins green.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green, incl. the new `test_settings_mobile_value_column_floor` and the untouched `test_styles_css_settings_block_uses_existing_tokens_only`.
|
||||||
|
- [ ] The pin fails if either mobile rule is removed or moved out of the ≤640px block (mutation check: temporarily drop the rule → red → restore).
|
||||||
|
- [ ] `frontend/` and `app/` are byte-unchanged by this task (the diff is `tests/unit/test_settings_frontend_view.py` only).
|
||||||
|
- [ ] ruff + pyright clean.
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 03 — E2E: the mobile Value-column width contract + regression + gates + commit
|
||||||
|
|
||||||
|
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the width contract in a real browser (AGENTS rule 4 — the phase's dedicated Playwright suite, run **in isolation**), prove the desktop is unchanged, run the regression set + full gates, and commit the phase atomically.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_settings_mobile_value_width.py` — the phase suite, structured on `tests/e2e/test_admin_settings_layout.py` (phase 171) VERBATIM where the contract allows (module docstring citing `TODO.md` L3 + the test→contract mapping; the `_boot_app` / `app_server` fixture with `env["BOR_SETTINGS_DB"] = "1"`; the per-test `_clean_settings_state` truncation of `app_settings` / `ui_settings` + the `e2e-`-labeled token cleanup — never a TRUNCATE on `api_tokens`; the module app env block; `login` from `e2e.auth_helpers`; `expect` from `playwright.sync_api`). Tests (each at the stated viewport via `page.set_viewport_size({"width": …, "height": 900})`):
|
||||||
|
- **`test_mobile_value_inputs_are_usable_width`** — at **375 px**: the admin arrives at the settings view (the phase-168 gate flow); for a representative spread of rows (at least one `type="search"`/text input, one `select`, one textarea — the catalogue is 75 rows; pick three distinct control types), the `.settings-control` bounding-box width is **≥ 192 px** (12 rem — owner-locked A3, the deliberate margin under the 13 rem rule) AND ≥ 200 px wide is NOT asserted (the 12 rem pin is the contract; 13 rem ≈ 208 px is the rule). Also the `.settings-table`'s own width (`scrollWidth`) **exceeds** the `.table-wrap` card's `clientWidth` (the horizontal scroll is active on the phone — the TODO's own premise) while `document.documentElement.scrollWidth ≤` the 375 px viewport (the PAGE does not overflow — the phase-171 pin, the scroll is the card's).
|
||||||
|
- **`test_mobile_table_still_scrolls_and_page_does_not_overflow`** — at **375 px**: `.table-wrap` computed `overflow` is `auto`; the card's `scrollWidth > clientWidth`; the page-level `documentElement.scrollWidth` ≤ 375 px + 1 px rounding; the sticky `thead` contract survives (the first row's group band/`thead` is reachable — scroll the card and the `thead` `th`s stay pinned, computed `position: sticky`).
|
||||||
|
- **`test_desktop_value_width_is_unchanged`** — at **1440 px**: a fixed probe row (the same first `type="search"` control) has a width STRICTLY GREATER than at 375 px is the wrong shape — the contract is the media query is INERT: the 1440 px `.settings-control` width equals the pre-phase natural layout; pinned as: the mobile rules do not apply (computed `min-width` of `.settings-value-cell` at 1440 px is the UA `0px`/auto — i.e. `getComputedStyle` returns `0px` or `auto`, never `208px`/`208.0px`) and the `.settings-table` `scrollWidth` ≤ the card's `clientWidth` (no forced scroll at desktop — the 720 px floor binds only inside the media query).
|
||||||
|
- **`test_keyboard_order_and_no_external_requests`** — at **375 px**: the tab order reaches `#settings-filter` → the first row's Value control → `#settings-save` → `#settings-clear-all` (the phase-168/171 contract under the new column floor); the whole test's `page.on("request")` log contains ZERO external (non-127.0.0.1) requests (rule 6).
|
||||||
|
- **The DB-isolation docstring + module teardown** (the phase-171 idiom): the suite's only write is none (no save is issued — the width contract needs no dirty row); `app_settings` pinned empty at teardown.
|
||||||
|
2. Run the **regression set** in isolation (each its own run, per the phase-171 completion criteria):
|
||||||
|
- `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` (THIS phase — must be green in isolation),
|
||||||
|
- `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171 — the rhythm/band/cursor + the 375/1440 no-page-overflow pins),
|
||||||
|
- `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168 — the panel contract).
|
||||||
|
3. Run the **full gates**: `uv run pytest` (green), `uv run pytest --cov=app --cov-report=term-missing` (**>90%** on `app/` — this phase changes no `app/` code: the gate stays green, not dropped), `uv run ruff check . && uv run pyright` (clean).
|
||||||
|
4. **Commit** the phase atomically (AGENTS rule 8):
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): mobile settings table — a usable Value-column width (the card's horizontal scroll absorbs it)"
|
||||||
|
```
|
||||||
|
then move the phase dir: `mv .agents/phases/todo/176_mobile_settings_value_width .agents/phases/complete/176_mobile_settings_value_width` (the dir move is part of the same phase completion — commit it with `git add .agents/ && git commit --no-gpg-sign -m "chore(agent): complete phase 176 mobile settings value width"` ONLY if the repo's convention is a separate commit; the phase-171 precedent shows a single commit — follow it: the dir move is folded into the atomic commit by `git add .agents/phases/` AFTER the `mv`).
|
||||||
|
|
||||||
|
- ASSUMPTION: A3 (the 12 rem / 192 px E2E pin threshold, the deliberate margin under the 13 rem rule) — owner-locked 2026-10-12. The probe-row choice (one text input, one select, one textarea) is executor mechanics within that lock: the three control types must exist among the 75 rows (the phase-168 catalogue guarantees text / number / select / textarea / password controls).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (AGENTS rule 4): `tests/e2e/test_settings_mobile_value_width.py` green **in isolation**; phase 171 + phase 168 suites green in isolation (regression).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged code).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` green in isolation (the 375 px input ≥ 192 px, the card scrolls, the page does not; 1440 px inert; keyboard + zero external requests).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` and `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] `uv run pytest` green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (the phase-171 single-commit precedent, dir move folded in); the phase dir lives in `.agents/phases/complete/176_mobile_settings_value_width/`.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Phase 177 — The favicon's colors follow the effective theme palette
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "We need the favicon to change colors to match the theme."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
|
||||||
|
**Context:** `frontend/assets/favicon.svg` (the current static art — a hexagon + center dot + 4 spokes whose three hardcoded colors map 1:1 onto the built-in palette: `#1a0f0f` = `surface`, `#f43f5e` = `brand`, `#fca5a5` = `brand_ink`), the five templates' L8 `<link rel="icon" type="image/svg+xml" href="/assets/favicon.svg">` (`frontend/index.html`, `frontend/document.html`, `frontend/login.html`, `frontend/shared.html`, `frontend/doc-edit.html`), `app/core/theming.py` (`BUILTIN_COLORS`, `COLOR_FIELDS`, `effective_settings()` — the DB-over-env/built-in resolver with NO process cache, `theme_style_tag()` + `inject_theme()` — the pre-paint tag pattern, `theme_csp_hash()`), `app/core/caching.py` (`CachingMiddleware`'s per-response short-lived-session theme read with the never-break DB-blip fallback; `/favicon.svg` is neither in `HTML_PAGES` nor under `/assets/` → passes through untouched), `app/main.py` `create_app()` (router includes end ~L199; the static catch-all mount at L252 — routes-first), `app/core/security_headers.py` (CSP `default-src 'self'; … img-src 'self' data:` — the phase-123 carve-out that makes a `data:` favicon URI legal; CSP3 governs `link[rel=icon]` by `img-src`), `frontend/assets/theme.js` (the admin Theme tab — `previewColor()` live preview, the `input` bindings, `saveTheme()`/`resetTheme()` + the `bor:view-refresh` re-show handler), `tests/unit/test_theming.py` (the static pin set incl. `test_builtin_colors_match_styles_css_root`), `tests/unit/test_theme_frontend.py` (the JS static-pin pattern), `tests/e2e/test_theme_save_and_coverage.py` + `test_theme_semantic_completion.py` (the admin theme-tab E2E pattern: `e2e/auth_helpers.login(page, app_url, next="/theme.html")`, the `#theme-*` inputs), AGENTS rules 4/6/8/9/10, PLAN §7.4 (the never-stale feedback contract).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The browser-tab favicon must carry the EFFECTIVE theme palette (built-in or the owner's saved one), painted on first load like everything else — no red hexagon on a graphite theme. The favicon is served by the app from the same resolver that paints the page (`effective_settings`), and the admin Theme tab's live (unsaved) picks repaint it too (owner-confirmed A4).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the pure renderer (task 01, A1).** `theming.favicon_svg(colors: dict[str, str]) -> str` — the SAME artwork (viewBox `0 0 64 64`, the hexagon path, the r=6.5 center circle, the 4 spokes, all stroke widths/caps unchanged) with only the three color roles swapped: hexagon `fill` → `surface`, hexagon `stroke` + circle `fill` → `brand`, spokes `stroke` → `brand_ink`. The input is the full 20-value effective dict (mirroring `theme_style_tag`'s self-contained signature; the function reads its three keys). **Byte-identical contract:** for the built-in palette the output is EXACTLY the current static file's `<svg>` element (lowercase hex, same attribute order) — pinned by a golden constant in `tests/unit/test_theming.py`; the static file's comment header is file packaging, not art, and is not part of the contract.
|
||||||
|
- **D2 — serving (task 02, A2/A3).** `GET /favicon.svg` (+ HEAD for static-file parity) — a no-prefix route in `create_app()` after the router includes and BEFORE the static catch-all mount (routes-first, the phase-51 shared-page pattern). Sync handler: one short-lived `SessionLocal()` session (the `caching.py` house pattern — NO process cache, so a runtime theme change is visible on the next request), `effective_settings()` → `favicon_svg()`; ANY exception → `logger.exception` + `favicon_svg(BUILTIN_COLORS)` (200, never 500 — the theme-tag's never-break contract, `caching.py`'s DB-blip pattern). Response: `media_type="image/svg+xml"`, `Cache-Control: no-cache` (the pages are no-cache; the palette changes at runtime, so the browser must revalidate — the route is NOT under `/assets/`, so no version token; freshness is revalidation, not busting). Deliberately OUTSIDE `/api/*`: no per-response byte-budget line (AGENTS rule 10 is `/api/*`-only), no auth dependency (the favicon is chrome, like the login page). `CachingMiddleware` passes it through untouched (neither an HTML page nor an asset); `SecurityHeadersMiddleware` still adds the CSP — no change needed (`img-src 'self' data:` covers the D4 preview).
|
||||||
|
- **D3 — the templates + the static file's deletion (task 03, A2).** All five L8 links → `href="/favicon.svg"`; `frontend/assets/favicon.svg` is **deleted** — the route is the single source (the built-in bytes stay pinned by D1's golden constant, so nothing is lost). The new href is not under `/assets/`, so `CachingMiddleware`'s `?v=` rewrite regex can never touch it (a pin keeps this explicit — the link stays bare).
|
||||||
|
- **D4 — the Theme tab live preview (task 04, A4 — owner-confirmed 2026-10-13).** `theme.js` gains a pure `themeFaviconDataUri(surface, brand, brandInk)` — the CLIENT-SIDE MIRROR of `theming.favicon_svg` (same artwork, same roles; a unit pin keeps the two from diverging — the `PAIRS`-mirror house pattern). The existing per-color `input` binding ALSO updates `link[rel="icon"].href` to the `data:` URI whenever one of the three driving fields (`surface`/`brand`/`brand_ink`) changes (rebuilt from the three inputs' current values; the other 14 picks never touch it). **Restore to `"/favicon.svg"`** on: Save success (after `applyServedTheme`), Reset success, the Reset-failure `clearPreview()` path, and the `bor:view-refresh` re-show — after every restore the SETTLED palette (re-read per request, no process cache) governs. CSP note (commented in the code): `data:` is legal for the favicon via the phase-123 `img-src 'self' data:` carve-out (CSP3: `link[rel=icon]` is governed by `img-src`; the carve-out is image-scoped — `data:` never becomes a script/style/fetch source).
|
||||||
|
- **NOT touched:** `CachingMiddleware`, the `#bor-theme` tag injection + CSP hash, the 8 semantic state colors (they are NOT in the artwork — a favicon has no state meaning), `app/` outside `theming.py`/`main.py`, `brand.js`, the other templates/assets, the `theme.js` save/reset/contrast machinery beyond the additions above.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `176_mobile_settings_value_width` (todo) — the immediately preceding phase (Protocol B ordering; no functional dependency).
|
||||||
|
- `92_theme_save_and_coverage` (complete) — the served-theme sync + the `#theme-*` input contract the D4 additions bind to.
|
||||||
|
- `91_admin_theme_tab` (complete) — the Theme tab's live-preview mechanism (`previewColor`) and the `bor:view-refresh` re-show hook.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_favicon_svg_renderer.md` — `theming.favicon_svg()`: the pure palette→artwork renderer, byte-identical for the built-in palette (golden pin) — A1.
|
||||||
|
2. `02_favicon_route.md` — `GET /favicon.svg` (+ HEAD) in `create_app()`: effective palette, built-in fallback on DB failure (200, never 500), `image/svg+xml` + `no-cache` — A2/A3.
|
||||||
|
3. `03_template_link_swap.md` — the five template links → `/favicon.svg`; `frontend/assets/favicon.svg` deleted (single source).
|
||||||
|
4. `04_theme_tab_live_preview.md` — the Theme tab's live picks repaint the favicon (the CSP-legal `data:` mirror, A4); Save/Reset/re-show restore the route URL; the dedicated E2E suite + the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_theming.py` (task 01): the golden byte-identical pin, the themed-role pin (the three hexes in their exact roles), the geometry-invariance pin (only the 4 color attributes differ); `tests/unit/test_theme_frontend.py` (task 04): the `data:` mechanism pin (the three driving field names, the exact `/favicon.svg` restore, the restore wiring).
|
||||||
|
- **Integration:** `tests/integration/test_favicon_route.py` (tasks 02–03): the three route states (default = golden bytes, themed = effective colors, DB failure = built-in fallback 200), the CSP header presence, the five template link pins (bare `/favicon.svg`, no `?v=`, no `/assets/favicon.svg` anywhere in the served HTML), the deleted file's 404.
|
||||||
|
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_themed_favicon.py`, run **in isolation** — the full contract in a real browser (default/themed route bytes, the pre-paint tag non-regression, the live-preview `data:` repaint, the restore contract, zero external requests — rule 6).
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); **lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /favicon.svg` on a default (no ui_settings row) deployment is 200 `image/svg+xml` with `Cache-Control: no-cache` and a body byte-identical to the current static art (the golden pin, the integration test).
|
||||||
|
- [ ] With a saved non-default palette (distinct `surface`/`brand`/`brand_ink`), the route's SVG carries those three hexes in their exact roles, and the served pages STILL inject the `#bor-theme` tag (the phase-91 pin stays green — no regression to the tag/CSP contract).
|
||||||
|
- [ ] A DB failure at request time serves the built-in art with 200 (never 500).
|
||||||
|
- [ ] All five templates link `href="/favicon.svg"` (bare — no `?v=`), `/assets/favicon.svg` is 404 (the file is deleted), and the favicon actually renders in the browser (the E2E).
|
||||||
|
- [ ] In the admin Theme tab, a live (unsaved) pick of `surface`/`brand`/`brand_ink` repaints the favicon via the `data:` URI; Save / Reset / re-show restore `href="/favicon.svg"` with the settled palette governing (the E2E).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` green **in isolation**; `test_theme_save_and_coverage.py` + `test_theme_semantic_completion.py` (phases 92/93) green in isolation (no theme-tab regression).
|
||||||
|
- [ ] ONE atomic Conventional Commits commit (AGENTS rule 8, `--no-gpg-sign`) covering the phase + this directory + the cleared TODO.md.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 01 — The pure favicon renderer in `theming.py`
|
||||||
|
|
||||||
|
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pure, unit-testable renderer that produces the favicon artwork from the effective palette — the same art as today's static `frontend/assets/favicon.svg`, byte-identical for the built-in palette.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/core/theming.py` — add `favicon_svg(colors: dict[str, str]) -> str` next to `theme_style_tag()`. The output is EXACTLY the `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">…</svg>` element (no comment header — the static file's comment is file packaging, not art). The artwork is unchanged: the hexagon path (`d="M32 4 55 18v28L32 60 9 46V18Z"`, `stroke-width="4"`, `stroke-linejoin="round"`), the center `<circle cx="32" cy="32" r="6.5">`, the spokes path (`d="M32 25.5V16M32 48v-9.5M25.5 32H16M48 32h-9.5"`, `stroke-width="3"`, `stroke-linecap="round"`). Only the three color roles swap, per the A1 mapping: hexagon `fill` ← `colors["surface"]`, hexagon `stroke` ← `colors["brand"]`, circle `fill` ← `colors["brand"]`, spokes `stroke` ← `colors["brand_ink"]`. The input is the full 20-value effective dict (mirroring `theme_style_tag`'s self-contained signature — the function reads its three keys); pure (dict in → str out, no DB, no app, no defensive branches — the `effective_settings`/`BUILTIN_COLORS` contract guarantees the values). Docstring: the A1 role table, the byte-identical contract, and the "no state colors" note (the 8 semantic state colors are not in the artwork — a favicon has no state meaning).
|
||||||
|
- ASSUMPTION: A1 (owner-confirmed) — the color mapping is hexagon fill→`surface`, hexagon stroke + center dot→`brand`, spokes→`brand_ink`; only these three identity colors drive the favicon.
|
||||||
|
2. `tests/unit/test_theming.py` — extend the pin set:
|
||||||
|
- `test_favicon_svg_builtins_is_byte_identical_to_the_static_art` — `favicon_svg(BUILTIN_COLORS)` equals a GOLDEN constant holding the current static file's `<svg>` element bytes. During this task's execution, verify the golden constant against the current `frontend/assets/favicon.svg` file (a one-time sanity check — the file is deleted in task 03, so the golden constant, not the file, is the durable contract).
|
||||||
|
- `test_favicon_svg_themed_carries_the_three_roles` — a palette with distinct `surface`/`brand`/`brand_ink` (e.g. `#102030` / `#4f46e5` / `#a5b4fc`) plus the other 14 built-ins: the three hexes appear in their exact attribute roles (assert by parsing the attributes, not by substring), and none of the other 14 hexes appears.
|
||||||
|
- `test_favicon_svg_geometry_is_invariant` — with a themed palette, everything except the four color attributes is byte-equal to the built-in output (viewBox, both path `d`s, circle `cx/cy/r`, stroke widths, `linejoin`/`linecap`, attribute order).
|
||||||
|
3. One-time sanity: diff the golden constant against `frontend/assets/favicon.svg`'s `<svg>` element and log the result (evidence for the hand-off; not a persistent test).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the three pins above (`tests/unit/test_theming.py`), run with `uv run pytest tests/unit/test_theming.py -v`.
|
||||||
|
- Coverage: **>90%** on this task's new code (a pure function — the pins drive it to ~100%).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_theming.py -v` green (new + existing pins).
|
||||||
|
- [ ] The one-time golden-vs-file sanity check passes (logged).
|
||||||
|
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] No behavior change in completed work (a new function with no existing call site).
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — Serve the themed favicon: `GET /favicon.svg`
|
||||||
|
|
||||||
|
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The favicon over the wire: a route that renders the artwork from the EFFECTIVE palette on every request (no process cache — a runtime theme change is visible on the next request), with the never-break DB-blip fallback.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/main.py` — in `create_app()`, after the router includes (ending at the `chats_shared_page_router` line ~L200) and BEFORE the static catch-all mount (`app.mount("/", …)` at L252, routes-first): register `app.api_route("/favicon.svg", methods=["GET", "HEAD"], include_in_schema=False)` (HEAD for static-file parity, the phase-76 `_shell_routes` parity idiom). The handler is a sync `def` (the house pattern for short-lived-session reads):
|
||||||
|
- open a short-lived `SessionLocal()` session (imported from `app.db`, the `caching.py` middleware's exact pattern — NO process cache, `try/finally` close);
|
||||||
|
- `effective = theming.effective_settings(db)` → `svg = theming.favicon_svg(effective)`;
|
||||||
|
- ANY exception (DB blip, pre-migration boot): `logger.exception("favicon: theme read failed — serving the built-in favicon")` and `svg = theming.favicon_svg(theming.BUILTIN_COLORS)` — **200, never 500** (the theme-tag's never-break contract, `caching.py`'s DB-blip pattern);
|
||||||
|
- return `Response(svg, media_type="image/svg+xml", headers={"Cache-Control": "no-cache"})`.
|
||||||
|
- Commented rationale in the code (house style): (a) the route is deliberately OUTSIDE `/api/*` → no per-response byte-budget line (AGENTS rule 10 is `/api/*`-only) and no auth dependency (the favicon is chrome — public like the login page); (b) `CachingMiddleware` passes it through untouched (neither in `HTML_PAGES` nor under `/assets/`) → the route's own `no-cache` is the freshness contract (the palette changes at runtime; the page is no-cache too, so the browser revalidates together with it); (c) `SecurityHeadersMiddleware` still adds the CSP — no change needed (`img-src 'self' data:` covers task 04's `data:` preview).
|
||||||
|
- ASSUMPTION: A2 (owner-confirmed) — serving shape: a new dynamic route `GET /favicon.svg` replaces the static file (task 03 deletes it).
|
||||||
|
- ASSUMPTION: A3 (owner-confirmed) — fault tolerance: a DB failure / pre-migration boot serves the built-in palette with 200 (never 500).
|
||||||
|
2. `tests/integration/test_favicon_route.py` — NEW, via the house `TestClient` fixture (follow the `tests/integration/test_ui_settings.py` app/db fixture pattern):
|
||||||
|
- **default state** (no `ui_settings` row): `GET /favicon.svg` → 200, `content-type: image/svg+xml`, `cache-control: no-cache`, body byte-identical to the golden art constant (import it from `tests/unit/test_theming.py` — one source), and the response carries the CSP header (the `SecurityHeadersMiddleware` pass-through).
|
||||||
|
- **themed state**: a `ui_settings` row (id 1) with distinct `surface`/`brand`/`brand_ink` (e.g. `#102030` / `#4f46e5` / `#a5b4fc`): the body carries the three hexes in their roles (attribute-parsed, same helper idiom as the unit pins) and none of the other 14; a second request after updating the row reflects the new palette immediately (no process cache — the runtime-change contract).
|
||||||
|
- **DB-failure state**: patch `effective_settings` (or `SessionLocal`) to raise → 200 + the built-in golden bytes (never 500).
|
||||||
|
- **HEAD parity**: `HEAD /favicon.svg` → 200, the same headers, empty body.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the four states above (`tests/integration/test_favicon_route.py`), run with `uv run pytest tests/integration/test_favicon_route.py -v`.
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (the handler's success + fallback branches both driven).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_favicon_route.py -v` green (all four states).
|
||||||
|
- [ ] The route wins over the static mount (a file named `favicon.svg` in the static dir could not shadow it — the routes-first position holds).
|
||||||
|
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] No behavior change in completed work (a new route only; `/assets/favicon.svg` still serves until task 03 deletes it).
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — Point all five templates at the route; delete the static file
|
||||||
|
|
||||||
|
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every page asks for the themed favicon from the route, and the static file stops existing — the route is the single source (the built-in bytes stay pinned by task 01's golden constant, so nothing is lost).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html`, `frontend/document.html`, `frontend/login.html`, `frontend/shared.html`, `frontend/doc-edit.html` — L8 in each file: `<link rel="icon" type="image/svg+xml" href="/assets/favicon.svg">` → `<link rel="icon" type="image/svg+xml" href="/favicon.svg">`. Nothing else in any template changes.
|
||||||
|
2. Delete `frontend/assets/favicon.svg` (A2, owner-confirmed — the single-source decision; the one-time golden-vs-file sanity was done in task 01, so the deletion loses no bytes: the golden constant in `tests/unit/test_theming.py` is the durable contract).
|
||||||
|
3. `tests/integration/test_favicon_route.py` — extend with the template pins:
|
||||||
|
- each of the five served page bodies (`/`, `/document.html`, `/login.html`, `/shared.html`, `/doc-edit.html` via `TestClient`) contains `<link rel="icon" type="image/svg+xml" href="/favicon.svg">` and does NOT contain `/assets/favicon.svg`;
|
||||||
|
- the favicon link in the served HTML stays BARE — no `?v=` (the href is not under `/assets/`, so `CachingMiddleware`'s `_ASSET_REF_RE` can never touch it; the pin keeps that contract explicit against a future regex drift);
|
||||||
|
- `GET /assets/favicon.svg` → 404 (the file is gone);
|
||||||
|
- a tree scan of `frontend/` (the test walks the directory, the house idiom) asserts zero remaining references to `/assets/favicon.svg`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the template pins above, run with `uv run pytest tests/integration/test_favicon_route.py -v`.
|
||||||
|
- Coverage: **>90%** on `app/` (no new `app/` code in this task — the gate must stay green, not drop).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All five templates link `href="/favicon.svg"`; the tree scan finds zero `/assets/favicon.svg` references.
|
||||||
|
- [ ] `GET /assets/favicon.svg` is 404; `GET /favicon.svg` still serves the golden bytes (the route is the single source now).
|
||||||
|
- [ ] In a running app, a browser tab loads the favicon from `/favicon.svg` (final browser proof in task 04's E2E).
|
||||||
|
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] No behavior change in completed work (the favicon art is byte-identical on a default deployment — the golden pin holds).
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Task 04 — Theme-tab live preview + the dedicated E2E suite + the commit
|
||||||
|
|
||||||
|
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The admin Theme tab's live (unsaved) picks repaint the favicon too (owner-confirmed A4), the settled palette always wins after Save/Reset/re-show — and the phase's dedicated Playwright suite proves the whole contract in a real browser.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/theme.js` —
|
||||||
|
- a pure helper `themeFaviconDataUri(surface, brand, brandInk)` → `"data:image/svg+xml," + encodeURIComponent(svg)`, where `svg` is the CLIENT-SIDE MIRROR of `theming.favicon_svg` (the same artwork, the same A1 roles: hexagon fill ← `surface`, hexagon stroke + circle fill ← `brand`, spokes stroke ← `brand_ink`). Guard with the existing `isHex` idiom: a non-hex input returns `null` (the link is left untouched — the color inputs are always valid hex, the guard is belt-and-braces; the inputs are the only source of the values, so there is no XSS path — the house "never build HTML" rule binds the DOM, the URI is built from constants + validated hexes).
|
||||||
|
- in the existing per-color `input` bindings (the live-preview loop): when the field is one of the three driving fields (`surface`, `brand`, `brand_ink`), ALSO set the `link[rel="icon"]` element's `href` to `themeFaviconDataUri(<the three inputs' current values>)` (rebuilt, not patched — a `surface` pick repaints with the current `brand`/`brand_ink` picks). The other 14 picks never touch the link.
|
||||||
|
- **restore to `"/favicon.svg"`** in the four settle points: `saveTheme()` success (after `applyServedTheme(settings)`), `resetTheme()` success (after the reconcile), the `resetTheme()` failure path (where `clearPreview()` runs — the picks are stale), and the `bor:view-refresh` re-show handler (after its `applyServedTheme`). After every restore the SETTLED palette governs — the route re-reads the DB per request (no process cache), so the restored URL is always fresh; the page is `no-cache`, so the browser revalidates the favicon with it.
|
||||||
|
- CSP comment (house style, per D4): `data:` is legal for the favicon via the phase-123 `img-src 'self' data:` carve-out (CSP3 governs `link[rel=icon]` by `img-src`; the carve-out is image-scoped — `data:` never becomes a script/style/fetch source).
|
||||||
|
- file-header docstring: extend with the favicon-preview paragraph (the mechanism, the four restore points, the CSP note — the theme.js header documents every behavior).
|
||||||
|
- ASSUMPTION: A4 (owner-confirmed 2026-10-13) — the live (unsaved) picks repaint the favicon via the CSP-legal `data:` URI; the settled palette always wins after Save/Reset/re-show. (The simpler alternative — saved palette only, no `data:` — was considered and declined.)
|
||||||
|
2. `tests/unit/test_theme_frontend.py` — extend the static pin set (the house JS-pin idiom — the pins parse `theme.js`'s source):
|
||||||
|
- the `data:image/svg+xml` URI mechanism is present (the literal prefix, `encodeURIComponent`);
|
||||||
|
- the favicon helper reads EXACTLY the three field names `surface`/`brand`/`brand_ink` (the driving-set pin — a fourth field or a missing one fails the pin);
|
||||||
|
- the restore string is EXACTLY `"/favicon.svg"` and is wired to all four settle points (save success, reset success, reset-failure `clearPreview` path, `bor:view-refresh`);
|
||||||
|
- the mirror's artwork constants match the server's (the viewBox + both path `d` strings + circle `r` — the divergence guard, the `PAIRS`-mirror house pattern).
|
||||||
|
3. `tests/e2e/test_themed_favicon.py` — NEW, the phase's dedicated Playwright suite (AGENTS rule 4), run in isolation, following the `test_theme_save_and_coverage.py` fixture idiom (`app_server`/`app_url`/`db_ready`, `e2e/auth_helpers.login(page, app_url, next="/theme.html")`, the `_clean` ui_settings reset):
|
||||||
|
- **default state:** `GET /favicon.svg` → 200 + the golden art bytes (fetched in the test); the five pages' HTML reference `/favicon.svg`; `/assets/favicon.svg` → 404;
|
||||||
|
- **themed state:** admin login → `PUT /api/ui-settings` with a distinct palette (e.g. `surface #102030`, `brand #4f46e5`, `brand_ink #a5b4fc`): the route's SVG carries the three hexes in their roles; a fresh page load STILL injects `#bor-theme` with the expected content (the phase-91 non-regression pin — the tag/CSP contract is untouched);
|
||||||
|
- **live preview:** admin opens `/theme.html` (the `#view-theme` view), sets the `#theme-surface` / `#theme-brand` / `#theme-brand-ink` inputs (the `input` event, the house `_fill_theme_form` idiom) → `link[rel="icon"]`'s href starts with `data:image/svg+xml` and, decoded, carries the three picked hexes in their roles; a non-driving pick (e.g. `#theme-bg`) does not change the href;
|
||||||
|
- **restore contract:** click **Save theme** → the href is EXACTLY `/favicon.svg` and the route now serves the saved palette's bytes; click **Reset to defaults** → the href is `/favicon.svg` and the route serves the built-in golden bytes; re-showing the tab (the `bor:view-refresh` path) restores the href too;
|
||||||
|
- **zero external requests** across the whole suite (AGENTS rule 6 — the house E2E assertion idiom: a `page.on("request")` collector asserting every URL is the local origin / a `data:` URI).
|
||||||
|
4. The atomic commit (AGENTS rule 8): ONE Conventional Commits commit, e.g. `feat(theming): serve the favicon from the effective theme palette`, `--no-gpg-sign` — covering the `app/` + `frontend/` + `tests/` changes, this phase directory (`.agents/` is tracked), and the cleared `TODO.md`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the JS static pins (`tests/unit/test_theme_frontend.py`), run with `uv run pytest tests/unit/test_theme_frontend.py -v`.
|
||||||
|
- E2E: `tests/e2e/test_themed_favicon.py` run **in isolation** — `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov`.
|
||||||
|
- Coverage: **>90%** on `app/` (the JS is gated by the pins + the browser E2E — the house idiom); ruff + pyright clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_theme_frontend.py -v` green (new pins).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` green **in isolation** (all six contract blocks).
|
||||||
|
- [ ] `test_theme_save_and_coverage.py` (phase 92) + `test_theme_semantic_completion.py` (phase 93) green in isolation (no Theme-tab regression).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] The atomic commit is landed (Conventional Commits, `--no-gpg-sign`, `.agents/` + the cleared `TODO.md` included).
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Phase 178 — Split thinking levels: a chat level and a summary level, separately tunable
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-11)
|
||||||
|
**Context:** `app/config.py` L108–115 — `llm_thinking_level: str = "low"` (the phase-172 shared knob, directly after `llm_summary_model` L103–107, before `stream_thinking` L116), `app/core/settings_catalog.py` L154–155 — the single `llm_thinking_level` `SettingMeta` row (label "Thinking level", group `LLM (aipi)`; the completeness pin in `tests/unit/test_settings_catalog.py` — `set(CATALOG) == set(Settings.model_fields)` — makes every `Settings` field a build failure until it has a `SettingMeta`), `.env.example` L61 — `BOR_LLM_THINKING_LEVEL=low`, `app/rag/llm.py` — `_thinking_extra_body(settings)` (L49–57, reads `settings.llm_thinking_level`), `LLMClient.chat()` (L470, the one-shot `chat.completions.create` L430–452 — `extra = _thinking_extra_body(self.settings)` L432, `**extra_kwargs` L444; callers: `app/rag/summarizer.py` L242 + L359 document summaries, `app/rag/folder_summaries.py` L347, `app/rag/overview.py` L254 KB overview, `check_models` L805 the pre-sync probe), `LLMClient.chat_stream()` (L511, the kwargs dict L615–622 + `kwargs["extra_body"] = extra` L625–628), `app/api/chat.py` — the per-turn log line (L1272–1291, `thinking=%s` L1275, `settings.llm_thinking_level.strip() or "-"` L1289), the module docstring L14–19, `frontend/assets/settings.js` (the phase-168 generated panel — **no code change here**: `str` catalogue entries render as free-form text rows automatically), `tests/unit/test_config.py` L108–133 (default / env override / empty-stays-empty), `tests/unit/test_settings_catalog.py` L99–111, `tests/unit/test_llm_client.py` L1330–1440 (the mocked-client `extra_body` family), `tests/unit/test_chat_gate.py` L1550–1630 (the per-turn log-line family), `tests/integration/test_admin_settings_api.py` L521–607 + `test_admin_settings_live.py` L171–204, `tests/e2e/test_thinking_level.py` (the phase-172 Playwright suite — the pattern to extend).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The operator tunes the chat turns' thinking level and the one-shot summaries' thinking level **independently**: `llm_chat_thinking_level` (renamed from the phase-172 `llm_thinking_level`, default `low` — chat stays at low, TODO L3) rides `chat_stream()`; `llm_summary_thinking_level` (default empty = **off** — the TODO's "none", the phase-172 locked A2 off-switch: the parameter is ABSENT from the request) rides the one-shot `chat()` (document summaries, folder summaries, KB overview, the `check_models` probe). Both are free-form text, both live in `.env` (`BOR_LLM_CHAT_THINKING_LEVEL` / `BOR_LLM_SUMMARY_THINKING_LEVEL`) and the admin Settings panel, both live-apply without a restart.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the two fields (task 01).** `app/config.py` LLM section: `llm_thinking_level` is **renamed** to `llm_chat_thinking_level` (default stays `"low"`) and `llm_summary_thinking_level: str = ""` is added next to it (both before `stream_thinking`). Plain `str`, no allowlist, no validator (phase-172 A4, kept). House docstrings name the env var, the default, the wire mapping and the off rule. `.env.example` L61: `BOR_LLM_THINKING_LEVEL=low` is **replaced** by `BOR_LLM_CHAT_THINKING_LEVEL=low` + `BOR_LLM_SUMMARY_THINKING_LEVEL=` (empty value, one-line comment each).
|
||||||
|
- **D2 — the consumers rename with the field (task 01, behavior-identical).** Every consumer of the old field is mechanically renamed to `llm_chat_thinking_level` in task 01 — `app/rag/llm.py` (module docstring, `_thinking_extra_body`), `app/api/chat.py` (module docstring, per-turn log line L1289) — and every test pin is renamed in place (unit `test_config.py` / `test_settings_catalog.py` / `test_llm_client.py` / `test_chat_gate.py`, integration `test_admin_settings_api.py` / `test_admin_settings_live.py`, e2e `test_thinking_level.py`). After task 01 the suite is green and behavior is **byte-identical** to pre-phase: both `chat()` and `chat_stream()` ride the (renamed) chat level, default `low`; the new summary field exists but is inert until task 03.
|
||||||
|
- **D3 — the wire split (task 03).** `_thinking_extra_body` takes the **level string** (pure function, no `Settings` read): `chat_stream()` passes `self.settings.llm_chat_thinking_level`; `chat()` passes `self.settings.llm_summary_thinking_level`. Trimmed non-empty → `{"reasoning_effort": <value>}`; empty/whitespace → `None` (the parameter is ABSENT — never a literal `"none"` or `"off"`, phase-172 A2). The `check_models` probe (L805) goes through `chat()` → summary level → off by default; no special-casing (it is a reachability ping). Embeddings path untouched.
|
||||||
|
- **D4 — the admin rows (task 02).** The single `SettingMeta("llm_thinking_level", …, "Thinking level")` becomes TWO rows: `llm_chat_thinking_level` → "Chat thinking level", `llm_summary_thinking_level` → "Summary thinking level" (both group `LLM (aipi)`, placed where the old row sat — after `llm_summary_model`, before `stream_thinking`). The panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes and the DB-over-env overlay are generic over the catalogue — the two free-form text rows, their save/reset and live-apply fall out with zero extra code. The completeness pin is the enforcement.
|
||||||
|
- **D5 — the per-turn line (task 04).** The per-turn log line's `thinking=<value>` (or `-` when empty) reads the **chat** level — the level the turn actually ran at; the summary level is a one-shot concern and gains no log field (the summary log lines have no knob field today). No `query_log` schema change (phase-172 A6, kept).
|
||||||
|
- **NOT touched:** `frontend/assets/settings.js`, `frontend/assets/styles.css`, the `app_settings` table shape, `app/core/settings_overlay.py` (generic), the embeddings path, `stream_thinking` (the phase-17 kill-switch for `thinking` SSE events), every other setting.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
— (none: the app is fully functional at the head of `complete/`; this phase refines the committed phase-172 thinking-level machinery)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_settings_catalog_env.md` — the rename + the new field in `app/config.py` + the two `CATALOG` rows + `.env.example` + the mechanical consumer/test rename (suite green, behavior byte-identical).
|
||||||
|
2. `02_admin_row_contract.md` — the integration set / reset / live-apply contract for BOTH keys (stored raw, live without restart, empty storable as the summary off-switch).
|
||||||
|
3. `03_llm_extra_body_split.md` — `chat_stream()` → chat level, `chat()` → summary level; the mocked-client unit tests per method.
|
||||||
|
4. `04_per_turn_log_field.md` — `thinking=<chat level>` (or `-`) pinned: the line follows the chat knob, never the summary knob.
|
||||||
|
5. `05_e2e_admin_settings_rows.md` — `tests/e2e/test_split_thinking_levels.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_config.py` — chat default `low` / env override / empty-stays-empty; summary default `""` / env override / empty-stays-empty (no coercion, no allowlist — D3/A2). `tests/unit/test_settings_catalog.py` — the completeness pin stays green (both fields have their `SettingMeta`; group/label/flags correct). `tests/unit/test_llm_client.py` — `chat_stream()` sends `extra_body={"reasoning_effort": <chat level>}` when the chat level is trimmed-non-empty and no `extra_body` when it is empty; `chat()` sends it from the **summary** level with the same absent-when-empty rule; an empty summary level (the default) keeps one-shot requests byte-identical to pre-phase-172; the embeddings path is unaffected. `tests/unit/test_chat_gate.py` — the per-turn line shows the chat level (with the summary level deliberately different — it must not leak into the line).
|
||||||
|
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — BOTH rows appear in `GET /api/admin/settings` (`str`, correct labels, effective defaults `low` / ``); `PUT` stores the raw string on each, the empty value is storable (the summary off-switch), the overlay resolves live without a restart; `DELETE` resets each to env.
|
||||||
|
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_split_thinking_levels.py`, run **in isolation** — both rows visible in the `LLM (aipi)` group (labels "Chat thinking level" / "Summary thinking level"), chat default `low`, summary default empty, both free-form text inputs (type, save, survives reload), both reset-to-env restore the defaults, the env-var display pins `BOR_LLM_CHAT_THINKING_LEVEL` / `BOR_LLM_SUMMARY_THINKING_LEVEL`; the `test_admin_settings_panel.py` + `test_admin_settings_layout.py` regression set stays green.
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest -k "config and thinking" -v` shows chat default `low` + summary default `""` (env overrides for both).
|
||||||
|
- [ ] `GET /api/admin/settings` (admin) lists `llm_chat_thinking_level` and `llm_summary_thinking_level` in the `LLM (aipi)` group (`str`, defaults `low` / empty); `PUT` / `DELETE` round-trip each per the overlay contract; the old `llm_thinking_level` key no longer appears.
|
||||||
|
- [ ] With the levels set, `chat_stream()` carries `reasoning_effort` from the chat level and `chat()` from the summary level (unit-asserted via the mocked client); with a level empty, that method's request kwargs contain no `extra_body` (byte-identical to pre-phase-172 for the default one-shot path).
|
||||||
|
- [ ] The per-turn log line carries `thinking=<chat level>` (`-` when empty) — never the summary level.
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 171) green in isolation.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(llm): split the thinking level into a chat level (default low) and a summary level (default off)`); the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-11, roadmap confirmation)
|
||||||
|
- **A1 — rename, not add-on:** `llm_thinking_level` is **renamed** to `llm_chat_thinking_level` (`BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`). Any DB-stored override under the old key becomes inert (it is no longer a catalogue row) and chat falls back to the env default `low` — chat behavior is preserved exactly as the TODO wants; no value migration.
|
||||||
|
- **A2 — the TODO's "none" = the phase-172 off-switch:** `llm_summary_thinking_level` defaults to empty (`""`) → `reasoning_effort` is ABSENT from one-shot `chat()` requests. Per phase-172's locked A2 ("nothing is sent as `off` or `null` — absence is the off"), no literal `"none"` string is ever sent.
|
||||||
|
- **A3 — scope:** `chat_stream()` (the chat turn) → chat level; every one-shot `chat()` (document summaries, folder summaries, KB overview, the `check_models` probe) → summary level; the embeddings path is untouched.
|
||||||
|
- **A4 — no frontend code change:** the panel is generated from the catalogue; the two free-form text rows appear automatically (phase-172 A5 pattern).
|
||||||
|
- **A5 — the per-turn line** keeps its single `thinking=<chat level>` field (`-` when empty), reads the chat level, and gains no summary field; no `query_log` schema change.
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Task 01 — The two settings + the two catalogue rows + `.env.example` + the mechanical rename
|
||||||
|
|
||||||
|
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The two settings exist in `app/config.py` (chat default `low`, summary default empty = off, both free-form), in the admin catalogue (the completeness pin stays green), and in `.env.example` — and every existing consumer/test of the old single `llm_thinking_level` is renamed so the suite is green and behavior is **byte-identical** to pre-phase (both LLM paths still ride the chat level, default `low`).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — in the LLM section, where the phase-172 field sits (L108–115, directly after the `llm_summary_model` block, before `stream_thinking` L116): **rename** `llm_thinking_level` → `llm_chat_thinking_level` (default stays `"low"`) and add the summary field next to it. House docstring style (env var named, default, wire mapping, off rule):
|
||||||
|
```python
|
||||||
|
#: The chat turn's thinking level (phase 178,
|
||||||
|
#: ``BOR_LLM_CHAT_THINKING_LEVEL``; the phase-172 knob, renamed) —
|
||||||
|
#: pi-style reasoning effort, free-form text exactly like
|
||||||
|
#: ``llm_chat_model`` (no allowlist — the endpoint is the source of
|
||||||
|
#: truth). Default ``low``. Rides the streaming chat-completions
|
||||||
|
#: requests to aipi as the top-level ``reasoning_effort`` parameter
|
||||||
|
#: (``app/rag/llm.py`` ``chat_stream``); the EMPTY value is the off
|
||||||
|
#: switch — the parameter is not sent at all.
|
||||||
|
llm_chat_thinking_level: str = "low"
|
||||||
|
#: The one-shot summary thinking level (phase 178,
|
||||||
|
#: ``BOR_LLM_SUMMARY_THINKING_LEVEL``) — rides the one-shot
|
||||||
|
#: ``chat()`` calls (document summaries, folder summaries, KB
|
||||||
|
#: overview, the ``check_models`` probe) as ``reasoning_effort``.
|
||||||
|
#: Default EMPTY = the off switch (phase-172 A2): the parameter is
|
||||||
|
#: not sent at all; the TODO's "none" means absence, never a
|
||||||
|
#: literal "none" string.
|
||||||
|
llm_summary_thinking_level: str = ""
|
||||||
|
```
|
||||||
|
No validator, no coercion — plain `str` (phase-172 A4, kept). Verify by reading: `llm_summary_model` (L103–107) and `stream_thinking` (L116–120) keep their shape byte-identical.
|
||||||
|
2. `app/core/settings_catalog.py` — replace the single row (L154–155) with TWO rows, placed exactly where the old row sat (after `llm_summary_model` L151–152, before `stream_thinking` L157):
|
||||||
|
```python
|
||||||
|
"llm_chat_thinking_level": SettingMeta(
|
||||||
|
"llm_chat_thinking_level", "LLM (aipi)", "Chat thinking level"
|
||||||
|
),
|
||||||
|
"llm_summary_thinking_level": SettingMeta(
|
||||||
|
"llm_summary_thinking_level", "LLM (aipi)", "Summary thinking level"
|
||||||
|
),
|
||||||
|
```
|
||||||
|
Nothing else: the panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes and the overlay are generic over the catalogue (D4).
|
||||||
|
3. `.env.example` — in the LLM block (L57–65), replace `BOR_LLM_THINKING_LEVEL=low` (L61) with:
|
||||||
|
```
|
||||||
|
BOR_LLM_CHAT_THINKING_LEVEL=low # the chat turn's thinking level (pi-style; free-form, rides reasoning_effort); empty = the parameter is not sent
|
||||||
|
BOR_LLM_SUMMARY_THINKING_LEVEL= # the one-shot summary thinking level (document/folder summaries, KB overview); empty (default) = the parameter is not sent
|
||||||
|
```
|
||||||
|
directly under `BOR_LLM_CHAT_MODEL=turbo` (L60), house comment style.
|
||||||
|
4. `app/rag/llm.py` — **mechanical rename, behavior-identical** (D2): the module docstring (L14–19 — rename `BOR_LLM_THINKING_LEVEL` / `default low` to `BOR_LLM_CHAT_THINKING_LEVEL`; the "rides every chat-completions request (one-shot `chat` and streaming `chat_stream`)" sentence stays TRUE after this task because both paths still ride the chat level; task 03 finalizes the wording) and `_thinking_extra_body` (L49–57, `settings.llm_thinking_level` → `settings.llm_chat_thinking_level`; docstring names the chat level). `chat()` (L432) and `chat_stream()` (L625) keep calling the same helper with the chat level — no split yet (that is task 03). The embeddings path is untouched.
|
||||||
|
5. `app/api/chat.py` — **mechanical rename, behavior-identical** (D2): the module docstring (L14–19, `BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`, "the effective chat level") and the per-turn log line argument (L1289, `settings.llm_thinking_level` → `settings.llm_chat_thinking_level`; the format string L1275 keeps `thinking=%s`).
|
||||||
|
6. `tests/unit/test_config.py` — L108–133: the three phase-172 tests become SIX: `llm_chat_thinking_level` (default `low` with `BOR_LLM_CHAT_THINKING_LEVEL` unset; env override `high`; empty env value stays empty) and `llm_summary_thinking_level` (default `""` with `BOR_LLM_SUMMARY_THINKING_LEVEL` unset; env override `high`; empty env value stays empty). Same monkeypatch pattern, no coercion asserted.
|
||||||
|
7. `tests/unit/test_settings_catalog.py` — L99–111: the single-row test becomes a BOTH-rows test (each `SettingMeta` in the `LLM (aipi)` group with the correct label, `str`-ish flags — not in `SECRETS`, not in `RESTART_REQUIRED`, not in `NON_OVERRIDABLE`); the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green.
|
||||||
|
8. `tests/unit/test_llm_client.py` — L1330–1440: rename the kwarg `llm_thinking_level=` → `llm_chat_thinking_level=` in the existing family (behavior assertions unchanged — the helper still reads the chat level); the `chat()`-path tests keep passing as-is.
|
||||||
|
9. `tests/unit/test_chat_gate.py` — L1550–1630: rename the `Settings(..., llm_thinking_level=…)` pins → `llm_chat_thinking_level=…`; the log-line assertions (L1553, `BOR_LLM_THINKING_LEVEL` docstring) name `BOR_LLM_CHAT_THINKING_LEVEL` / the chat level.
|
||||||
|
10. `tests/integration/test_admin_settings_api.py` — L521–607: the `llm_thinking_level` row contract is renamed to `llm_chat_thinking_level` (env var `BOR_LLM_CHAT_THINKING_LEVEL`, default `low`; PUT/DELETE/anonymous-401 pins unchanged in shape). Task 02 extends it for the summary row.
|
||||||
|
11. `tests/integration/test_admin_settings_live.py` — L171–204: the live-apply test is renamed to `llm_chat_thinking_level` (same save/empty/delete pattern against the new key).
|
||||||
|
12. `tests/e2e/test_thinking_level.py` — rename the pins in place: `tr[data-key="llm_thinking_level"]` → `tr[data-key="llm_chat_thinking_level"]`, `BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`, the `Settings.model_fields["llm_thinking_level"]` env lookup → `llm_chat_thinking_level`. E2E runs only in isolation (the per-task gate covers unit + integration), but the file must not rot: task 05 turns it into `test_split_thinking_levels.py`.
|
||||||
|
- ASSUMPTION: A1 (phase-locked) — the rename orphans any stored `app_settings` row under `llm_thinking_level` (it becomes inert; chat falls back to env default `low`); no value migration is performed.
|
||||||
|
- ASSUMPTION: A2 (phase-locked) — the summary default is the empty string (the phase-172 off-switch: `reasoning_effort` ABSENT), never a literal `"none"`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_config.py` (six thinking-level tests), `tests/unit/test_settings_catalog.py` (the completeness pin + both rows), `tests/unit/test_llm_client.py` (renamed family, unchanged assertions), `tests/unit/test_chat_gate.py` (renamed pins).
|
||||||
|
- Integration: `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (renamed row contract, green).
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "llm_thinking_level\b\|BOR_LLM_THINKING_LEVEL\b" app/ tests/ .env.example` returns **nothing** (the old name is gone everywhere; `llm_chat_thinking_level` / `BOR_LLM_CHAT_THINKING_LEVEL` in its place).
|
||||||
|
- [ ] `uv run pytest -k "config and thinking" -v` — chat default `low` + summary default `""` + env overrides for both.
|
||||||
|
- [ ] `uv run pytest` green (unit + integration), `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] Behavior check (read, not run): `chat()` and `chat_stream()` both still ride the chat level; with both defaults (`low` / empty) and the old `.env` value, streaming requests are byte-identical to pre-phase-172.
|
||||||
|
- [ ] No behavior change in completed work.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — The admin row contract: both keys, set / reset / live-apply
|
||||||
|
|
||||||
|
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The integration contract proves both rows behave per the overlay: they appear in the catalogue with the right group/label/default, `PUT` stores the raw string (empty included — the summary off-switch), the value resolves live without a restart, and `DELETE` resets to env — for the chat row AND the summary row, independently.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/integration/test_admin_settings_api.py` — the phase-172 row contract (now `test_llm_chat_thinking_level_row_contract`, from task 01) gains a **summary twin** `test_llm_summary_thinking_level_row_contract` following the same shape, with the summary specifics:
|
||||||
|
- `GET /api/admin/settings` lists `llm_summary_thinking_level` in the `LLM (aipi)` group, `str`, label "Summary thinking level", `env_var` `BOR_LLM_SUMMARY_THINKING_LEVEL`, effective default `""` (with `BOR_LLM_SUMMARY_THINKING_LEVEL` deleted from the env for the test — the house `monkeypatch.delenv(…, raising=False)` pattern).
|
||||||
|
- `PUT {API}/llm_summary_thinking_level` with `{"value": "high"}` → 200, the raw string stored (`SELECT value FROM app_settings WHERE key = 'llm_summary_thinking_level'` → `"high"`), `get_settings().llm_summary_thinking_level == "high"` **without a restart**.
|
||||||
|
- `PUT` with `{"value": ""}` → 200 (the off-switch is storable — an empty stored value must not be rejected as "no value"), `get_settings().llm_summary_thinking_level == ""`.
|
||||||
|
- `DELETE` → 200, the row gone from `app_settings`, `get_settings().llm_summary_thinking_level == ""` (the env default).
|
||||||
|
- An anonymous (non-admin) `PUT` → the 401/403 pin, same as the chat twin.
|
||||||
|
- Independence pin: after `PUT`-ing the summary row `"high"`, the chat row's effective value is still `"low"` (the two overlays resolve independently — the whole point of the split).
|
||||||
|
2. `tests/integration/test_admin_settings_live.py` — the chat live-apply test (from task 01) gains a **summary twin** `test_llm_summary_thinking_level_live_apply_no_restart`: `delenv` `BOR_LLM_SUMMARY_THINKING_LEVEL` → default `""`; `_save(db, "llm_summary_thinking_level", "high")` → `get_settings().llm_summary_thinking_level == "high"` **NO restart**; `_save(…, "")` → `""`; `DELETE FROM app_settings WHERE key = 'llm_summary_thinking_level'` → back to `""`.
|
||||||
|
3. Verify by running: `uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v` — both families green, and the old single-row test is gone (no `llm_thinking_level` key referenced anywhere: `grep -rn "llm_thinking_level\b" tests/` empty).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the two extended families above (row contract + live-apply, chat AND summary).
|
||||||
|
- Coverage: **>90%** on `app/` (this task is test-only — `app/` code is untouched, so the task-01 coverage must simply stay green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py -v -k "thinking" -v` — the chat + summary row contracts green (group/label/env-var/default, PUT raw storage incl. empty, DELETE reset, anonymous rejected, independence).
|
||||||
|
- [ ] `uv run pytest tests/integration/test_admin_settings_live.py -v -k "thinking"` — both live-apply twins green (no restart, empty storable, delete resets).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%.
|
||||||
|
- [ ] No behavior change in completed work.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Task 03 — The wire split: `chat_stream()` → chat level, `chat()` → summary level
|
||||||
|
|
||||||
|
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The one-shot `chat()` path (document summaries, folder summaries, KB overview, the `check_models` probe) sends `reasoning_effort` from the **summary** level — absent by default (A2) — while `chat_stream()` keeps sending it from the **chat** level (default `low`), so the two surfaces are independently tunable on the wire.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/llm.py` — `_thinking_extra_body` (L49–57) becomes a pure function of the level string (drop the `Settings` read):
|
||||||
|
```python
|
||||||
|
def _thinking_extra_body(level: str) -> dict[str, str] | None:
|
||||||
|
"""The thinking level as the chat-completions extra body (phase 178).
|
||||||
|
|
||||||
|
Non-empty (trimmed) value → ``{"reasoning_effort": <value>}`` (the
|
||||||
|
OpenAI-standard top-level parameter, phase-172 A1); empty /
|
||||||
|
whitespace → ``None`` (the off switch — the parameter is ABSENT
|
||||||
|
from the request, phase-172 A2; never a literal ``"none"`` or
|
||||||
|
``"off"``).
|
||||||
|
"""
|
||||||
|
trimmed = level.strip()
|
||||||
|
return {"reasoning_effort": trimmed} if trimmed else None
|
||||||
|
```
|
||||||
|
2. `app/rag/llm.py` — `chat()`'s single-attempt body (L432): `extra = _thinking_extra_body(self.settings.llm_summary_thinking_level)`; the comment (L443–444) becomes "Phase 178 (A3/A2): the **summary** level rides as `reasoning_effort` (document summaries, folder summaries, KB overview, the `check_models` probe); the key is absent (never None) when off — the default (empty) keeps one-shot requests byte-identical to pre-phase-172."
|
||||||
|
3. `app/rag/llm.py` — `chat_stream()` (L625–628): `if (extra := _thinking_extra_body(self.settings.llm_chat_thinking_level)) is not None: …` (the comment L626–627 becomes "Phase 178 (A3/A2): the **chat** level rides as `reasoning_effort`; the key is absent (never None) when off.").
|
||||||
|
4. `app/rag/llm.py` — finalize the module docstring (L1–25, rewritten in task 01): replace the single-level paragraph with the split wording — the chat turn's level (`BOR_LLM_CHAT_THINKING_LEVEL`, default `low`) rides streaming `chat_stream`; the one-shot level (`BOR_LLM_SUMMARY_THINKING_LEVEL`, default empty = off) rides `chat` (document summaries, folder summaries, KB overview, the `check_models` probe); an empty value is the off switch (parameter absent, byte-identical to pre-phase-172); the embeddings path never carries it. Note the probe's consequence: `check_models` (L805) goes through `chat()` → off by default — it is a reachability ping, no special-casing.
|
||||||
|
5. `tests/unit/test_llm_client.py` — the mocked-client family (L1330–1440, renamed in task 01):
|
||||||
|
- The `chat()`-path tests (currently `llm_chat_thinking_level="low"` / `"low "` / `" "` at L1335–1368) are re-pointed at the **summary** kwarg: `llm_summary_thinking_level="high"` → the `chat.completions.create` call carries `extra_body={"reasoning_effort": "high"}`; `" high "` → trimmed `"high"`; `""` / `" "` → the call kwargs contain **no** `extra_body` key (never `None`, never a literal "none").
|
||||||
|
- The `chat_stream()`-path tests (L1385–1422) keep their shape: `llm_chat_thinking_level="high"` / `" high "` → `extra_body={"reasoning_effort": "high"}`; `""` / `" "` → no `extra_body` key.
|
||||||
|
- Add the cross-pins (the whole point of the split): client with `llm_chat_thinking_level="high"` + `llm_summary_thinking_level=""` (the defaults inverted) → `chat_stream()` carries `reasoning_effort="high"` while `chat()` carries no `extra_body`; the default client (chat `"low"` + summary `""`) → `chat()` is byte-identical to pre-phase-172 (no `extra_body`) while `chat_stream()` carries `reasoning_effort="low"`.
|
||||||
|
- The embeddings-path test (no `extra_body` ever) stays green untouched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_llm_client.py` (the re-pointed + extended family above; the mocked-SDK-client pattern).
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_llm_client.py -v -k "thinking" -v` — per-method `extra_body` behavior green (chat level → `chat_stream` only; summary level → `chat` only; absent-when-empty; cross-pins).
|
||||||
|
- [ ] Default-config read check: with chat `low` + summary `""`, a `chat()` request carries no `extra_body` and a `chat_stream()` request carries `reasoning_effort="low"` (the test pins above).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change in completed work (the streaming path is byte-identical at the defaults; the one-shot path is off by default — the TODO's "summary should be none").
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 04 — The per-turn line: `thinking=<chat level>`, never the summary knob
|
||||||
|
|
||||||
|
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The per-turn log line (PLAN §9, AGENTS rule 10) is pinned to the **chat** level — the level the turn actually ran at: `thinking=<trimmed chat level>` (`-` when empty) immediately before `thinking_chars=N`, with a cross-pin proving the summary level (set to a different value) never leaks into the line. No `query_log` schema change (phase-172 A6 / phase-178 A5, kept).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/chat.py` — the per-turn log line (L1272–1291) already reads the chat level after task 01's mechanical rename (`settings.llm_chat_thinking_level.strip() or "-"` L1289, format `thinking=%s` L1275): verify by reading, no functional change. Finalize the inline comment (L1289) to "phase 178 (A5): the effective **chat** thinking level, trimmed, `-` when empty — the level this turn ran at; the summary level is a one-shot concern and gains no field here."
|
||||||
|
2. `app/api/chat.py` — the module docstring (L14–19, renamed in task 01): finalize the wording — "The per-turn log line records `thinking=<level>` (the effective `BOR_LLM_CHAT_THINKING_LEVEL`, trimmed, `-` when empty) before `thinking_chars` (phase 172, A6; phase 178, A5 — the **chat** level; the one-shot `BOR_LLM_SUMMARY_THINKING_LEVEL` never appears on the line)."
|
||||||
|
3. `tests/unit/test_chat_gate.py` — the log-line family (L1545–1630, pins renamed in task 01: the default `thinking=low`, the `"high"` override, the `""` → `-` off-switch, the `" "` → `-` trim pin, the full field-order pin):
|
||||||
|
- Update the test docstring (L1550–1557) to name `BOR_LLM_CHAT_THINKING_LEVEL` / the chat level (phase 178, A5).
|
||||||
|
- Add the **cross-pin** (the split's proof on the log line): `Settings(_env_file=None, relevance_threshold=0.30, llm_chat_thinking_level="low", llm_summary_thinking_level="high")` → the line is `thinking=low thinking_chars=…` — the summary's `high` does NOT appear as the line's `thinking=` value.
|
||||||
|
- Add the inverse off-pin: `llm_chat_thinking_level=""` + `llm_summary_thinking_level="high"` → `thinking=- thinking_chars=…` (the chat off-switch logs `-` even though the summary knob is set).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_chat_gate.py` (the extended log-line family above).
|
||||||
|
- Coverage: **>90%** on `app/` (this task touches only the docstring/comment + tests — coverage must stay green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_chat_gate.py -v -k "log" -v` — the full log-line family green: default `thinking=low`, override `high`, off-switch `-`, trim `-`, field order, AND the two cross-pins (summary level never leaks into `thinking=`).
|
||||||
|
- [ ] `grep -n "BOR_LLM_SUMMARY_THINKING_LEVEL" app/api/chat.py` returns nothing (the one-shot knob has no presence on the chat turn's log line, A5).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change in completed work.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 05 — The E2E suite: `test_split_thinking_levels.py` + regression + gates + commit
|
||||||
|
|
||||||
|
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's dedicated Playwright suite (AGENTS rule 4 — one test file for the phase, run in isolation) proves the operator's acceptance in a real browser: TWO free-form thinking-level rows in the admin Settings panel (chat default `low`, summary default empty = the TODO's "none"), independently saveable/resettable with live-apply, plus the full phase gates and the atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_thinking_level.py` → `tests/e2e/test_split_thinking_levels.py` — `git mv` the phase-172 suite (the module scaffolding is the house pattern, kept: `_boot_app` / `_stop_app` / `_clean_settings_state` / the autouse `_clean` (truncates `app_settings` + `ui_settings`, deletes `e2e-`-labeled tokens, module teardown pins `app_settings` empty) / the `.env` mtime guard / the `_cookies` / `_admin_client` / `_catalog_item` helpers) and rewrite the module docstring (Source → `TODO.md` L3 this item; the test → contract mapping below; the env pins: the module app opts BACK IN over the conftest's global `BOR_SETTINGS_DB=0` pin (`env["BOR_SETTINGS_DB"] = "1"`), and pins `BOR_LLM_CHAT_THINKING_LEVEL=low` AND `BOR_LLM_SUMMARY_THINKING_LEVEL=` (empty string SET — so both badges read `env`, not `default`, and an operator's local `.env` cannot leak a different level into the suite; the house leak-guard pattern) and `BOR_LLM_CHAT_MODEL` (the "like chat model" reference)).
|
||||||
|
2. The three tests (renamed/extended from the phase-172 trio):
|
||||||
|
- `test_both_rows_render_free_form_with_defaults` — the `LLM (aipi)` group band is visible (the phase-171 band contract); a row labelled **Chat thinking level** with the `BOR_LLM_CHAT_THINKING_LEVEL` env-var display shows `low` with the `env` badge; a row labelled **Summary thinking level** with the `BOR_LLM_SUMMARY_THINKING_LEVEL` display shows an **empty** input with the `env` badge (the TODO's "none" = the off switch, A2); BOTH inputs are `type="text"` (free-form — NOT `number`/`select`, the A4 contract); the `llm_chat_model` row's input is the same control (the "like chat model" reference, one assertion pair); `tr[data-key="llm_thinking_level"]` is **absent** (the old key is gone, A1).
|
||||||
|
- `test_saved_levels_persist_without_restart` — the phase-168 §7.4 Save lifecycle per row: type `think-soft` (no allowlist — A4) into the summary row and `high` into the chat row (dirty hints appear), save both; the API catalogue (`_catalog_item`) AND the raw `app_settings` rows carry the saved values for EACH key independently; a reload of `/settings.html` still shows `think-soft` / `high` with the `DB` badges (no restart — the phase-168 contract); the independence pin: the two rows' controls are distinct and the saved values do not cross.
|
||||||
|
- `test_reset_returns_to_env_defaults` — the per-row Reset (the per-row `DELETE`): after the saves, both Resets are live; clicking the summary Reset lands the value at **empty** with the `env` badge, clicking the chat Reset lands `low` with the `env` badge (the overlay reset, task 02's contract, in the browser); the status lines show the house "… is back to the environment value." toast for each.
|
||||||
|
3. Regression set (in isolation, per the house E2E protocol): `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168) and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171) — both green (they do not pin the thinking rows — verified: no `thinking` references — but the two new rows must not break the panel/layout contracts).
|
||||||
|
4. The full gates: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage **>90%**; `uv run ruff check . && uv run pyright` clean; `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`).
|
||||||
|
5. Commit (AGENTS rule 8): ONE atomic Conventional Commit, `--no-gpg-sign` — e.g. `feat(llm): split the thinking level into a chat level (default low) and a summary level (default off)` — body: the TODO L3 source, the rename (A1), the wire split (A3), the log line (A5), the two admin rows; then move the phase dir: `git mv .agents/phases/todo/178_split_thinking_levels .agents/phases/complete/178_split_thinking_levels` and include it in the commit (the `.agents/` tree is tracked — only `.agents/phase-sessions/` + `.agents/pipeline.log` are gitignored).
|
||||||
|
6. `TODO.md` — ALREADY CLEARED by the conversion (the file holds only the `# TODO` title; verify by reading — do not re-add the item).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (AGENTS rule 4): `tests/e2e/test_split_thinking_levels.py` run **in isolation** (the three tests above).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
- Lint/types: ruff + pyright clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation** (both rows, defaults `low`/empty, free-form, save + reload persistence per key, reset to env defaults, old key absent).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` green in isolation (regression).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit including the `.agents/` phase move; `TODO.md` holds only the `# TODO` title.
|
||||||
|
- [ ] No behavior change in completed work.
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Phase 179 — Git clone server freeze: investigate the owner's suspicion, then fix it
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "A long git clone seems to freeze the server? Investigate to find out if this is true."
|
||||||
|
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-11)
|
||||||
|
**Context:** `app/api/sync.py` — `start_sync` (L421–438, `asyncio.create_task(_run_sync())` L438), `_run_sync` (L482, **async**, runs on the app's single uvicorn event loop), `await check_models(llm)` (L533 — already async HTTP probes), the short-lived `SessionLocal()` + `effective_sources(db)` (L538–540), the per-source `_check_cancelled()` boundary (L565), **`root = clone_or_pull(…)` (L577 — synchronous)**, **`file_commit_dates(root)` (L589 — synchronous)**, `sync_status` (L353–354 — "the UI polls this every 2 s"), `cancel_sync` (L442–459); `scripts/git_sync.py` — `clone_or_pull` (L96), `file_commit_dates` (L270), `run_git` (L320 — `subprocess.run` L342, **blocking**); `app/api/health.py` — `GET /api/health` (L13, mounted at `/api/health` via `app/main.py` L211); `scripts/import_docs.py` — the standalone script path (a SEPARATE process, never on the app's event loop — NOT a freeze vector); `app/api/doc_drafts.py` — `push_doc_draft` (a sync `def` → FastAPI threadpool — NOT a freeze vector); `app/api/sync.py` L233 (`from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates` — the names to monkeypatch); `tests/integration/test_sync_api.py` — the `sync_client` context-managed `TestClient` pattern (L176–179 — one app event loop across requests) + the env-fallback `BOR_GIT_SOURCES` driving (L64–65 docstring); `tests/e2e/test_sync_cancel.py` — the dedicated module-app E2E pattern (dedicated port, `file://` fixture repo via the `_git` helper, the `_wall_guard` module fixture); `tests/e2e/slow_llm.py` (not needed here — the mock LLM import is fast; the fixture's slowness is in the CLONE, not the import).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Verify the owner's suspicion with a deterministic reproduction (a long `git clone` during a sync blocks the app's event loop, stalling the entire HTTP surface — including the 2 s sync-status poll the UI relies on), record the verdict, and — the suspicion being confirmed — make the sync's git operations run OFF the event loop so the server stays fully responsive for the whole duration of a long clone (and the phase-134 cooperative cancel keeps its contract).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — the repro is in-process and deterministic (task 01).** A new `tests/integration/test_sync_git_freeze.py` follows `test_sync_api.py`'s `sync_client` pattern (context-managed `TestClient` — one app event loop across requests). `app.api.sync.clone_or_pull` and `app.api.sync.file_commit_dates` are monkeypatched (the module-level names imported at L233) with stubs that record a `time.monotonic()` start/end and sleep 3 s. The probe test measures the wall-clock latency of a `GET /api/sync/status` issued while the stub is in its sleep. The probe asserts the **PRE-fix** behavior (the probe is blocked — latency ≥ 2.5 s of the 3 s stub) → the test is GREEN before the fix and IS the demonstration of the freeze; the verdict (**CONFIRMED** / not) + mechanism + the measured latency are recorded in the test module docstring. No app/ code is touched in task 01.
|
||||||
|
- **D2 — the fix is `asyncio.to_thread`, nothing else (task 02).** Inside `_run_sync`: `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` and `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`. The module docstring and the phase-134 boundary comment (L565–576) update the semantics: a cancel registered while a git op is in flight no longer parks the EVENT LOOP — the git **worker thread** runs to completion (Python threads are not interruptible), the loop stays free (every other request, status poll included, is served), and the run still stops at the NEXT `_check_cancelled()` boundary (the phase-134 contract, unchanged). The task-01 probe flips to the post-fix assertion (probe latency < 1.5 s during the 3 s stub clone). The short-lived `SessionLocal()` / `effective_sources` DB calls stay on the loop (A3 — millisecond-scale, out of scope, noted in the docstring). `check_models` is already async HTTP — untouched.
|
||||||
|
- **D3 — the E2E pins the real pipeline (task 03).** `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): the dedicated module app on a dedicated port (the `test_sync_cancel` pattern — the session app is never started in the isolated run), mock LLM, a dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<big repo>`, the shared E2E Postgres truncated before AND after every test. The `big_git_repo` fixture is a GENERATED local repo — N commits × ~1 MiB of near-incompressible (seeded pseudo-random) `.md` files — sized at authoring so a FRESH `git clone` of it takes ≥ 5 s on the test host (the pinned constants + the calibration note live in the module; `git` on PATH is the documented prerequisite, phase 28). Contract 1: while the run is in the clone phase, `GET /api/sync/status` polls (via `page.request`, ~2 s apart like the UI) each complete in < 1.5 s and the Sources page keeps updating with no network-error state; the run lands `success` and the sentinel file is in the KB. Contract 2 (phase 134 under to_thread): a cancel registered during the clone — the checkout dir is deleted first so the run re-clones (a long git phase again) — the status endpoint stays responsive (< 1.5 s) while the in-flight git thread finishes, and the run lands the `cancelled` terminal (error null, the UI's "Sync cancelled" neutral state). The `_wall_guard` module fixture pins the budget (the `test_sync_cancel` pattern — cut the fixture size if it drifts, never the guard). `E2E_REAL_LLM=1` is skipped (mock-only — the clone, not the import, is the timing fixture).
|
||||||
|
- **NOT touched:** `scripts/import_docs.py` (standalone process), `scripts/git_sync.py` (the git layer is unchanged — only its SCHEDULING changes), `app/core/docs_push.py` / `app/api/doc_drafts.py` (sync `def` → threadpool), `check_models`, the async LLM import path, the frontend, any other endpoint.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
— (none: the app is fully functional at the head of `complete/`; this phase investigates + fixes the in-process sync path)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_repro_verdict.md` — the deterministic in-process repro (3 s stubbed clone, status-poll latency probe) that demonstrates the event-loop freeze and records the verdict in the test module docstring.
|
||||||
|
2. `02_offloop_git_ops.md` — the fix: `clone_or_pull` / `file_commit_dates` off the event loop via `asyncio.to_thread`, the docstring/boundary-note update, the probe flipped to the post-fix assertion.
|
||||||
|
3. `03_e2e_freeze_contract.md` — `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): a real long `file://` clone, the responsive-status-poll contract, the phase-134 cancel-during-clone contract, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Integration:** `tests/integration/test_sync_git_freeze.py` — the probe: pre-fix (task 01) the status poll issued during the 3 s stubbed clone is blocked (≥ 2.5 s); post-fix (task 02) it completes in < 1.5 s; the run reaches a terminal state in both (fixture cleanliness). `test_sync_api.py` + `test_import_docs_git.py` stay green (no behavior change beyond loop-freeness).
|
||||||
|
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_sync_git_freeze.py`, run **in isolation** — the clone-phase status polls responsive (< 1.5 s each, ~2 s apart), the Sources page updating, the `success` terminal + sentinel in the KB; the cancel-during-clone → `cancelled` terminal with the status endpoint responsive while the git thread finishes.
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the only `app/` change is two `await asyncio.to_thread(…)` lines + docstrings in `app/api/sync.py`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the post-fix assertion (probe < 1.5 s during the 3 s stub clone) and the verdict ("CONFIRMED" + mechanism + measured latency) recorded in the module docstring.
|
||||||
|
- [ ] `app/api/sync.py::_run_sync` runs the per-source git ops off the event loop (`asyncio.to_thread`); the phase-134 boundary note documents the worker-thread-runs-to-completion semantics; the DB calls stay on the loop (the A3 note).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (the responsive-poll contract + the cancel-during-clone contract, wall-time guard holds).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server`); the phase dir moves to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-11, roadmap confirmation)
|
||||||
|
- **A1 — scope: investigate + fix.** The TODO item is an investigation ("find out if this is true"); the owner confirmed the phase goes on to the minimal fix. If the repro in task 01 had landed NOT-confirmed, task 02 would document the no-fix decision instead — the code-path analysis (the synchronous `subprocess.run` inside the async `_run_sync`) and the repro together confirm the freeze.
|
||||||
|
- **A2 — fix mechanism: `asyncio.to_thread`.** No new packages; the single-event-loop uvicorn is unchanged; a cancel at the git phase leaves the in-flight git worker thread running to completion (Python threads are not interruptible) while the event loop is free — the phase-134 cooperative-cancel contract (stop at the next boundary, `cancelled` terminal, no corruption) is preserved.
|
||||||
|
- **A3 — out of scope: the sync's short-lived DB calls.** `SessionLocal()` / `effective_sources` (L538–540) are millisecond-scale SQLAlchemy calls — not the reported symptom; they stay on the loop and are noted in the docstring as a known, bounded, non-fix.
|
||||||
|
- **A4 — the E2E slow clone is a generated local `file://` repo.** N commits × ~1 MiB seeded pseudo-random `.md` files, sized at authoring so a fresh clone takes ≥ 5 s on the test host; deterministic, no network, `git` on PATH is the documented prerequisite (phase 28).
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — Reproduce the freeze in-process and record the verdict
|
||||||
|
|
||||||
|
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A deterministic, in-process reproduction that proves (or disproves) the owner's suspicion: while a sync's git clone is in flight, the app's event loop is blocked and the whole HTTP surface — including `GET /api/sync/status`, the endpoint the UI polls every 2 s — is stalled. The verdict (CONFIRMED / not, the mechanism, the measured latency) is recorded in the test module docstring.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/integration/test_sync_git_freeze.py` — a NEW file following `tests/integration/test_sync_api.py`'s house patterns:
|
||||||
|
- The `sync_client` fixture (L176–179 pattern): a **context-managed** `TestClient(fastapi_app)` — one app event loop across requests (the docstring L64–65 explains why the context manager is required). Admin login via the house helper (the `_login` pattern, `test_sync_api.py` L260–265).
|
||||||
|
- The git source via the **env fallback**: drive `BOR_GIT_SOURCES=file://<tmp repo>` the way `test_sync_api.py` does (a fresh `Settings` on the resolver's module; the dev `.env` never leaks in). The tmp repo is a real one-commit git repo built with the house `_git` helper pattern (`tests/e2e/test_sync_cancel.py` L156–195 shape: `git init -q`, `add -A`, `-c user.email=… -c user.name=… -c commit.gpgsign=false commit -qm`); the stub never actually clones it, so the repo's content is irrelevant — one small `.md` file suffices.
|
||||||
|
- **The stubs (the timing fixture):** `monkeypatch.setattr("app.api.sync.clone_or_pull", …)` and `monkeypatch.setattr("app.api.sync.file_commit_dates", …)` — the module-level names imported at `app/api/sync.py` L233 (patching the `scripts.git_sync` originals would not affect the runner). The `clone_or_pull` stub: on entry sets a `threading.Event` (`clone_started`) and records `time.monotonic()`, sleeps **3.0 s** (simulating a long clone), records the end, and returns the tmp repo's `Path`; the `file_commit_dates` stub returns `{}`. Both are plain sync functions — the point is that PRE-fix they run ON the event loop.
|
||||||
|
- **The probe test** `test_long_git_clone_freezes_the_event_loop`: `POST /api/sync` (admin) → wait for `clone_started` (bounded wait, ~10 s) → `t0 = time.monotonic()`, `client.get("/api/sync/status")`, `elapsed = time.monotonic() - t0` → **assert `elapsed >= 2.5`** (the probe request, issued into the middle of the 3 s stubbed clone, is BLOCKED — it cannot be served until the stubbed clone returns and the event loop is free again). That assertion IS the freeze demonstration and it is GREEN pre-fix. Then poll `GET /api/sync/status` to the terminal state (the stub returns, so the run completes: `success` with the import, or a clean terminal — whichever the stubbed root yields; pin whichever the suite shows and document it) so the fixture is left clean for the next test.
|
||||||
|
- **The module docstring (the verdict record):** the owner's TODO item quoted (L3), the code path (`POST /api/sync` → `asyncio.create_task(_run_sync())` — `app/api/sync.py` L438/L482 → the synchronous `clone_or_pull` L577 → `run_git`'s blocking `subprocess.run` — `scripts/git_sync.py` L342 — with no `to_thread`/`create_subprocess_exec` anywhere in `app/` or `scripts/`), the measured probe latency (e.g. "the status probe took ≈3.0 s of the 3.0 s stubbed clone — the event loop was stalled for the full git op"), and the verdict line: **CONFIRMED** (or NOT — if the probe came back fast, stop and flag it in the docstring + the phase report rather than proceeding to a fix).
|
||||||
|
- `GET /api/health` (`app/api/health.py` L13, mounted at `/api/health`) may be used as a SECOND probe in the same test (the same assertion — it is blocked too: any endpoint on the loop stalls); keep the status probe primary (it is the endpoint the UI's 2 s poll depends on).
|
||||||
|
2. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py -v` — green (the probe is green with the PRE-fix assertion: it documents the freeze, it is not a failing test), and `uv run pytest` — the full suite green (no other test is affected: the stubs live inside this one test's monkeypatch).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the probe family above (the blocked-latency assertion, the terminal-state assertion, the admin/anonymous gating falls out of the house `sync_client` pattern — add the anonymous-403 pins if the house pattern does not already carry them in this file).
|
||||||
|
- Coverage: **>90%** on `app/` (this task is test-only — `app/` code is untouched, so the suite's coverage must simply stay green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the pre-fix assertion (`elapsed >= 2.5` for the status probe issued mid-clone).
|
||||||
|
- [ ] The module docstring records: the TODO item, the code path, the measured latency, and the verdict (**CONFIRMED** — expected, per the code-path analysis).
|
||||||
|
- [ ] `uv run pytest` green.
|
||||||
|
- [ ] No `app/` code change in this task.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — The fix: the sync's git operations run off the event loop
|
||||||
|
|
||||||
|
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
With the freeze confirmed (task 01's verdict), make the server stay fully responsive for the whole duration of a long `git clone` / `git pull` (and the `file_commit_dates` walk): the sync's per-source git operations run in a worker thread via `asyncio.to_thread`, the phase-134 cooperative-cancel contract is preserved, and the task-01 probe flips to pin the post-fix behavior.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py` — `_run_sync` (L482), the per-source git block (L566–589):
|
||||||
|
- L577: `root = clone_or_pull(…)` → `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` (the three-argument call, phase 121/135, unchanged — only its scheduling changes).
|
||||||
|
- L589: `doc_dates_by_root[str(root)] = file_commit_dates(root)` → `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`.
|
||||||
|
- `import asyncio` if not already imported at module level (check first — the module already uses `asyncio.create_task` at L438, so `asyncio` is imported; verify, do not duplicate).
|
||||||
|
- **The phase-134 boundary note (L565–576)** — update the comment: a cancel registered while a git op is in flight no longer parks the EVENT LOOP; the git **worker thread** runs its blocking `subprocess` to completion (Python threads are not interruptible — `_task.cancel()` unblocks the run at the `to_thread` await point / the run stops at the NEXT `_check_cancelled()` boundary), while every other request — status polls included — is served immediately. The observable contract is unchanged (phase 134: stop at the next boundary, `cancelled` terminal, no corruption).
|
||||||
|
- **The module docstring** — add the freeze note: pre-phase-179 the synchronous `clone_or_pull` / `file_commit_dates` (their `subprocess.run` calls — `scripts/git_sync.py` L342) ran ON the app's event loop, stalling the whole HTTP surface (including the 2 s `sync_status` poll) for the duration of a long clone (the TODO L3 suspicion — confirmed by `tests/integration/test_sync_git_freeze.py`); phase 179 runs them off the loop via `asyncio.to_thread`. Include the **A3 note**: the short-lived `SessionLocal()` / `effective_sources` DB calls (L538–540) remain on the loop — millisecond-scale, a known bounded non-fix, deliberately out of scope.
|
||||||
|
- `check_models` (L533) is untouched (already async HTTP probes). The kind-local branch (L590+) is untouched (no subprocess in it).
|
||||||
|
2. `tests/integration/test_sync_git_freeze.py` — flip the task-01 probe to the post-fix assertion: the status probe issued into the middle of the 3 s stubbed clone now completes in **< 1.5 s** (the event loop is free while the stubbed clone runs in the worker thread); keep the terminal-state assertion. Update the module docstring: the verdict section now reads CONFIRMED (task 01) → FIXED (phase 179, `asyncio.to_thread`), with the post-fix measured latency recorded (the probe ≈ instant, e.g. "the status probe took ≈0.01 s while the 3.0 s stubbed clone ran in the worker thread").
|
||||||
|
3. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` — all green (the flipped probe green; the sync families show no behavior change beyond loop-freeness), then `uv run pytest` — the full suite green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the flipped probe (post-fix: status probe < 1.5 s during the 3 s stubbed clone) + the `test_sync_api.py` / `test_import_docs_git.py` regression set.
|
||||||
|
- Coverage: **>90%** on `app/` (the only `app/` change is two `await asyncio.to_thread(…)` lines + docstring/comment in `app/api/sync.py` — the suite's coverage must stay green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the POST-fix assertion (probe < 1.5 s during the 3 s stub clone) and the docstring verdict updated (CONFIRMED → FIXED, post-fix latency recorded).
|
||||||
|
- [ ] `app/api/sync.py::_run_sync` awaits `asyncio.to_thread` for BOTH git ops; the phase-134 boundary note + the module docstring document the worker-thread semantics and the A3 DB note; `check_models` untouched.
|
||||||
|
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` green; `uv run pytest` green.
|
||||||
|
- [ ] No behavior change in completed work beyond loop-freeness (the sync pipeline's sequence, errors, and cancel terminals are unchanged).
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 03 — E2E: the status poll stays responsive through a real long clone (the phase's dedicated Playwright suite)
|
||||||
|
|
||||||
|
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Pin the fix against the REAL pipeline (the owner's exact suspicion, end to end): a sync whose first action is a genuinely long `git clone` (a generated local `file://` repo) must keep the whole HTTP surface — the 2 s `GET /api/sync/status` poll and the Sources page — responsive for the entire duration of the clone, the run must land `success`, and the phase-134 cancel-during-clone contract must hold under the off-loop scheduling. Then run the full gates and commit the phase.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_sync_git_freeze.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped — the CLONE, not the import, is the timing fixture), following `tests/e2e/test_sync_cancel.py`'s house pattern point for point:
|
||||||
|
- **Isolation:** a per-module app on a **dedicated port** (the conftest session app is never started in this isolated run), the mock LLM on the conftest port, a dedicated `BOR_SOURCES_DIR` for the fixture repo's checkout, `BOR_GIT_SOURCES=file://<big repo>` (the env-fallback row — the `test_sync_cancel` L256 shape), the shared E2E Postgres truncated before AND after every test.
|
||||||
|
- **The `big_git_repo` fixture (the timing fixture — A4):** a module-scoped helper (the `_git` pattern, `test_sync_cancel.py` L156–195) that generates N commits, each adding one ~1 MiB `notes/note-<k>.md` whose body is **seeded pseudo-random** text (deterministic — a fixed-seed generator, e.g. `random.Random(k)` over a word list — so the pack is near-incompressible and the clone time is stable; plain repeated text would compress away and the clone would be fast). **Calibrate at authoring time**: pick N (start ~200) so a FRESH `git clone file://<repo>` takes **≥ 5 s** on the test host (measure it during authoring, record the measured clone time in the module docstring, and pin N + the file size as module constants with the calibration note). The last file carries the sentinel marker (the `test_sync_cancel` sentinel shape).
|
||||||
|
- **Test 1 — `test_status_polls_responsive_during_long_clone`:** admin login (the house helper) → trigger the sync (the UI's sync button, or the API via `page.request` — the `test_sync_cancel` L100-ms `page.request` poll shape) → while the run reports `running` and is still in the clone phase (the status's `current_file` is null / no file positions — the import has not started), poll `GET /api/sync/status` via `page.request` at ~2 s intervals (the UI's poll cadence): **every poll must complete in < 1.5 s** (the UI's 2 s poll budget — pre-fix the polls issued during the clone would stall for the whole clone and this is the owner's frozen server, observable as hung status polls / a stalled Sources page) → the Sources page keeps updating (no network-error state, the status label keeps advancing) → the run lands the `success` terminal → the sentinel file is in the KB (the house drill-down/`GET /api/docs` assertion shape).
|
||||||
|
- **Test 2 — `test_cancel_during_clone_reaches_terminal`:** first DELETE the fixture repo's checkout dir under `BOR_SOURCES_DIR` (the run re-clones — a long git phase again), trigger a second sync, and while it is in the clone phase register the cancel (the UI's Cancel control or `POST /api/sync/cancel` via `page.request`) → **the status endpoint stays responsive** (the `page.request` polls during the in-flight git worker thread all complete in < 1.5 s — the A2 observable: the git thread runs to completion in the background while the loop serves requests) → the run lands the `cancelled` terminal (the phase-134 contract: `error: null`, the in-flight positions cleared, the UI's neutral "Sync cancelled" state — no error banner, the main button retry-ready).
|
||||||
|
- **The `_wall_guard` module fixture** (the `test_sync_cancel` pattern): budget the module (two syncs, the first a full clone + mock import of N files, the second a re-clone + cancel) — budget it at the measured clone time + import span + setup, pin the guard; **if the suite drifts, cut N, not the guard.**
|
||||||
|
- The module docstring: the owner's TODO item quoted, the fix (task 02's `asyncio.to_thread`), the fixture calibration (N, the measured clone time), the timing paragraph, the run-in-isolation command (`uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov`), the environment prerequisites (DB up: `podman compose up -d db`; `git` on PATH — the phase-28 documented prerequisite), the test→observable mapping (the house shape).
|
||||||
|
2. **The full gates** (the phase's final pass):
|
||||||
|
- `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` — green **in isolation**.
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — the full suite green, `app/` coverage **>90%**.
|
||||||
|
- `uv run ruff check . && uv run pyright` — clean.
|
||||||
|
- The regression spot-check in isolation: `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` (the phase-134 cancel E2E — the contract this phase must not have broken; run it AFTER the new suite so the DB truncate ordering is clean).
|
||||||
|
3. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/api/sync.py`, the two test files, and the `.agents/` tree — e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server` (the commit body names the TODO L3 investigation + the verdict). Move the phase dir to `.agents/phases/complete/179_git_clone_server_freeze/`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the dedicated suite above (the responsive-poll contract + the cancel-during-clone contract, the `test_sync_cancel` regression spot-check).
|
||||||
|
- Coverage: **>90%** on `app/`; ruff + pyright clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (every status poll < 1.5 s during the real clone, `success` terminal + sentinel in the KB, cancel-during-clone → `cancelled` terminal with the status endpoint responsive, wall-time guard holds).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` green in isolation (the phase-134 contract intact).
|
||||||
|
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# Phase 180 — Markdown inline links, inline images, and the full heading ladder
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 ("Markdown [links](some-address) aren't being render correctly and show up as plain text"), L5 ("Quad-level headers (####) aren't being rendered and show up as plain text") and the in-answer half of L3 ("Need SVG render support on the front end when citing … source")
|
||||||
|
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
|
||||||
|
**Context:** `frontend/assets/markdown.js` — the SHARED escape-first renderer used by all three pages (`index.html` chat bubbles + thinking, `document.html` viewer, `shared.html`) and by `document-modal.js` through `document.js`'s `renderDocument`. Pipeline order: step 1 fence protection (`\u0000CODE n\u0000`) → step 1b GFM pipe-table protection (`\u0000TABLE n\u0000`, phase 44) → step 2 `text = inline(escapeHtml(text))` → step 2b blockquotes (phase 138) → step 3 the paragraph pass (split on `\n{2,}`, block guard `/^<(h\d|ul|ol|pre|li|blockquote)/`) → step 4 restore. The `inline(s)` chain (the single source of every inline transform, also applied per table cell) is today, in order: code span → `**strong**` → `*em*` → `^### `→`<h4>` → `^## `→`<h3>` → `^# `→`<h3>` → `^\s*[-*] `→`<li>` → `<ul>` grouping → `^\d+\. `→`<li>`. Three verified gaps: **no link transform exists at all** (`renderMarkdown("[a](https://x)")` returns the text verbatim inside a `<p>`), **no image transform exists at all** (`` prints literally), and the heading ladder stops at `###` — `####`/`#####`/`######` match no rule and print as plain text (each rule requires a space after its hashes, so no rule partially eats a deeper level). `frontend/assets/styles.css` — the `:root` palette (`--ink`, `--ink-soft`, `--line`, `--surface`, `--bg`, the `--brand-*` family, `--radius*`) plus the precedent rule families: `.md-table`/`.md-table-wrap` (L539-557), `.md-quote` + the `.msg.user .bubble .md-quote` inversion (L561-588), `.source-image-img` / `.doc-image-img` (L981-1027, L5611-5640) for image treatment. `app/core/security_headers.py::CSP` (L67-68) — `default-src 'self'; base-uri 'none'; frame-ancestors 'none'; img-src 'self' data:`: served-path and `data:` images are ALREADY permitted, no CSP change is needed or allowed here. Test patterns: `tests/unit/test_markdown_blockquotes.py` (source pins + a `HAVE_NODE`-skipped node behaviour harness + a byte-identity fixture string) and `tests/e2e/test_markdown_blockquotes.py` (dedicated fixture tree `tests/fixtures/quote_docs/`, mock LLM trigger, chat + viewer + localStorage-restore assertions).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`[text](url)` renders as a real, safe `<a>`, `` renders as a lazy `<img>`, and `####`–`######` render as `<h5>`/`<h6>` — in chat answers, the thinking block, saved/shared chats, the document viewer and the document modal, from the ONE shared renderer. Every input that contains none of the three constructs renders byte-identical to today, so the phase-44 table suite, the phase-138 blockquote suite and every renderer-dependent unit suite stay green unedited.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — one chain, one place (tasks 01–03).** All three fixes are edits to the same `inline(s)` chain in `frontend/assets/markdown.js`. Nothing else in the pipeline moves: fences (step 1) and tables (step 1b) keep their precedence, and because a table cell goes through `inline(escapeHtml(cell))`, links/images/deep headings inside a cell work for free (pinned in task 05).
|
||||||
|
- **D2 — escape-first is the security boundary, so NEVER unescape.** `inline()` always receives HTML-escaped text. The captured URL may therefore contain `&`/`'`/`"` — that is *valid* in an `href`/`src` and must be passed through verbatim (never decoded, never re-escaped). Any attempt to decode would re-open the injection surface the renderer exists to close.
|
||||||
|
- **D3 — links (task 01).** `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace inside the URL), applied AFTER the `*em*` rule and BEFORE the heading ladder. The URL is validated by an allowlist *function*, not a regex-in-replacement: allow `^https?://`, `^mailto:`, `^/`, `^#`, `^\.{1,2}/`; anything else (`javascript:`, `data:`, `vbscript:`, `file:`, bare `javascript:` variants) returns the ORIGINAL matched text unchanged — it stays visible as inert escaped text, never a live element. Absolute `http(s)://` links get `target="_blank" rel="noopener noreferrer"`; `mailto:`, root-relative and `#fragment` links get neither (a same-origin target in the same tab is the correct behaviour). Output marker: `<a class="md-link" …>`.
|
||||||
|
- **D4 — images (task 02), matched BEFORE the link rule.** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="…" alt="…" loading="lazy">`. The image rule runs FIRST so the `!` is never orphaned into `!<a>`. Same allowlist discipline: `/`-rooted served paths, `http(s)://` and `data:image/*` render; anything else stays inert text. `alt` is ALWAYS emitted (empty allowed) — an `<img>` without `alt` fails WCAG. Linked images (`[](url)`) get their OWN rule ahead of both (locked A6): `<a class="md-link" …><img class="md-img" …></a>` — otherwise the image rule leaves a stray `[` inside the link text.
|
||||||
|
- **D5 — the heading ladder (task 03, locked A4).** Insert `/^#### (.*)$/gm → <h5>$1</h5>` and `/^#{5,6} (.*)$/gm → <h6>$1</h6>` immediately BEFORE the existing `/^### /` rule (deeper-first is the readable order; the rules cannot cross-match because every rule requires a space after its hashes). `#` and `##` keep mapping to `<h3>` — existing behaviour, deliberately unchanged. The step-3 guard already matches `h\d`, so no paragraph-guard change is needed (pinned, not assumed).
|
||||||
|
- **D6 — CSS (task 04).** `.md-link` (underlined; `color: var(--brand-ink)` — 9.0:1 on `--surface`, the file's documented AA-safe brand text token — with `--brand-hover` on `:hover`, the shared 3px `--brand` `:focus-visible` ring already used at L163, and the `.msg.user .bubble .md-link { color: var(--bg) }` inversion following the `.md-quote` precedent at L588, 5.2:1 on the brand bubble), `.md-img` (`max-width:100%; height:auto; object-fit:contain` + the `.source-image-img` border/radius/surface language), and the `.doc-md` / `.bubble` / `.thinking-text` `h5`/`h6` type scale. Palette variables ONLY, no new hue (the monochrome invariant), every non-obvious color carrying its `/* n.n:1 on --x (AA) */` comment.
|
||||||
|
- **D7 — byte-identity is the regression contract (task 05).** A checked-in fixture string with no links, no image refs and no `####`+ heading, plus its byte-exact pre-phase renderer output (the phase-138 `NO_QUOTE_INPUT`/`NO_QUOTE_PRE_PHASE_OUTPUT` shape), pinned in the node behaviour harness.
|
||||||
|
- **Known, documented limits (recorded in the module docstring, not fixed):** inline code spans are NOT placeholder-protected (pre-existing: `**bold**` inside backticks already transforms), so markdown syntax inside an inline code span transforms too; a URL containing a literal `)` or whitespace is a miss; a size-less SVG (`viewBox` only, no `width`/`height`) in `.md-img` paints at the replaced-element default box — the vector sizing contract belongs to phase 181.
|
||||||
|
- **NOT touched:** `app/**` (no server change in this phase), the CSP string, `document.js`'s `renderDocument` structure, `app.js`'s chip/figure builders, `styles.css` families outside `.md-*` + the `h5`/`h6` scale.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `179_git_clone_server_freeze` (todo) — numbering/pipeline order only. The code dependency is the shared renderer at the head of `complete/` (phases 44 tables, 138 blockquotes).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_inline_links.md` — the `[text](url)` → `<a class="md-link">` transform with the protocol allowlist in `markdown.js`.
|
||||||
|
2. `02_inline_images.md` — the `` → `<img class="md-img">` transform (ahead of the link rule) + the linked-image rule.
|
||||||
|
3. `03_heading_ladder.md` — `####` → `<h5>`, `#####`/`######` → `<h6>` in the ladder.
|
||||||
|
4. `04_styles_markdown_inline.md` — `.md-link`, `.md-img` and the `h5`/`h6` type scale in `styles.css`.
|
||||||
|
5. `05_unit_and_e2e.md` — `tests/unit/test_markdown_inline_links_images.py` + the phase's dedicated Playwright suite `tests/e2e/test_markdown_inline_links_images.py`, the full gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_markdown_inline_links_images.py` (new, the phase-138 pattern): source pins (the chain order code → strong → em → **image → linked-image → link** → headings → lists; the allowlist function; the exact `md-link`/`md-img` markers; no `unescape`/`decodeURIComponent` anywhere in the file; the `.md-link`/`.md-img` CSS rules reference palette vars only) + node behaviour (link renders with `target`/`rel`; `javascript:`/`data:` inert; relative + `#fragment` + `mailto:` link without `target`; `&` in a query string survives verbatim; image renders with `alt` (empty alt allowed), disallowed src inert; linked image wraps without a stray `[`; `####`→`<h5>`, `#####`/`######`→`<h6>`, `###`→`<h4>` unchanged; a link inside a fence stays code; a link/image inside a table cell renders; byte-identity fixture).
|
||||||
|
- **E2E (AGENTS rule 4):** `tests/e2e/test_markdown_inline_links_images.py`, run **in isolation** (`uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`, DB up).
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — this phase adds no `app/` code, so the full suite staying green IS the gate.
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `[text](url)` renders as `<a class="md-link">` in a chat answer, the thinking block, a restored shared chat, the viewer and the document modal; `javascript:`/`data:` URLs stay visible as inert text
|
||||||
|
- [ ] `` renders as `<img class="md-img" loading="lazy">` with an `alt` present; a non-allowlisted src stays inert text
|
||||||
|
- [ ] `####` renders `<h5>`, `#####`/`######` render `<h6>`; `#`/`##`/`###` are unchanged
|
||||||
|
- [ ] a link/image/`####` inside a fenced code block is not transformed; a cell-level link renders in a `.md-table`
|
||||||
|
- [ ] an input with none of the three constructs renders byte-identical (phase-44 + phase-138 suites pass unedited)
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A4 — `####` → `<h5>`; `#####` and `######` both → `<h6>`; `#`/`##` keep rendering as `<h3>` (owner-confirmed 2026-10-14, roadmap confirmation).**
|
||||||
|
- **A5 — link/image URL allowlist = `http:`, `https:`, `mailto:` (links), `/`-rooted + `./`/`../` relative paths, `#fragment`, and `data:image/*` (images). Everything else renders as inert text, never a live element (owner-confirmed 2026-10-14, roadmap confirmation).**
|
||||||
|
- **A6 — `[](url)` renders as a link wrapping the image (an explicit rule ahead of the two others), not as text with a stray bracket (assistant decision, surfaced here so the executor does not re-decide it).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer"
|
||||||
|
```
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — The link transform in the shared renderer
|
||||||
|
|
||||||
|
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4` — "Markdown [links](some-address) aren't being render correctly and show up as plain text"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`renderMarkdown` turns `[text](url)` into `<a class="md-link" …>text</a>` in the shared `inline()` chain — through an explicit URL allowlist, escape-first, with every existing transform's output byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/markdown.js` — in `const inline = (s) => …`, insert one link `.replace()` **after** the `*em*` rule and **before** the `^### ` heading rule: `/(\[)([^\]]*)(\]\()([^)\s]+)(\))/g` written plainly as `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace in the URL). The replacement is a FUNCTION, not a string, so the URL passes the allowlist check:
|
||||||
|
- `allowLinkUrl(url)` — returns the url for `^https?://`, `^mailto:`, `/`-rooted (`^/`), `^\.{1,2}/` relative, and `^#` fragment URLs; returns `null` for anything else (`javascript:`, `data:`, `vbscript:`, `file:`, `blob:`) — LOCKED A5;
|
||||||
|
- allowed → `<a class="md-link" href="<url>"[ target="_blank" rel="noopener noreferrer"]>text</a>`; the `target`/`rel` pair is added ONLY for `^https?://` (a `mailto:`/relative/fragment link stays in the same tab);
|
||||||
|
- not allowed → return the ORIGINAL matched substring untouched, so it renders as the inert escaped text it is today (never a live element, never dropped).
|
||||||
|
2. `frontend/assets/markdown.js` — pass the captured URL through VERBATIM: the chain's input is already `escapeHtml`-ed, so `&`/`'`/`"` in a query string are already attribute-safe. No `decodeURIComponent`, no manual unescape, no re-escape (design D2 — this is the security boundary).
|
||||||
|
3. `frontend/assets/markdown.js` — module docstring: add the phase-180 paragraph (the chain order is now code → strong → em → **linked-image → image → link** (tasks 02 adds the first two ahead of this one) → headings → lists), the allowlist (LOCKED A5), the `target`/`rel` rule, and the documented limits (a URL containing a literal `)` or a space is a miss; inline code spans are not placeholder-protected, so markdown syntax inside `` `…` `` transforms — the pre-existing `**strong**` behaviour, unchanged here).
|
||||||
|
- ASSUMPTION: link text may itself contain inline markup (`**bold**`, `*em*`, `` `code` ``) — those rules run FIRST in the chain, so the text is already transformed when the link rule wraps it; no extra handling.
|
||||||
|
- ASSUMPTION: autolinks (`<https://x>`) and reference-style links (`[text][ref]`) stay out of scope — the TODO names the inline `[text](url)` form only.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: pinned by task 05's file (source pins: the link rule sits between the `em` and heading rules, the marker is exactly `<a class="md-link"`, the file contains no `decodeURIComponent`/`unescape`; node behaviour: absolute link with `target="_blank" rel="noopener noreferrer"`, `mailto:`/relative/`#` link without them, `javascript:` + `data:` inert, `?a=1&b=2` keeping `&`, an empty link text, a link inside a table cell).
|
||||||
|
- Coverage: n/a (frontend) — the suite + the pins are the gate.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] node repro: `renderMarkdown("see [the docs](https://example.com/a?x=1&y=2)")` → a `<p>` containing `<a class="md-link" href="https://example.com/a?x=1&y=2" target="_blank" rel="noopener noreferrer">the docs</a>`
|
||||||
|
- [ ] node repro: `renderMarkdown("[click](javascript:alert(1))")` contains no `<a` and no `href`
|
||||||
|
- [ ] every other existing construct (`strong`, `em`, `code`, `#`–`###`, lists, tables, fences, blockquotes) renders byte-identical to the pre-task renderer
|
||||||
|
- [ ] `uv run pytest` green (the renderer-dependent unit suites unedited); `uv run ruff check .` clean
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — The image transform (and the linked-image rule) in the shared renderer
|
||||||
|
|
||||||
|
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … Also would be nice to click an svg and see a preview in the RAG tab rather than the xml" (the in-answer rendering half; the document-preview half is phase `181_svg_document_preview`)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`renderMarkdown` turns `` into a lazy `<img class="md-img">` (and `[](url)` into that image wrapped in a link), through the same URL allowlist as task 01 — so a diagram referenced inside markdown paints instead of printing its reference text.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules **ahead of** task 01's link rule, in this order:
|
||||||
|
1. **linked image first:** `/\[!\[([^\]]]*)\]\(([^)\s]+)\)\]\(([^)\s]+)\)/g` → `<a class="md-link" href="url"[ target="_blank" rel="noopener noreferrer"]><img class="md-img" src="src" alt="alt" loading="lazy"></a>` (LOCKED A6 — without this rule the image rule consumes the inner `` and the link rule then leaves a stray `[` in the link text). Both URLs are allowlist-checked (`allowLinkUrl` for the href, `allowImageUrl` for the src); either failing → the original text is returned untouched.
|
||||||
|
2. **image second:** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="src" alt="alt" loading="lazy">`.
|
||||||
|
2. `frontend/assets/markdown.js` — add `allowImageUrl(url)` next to task 01's `allowLinkUrl`: allows `/`-rooted served paths (`/api/documents/<id>/image`, `/assets/…`), `^.{0,2}/` relative paths, `^https?://` and `^data:image/` — the `data:` case exists because the app's own CSP (`app/core/security_headers.py::CSP`, `img-src 'self' data:`) already permits `data:` images; everything else (`javascript:`, `file:`, non-image `data:`) returns `null` → the original escaped text stays visible and inert (LOCKED A5).
|
||||||
|
3. `frontend/assets/markdown.js` — `alt` is ALWAYS emitted, escaped-passthrough like the URL, empty allowed (`alt=""` for a decorative image) — an `<img>` with no `alt` attribute fails WCAG 1.1.
|
||||||
|
4. `frontend/assets/markdown.js` — docstring: record the final chain order (code → strong → em → **linked-image → image → link** → headings → lists), the image allowlist, and the documented limit that a size-less SVG (`viewBox` only) in `.md-img` paints at the replaced-element default box (vector sizing is phase 181's `04_svg_preview_render.md` contract).
|
||||||
|
- ASSUMPTION: no `onerror` handler is added — the renderer emits an HTML STRING into `innerHTML`-style surfaces and the no-CDN/CSP posture forbids inline handlers; a broken src degrades to the browser's alt text (the house pattern for the shared renderer). The JS `onerror` fallbacks live in `document.js`/`app.js`, which do not route through this function.
|
||||||
|
- ASSUMPTION: title syntax (``) is a miss (the URL capture forbids whitespace) — out of scope.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: pinned by task 05 (chain order pins the two image rules before the link rule; markers `<img class="md-img"` / `loading="lazy"`; `alt=` always present; node behaviour: served-path image, `data:image/*` image, disallowed-protocol image inert, linked image produces `<a class="md-link">` wrapping `<img class="md-img">` with no stray bracket, empty alt allowed, an image inside a table cell renders).
|
||||||
|
- Coverage: n/a (frontend).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] node repro: `renderMarkdown('')` → `<p><img class="md-img" src="/api/documents/aaaa/image" alt="topology" loading="lazy"></p>`
|
||||||
|
- [ ] node repro: `renderMarkdown('[](https://example.com)')` → one `<a class="md-link"` containing one `<img class="md-img"`, and the output contains no stray `[` or `]`
|
||||||
|
- [ ] node repro: `renderMarkdown(')')` contains no `<img` and no `src=`
|
||||||
|
- [ ] input with no image reference renders byte-identical; `uv run pytest` green
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — Extend the heading ladder to `####`, `#####`, `######`
|
||||||
|
|
||||||
|
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:5` — "Quad-level headers (####) aren't being rendered and show up as plain text"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The ladder renders `#### x` as `<h5>x</h5>` and `##### x` / `###### x` as `<h6>x</h6>`; `#`, `##` and `###` keep their existing output exactly.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules immediately BEFORE the existing `/^### (.*)$/gm → "<h4>$1</h4>"` rule (deeper-first, so the ladder reads top-down by depth):
|
||||||
|
- `/^#### (.*)$/gm` → `"<h5>$1</h5>"`
|
||||||
|
- `/^#{5,6} (.*)$/gm` → `"<h6>$1</h6>"` (LOCKED A4: both `#####` and `######` map to `<h6>` — the app's type scale stops at two levels below `###`, no `<h7>` exists in HTML)
|
||||||
|
The rules cannot cross-match each other or the existing ones because every heading rule requires a space after its hash run (`####` never matches `/^### /`).
|
||||||
|
2. `frontend/assets/markdown.js` — leave `/^## /` and `/^# /` mapping to `<h3>` UNTOUCHED (LOCKED A4 — the existing flattening of `#`/`##` to `<h3>` is deliberate; the phase adds levels below `###`, it does not re-rank the existing ones).
|
||||||
|
3. `frontend/assets/markdown.js` — the step-3 paragraph guard `/^<(h\d|ul|ol|pre|li|blockquote)/` already matches `h5`/`h6`, so NO change is needed there: pin it with a unit assertion rather than editing it.
|
||||||
|
4. `frontend/assets/markdown.js` — docstring: record the ladder (`#`/`##` → `h3`, `###` → `h4`, `####` → `h5`, `#####`/`######` → `h6`) so the deliberate flattening is documented, not re-discovered.
|
||||||
|
- ASSUMPTION: a heading line must start at column 0 of its line (the `m` flag, the existing rule shape) — an indented `#### x` stays plain text, matching `^### ` today.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: pinned by task 05 (source pins: the two new rules sit before `/^### /`; the guard regex unchanged; node behaviour: `#### A` → `<h5>A</h5>`, `##### A`/`###### A` → `<h6>A</h6>`, `### A` → `<h4>A</h4>` unchanged, `## A`/`# A` → `<h3>` unchanged, a `####` line inside a fence stays code, a `####` heading immediately followed by a paragraph is not wrapped in `<p>`).
|
||||||
|
- Coverage: n/a (frontend).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] node repro: `renderMarkdown("#### Deep")` → `<h5>Deep</h5>` (no `<p>` wrapper, no literal `####`)
|
||||||
|
- [ ] node repro: `renderMarkdown("##### Deeper\n###### Deepest")` → two `<h6>` elements
|
||||||
|
- [ ] node repro: a `####` line inside a ``` fence renders inside `<pre><code>` unchanged
|
||||||
|
- [ ] existing heading output byte-identical; `uv run pytest` green (phase-44/138 suites unedited)
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 04 — Styling for `.md-link`, `.md-img` and the `h5`/`h6` type scale
|
||||||
|
|
||||||
|
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4–5` (link + heading rendering), `TODO.md:3` (inline image rendering)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The three new elements are legible and theme-correct in every renderer context (chat bubble — brain and user, the thinking block, the shared page, the viewer, the document modal), built from the existing `:root` palette only.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — `.md-link` next to the `.md-quote` family (L561-588): `color: var(--brand-ink)` (the file's documented AA-safe brand text token, 9.0:1 on `--surface`), `text-decoration: underline` (a link must not rely on colour alone — WCAG 1.4.1), `text-underline-offset: 2px`, `:hover { color: var(--brand-hover) }`. The global `:focus-visible { outline: 3px solid var(--brand) }` (L162-166) already covers the link — do NOT add a competing focus rule, and do not reset the outline.
|
||||||
|
2. `frontend/assets/styles.css` — `.msg.user .bubble .md-link { color: var(--bg); }` — the exact inversion the `.md-quote` precedent applies at L588 (the user bubble is a `--brand` fill: `--bg` on `--brand` is 5.2:1, AA). Add the same inversion for `:hover` if the default `--brand-hover` fails contrast on the brand fill — a colour-mix toward `--bg` is acceptable, a new hue is not.
|
||||||
|
3. `frontend/assets/styles.css` — `.md-img`: `display: inline-block; max-width: 100%; height: auto; object-fit: contain;` plus the `.source-image-img` card language (`background: var(--surface)`, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`) and `margin: 0.4rem 0`. No fixed width/height — an oversized screenshot must never widen the page.
|
||||||
|
4. `frontend/assets/styles.css` — the deep-heading type scale, ONE rule set for all renderer contexts (no chat-specific duplication): `.doc-md h5, .msg .bubble h5, .thinking-text h5 { margin: 0.8rem 0 0.3rem; font-size: 0.95rem; }` and the `h6` sibling at `0.88rem`, following the existing `.doc-md h3` / `.doc-md h4` rhythm (L5282-5283) — the scale stays monotonic (`h3` 1.15 → `h4` 1.0 → `h5` 0.95 → `h6` 0.88) and every level keeps a visible size difference from body text.
|
||||||
|
5. `frontend/assets/styles.css` — comment block above each rule in the house style: the phase number, the TODO source line, the contrast math for every non-obvious colour, and the "static content, no animation" note (the background-motion invariant).
|
||||||
|
- ASSUMPTION: `h5`/`h6` in the chat bubble get explicit sizing (they inherit UA defaults today, which are larger than `0.95rem`) so a deep heading in an answer cannot out-shout the answer text.
|
||||||
|
- ASSUMPTION: no `.md-img--vector` modifier here — vector (viewBox-only SVG) sizing belongs to `181_svg_document_preview/04_svg_preview_render.md`, which knows the document's format.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: pinned by task 05 — the `.md-link`, `.md-img`, `h5`/`h6` rules exist and reference ONLY `var(--…)` palette tokens (no hex literals, no `rgb(`) — the established `:root`-drift pin pattern used by the blockquote/table CSS tests; the `.msg.user .bubble .md-link` inversion is present.
|
||||||
|
- Coverage: n/a (frontend).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a link in a brain bubble, a user bubble, the thinking block, the shared page, the viewer and the document modal is underlined and passes ≥4.5:1 against its surface
|
||||||
|
- [ ] Tab focus on a link shows the app's 3px `--brand` ring
|
||||||
|
- [ ] a wide image in an answer does not produce a horizontal scrollbar on the chat page at 375px and at 1440px
|
||||||
|
- [ ] `uv run pytest` green (the CSS-pinning unit suites unedited)
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 05 — Unit pins, the dedicated Playwright suite, the gates, the commit
|
||||||
|
|
||||||
|
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3–5` (the acceptance evidence for tasks 01–04)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Lock the three renderer fixes with a source-pin + node-behaviour unit file and the phase's own Playwright E2E suite (AGENTS rules 4/9), run every gate, and land the single atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_markdown_inline_links_images.py` (new — the `tests/unit/test_markdown_blockquotes.py` shape, copy its `HAVE_NODE = shutil.which("node") is not None` skip and its `subprocess` node harness):
|
||||||
|
- **source pins on `frontend/assets/markdown.js`:** the chain order is code → strong → em → linked-image → image → link → `^#### ` → `^#{5,6} ` → `^### ` → `^## ` → `^# ` → lists (assert the index of each rule's literal in the file is strictly increasing); the exact markers `<a class="md-link"`, `<img class="md-img"`, `loading="lazy"`, `<h5>`, `<h6>`; `allowLinkUrl` / `allowImageUrl` named functions exist; the file contains NO `decodeURIComponent` / `unescape` / `innerHTML`; the step-3 guard regex is still exactly `/^<(h\d|ul|ol|pre|li|blockquote)/`.
|
||||||
|
- **CSS pins on `frontend/assets/styles.css`:** `.md-link`, `.md-img`, the `h5` and `h6` rules exist and their colour values reference `var(--…)` only (no hex, no `rgb(`); `.msg.user .bubble .md-link` exists (the inversion).
|
||||||
|
- **node behaviour (skipped without node):** every repro listed in tasks 01–04's Completion Criteria, PLUS the byte-identity contract — a checked-in `NO_INLINE_INPUT` / `NO_INLINE_PRE_PHASE_OUTPUT` fixture pair (a paragraph + a list + a pipe table + a fence + a blockquote, i.e. everything the renderer already supported) asserted byte-equal.
|
||||||
|
2. `tests/fixtures/links_docs/` (new dedicated fixture tree — the phase-138 `quote_docs` precedent: a NEW tree, never the shared 13-doc fixture set, whose import count many suites pin). One document `homelab/inline-markdown.md` whose body carries: a `[absolute link](https://example.com/a?x=1&y=2)`, a `[relative link](/sources)`, a `mailto:` link, a `#anchor`, an inert `[oops](javascript:alert(1))` probe, `` (a tiny inline SVG — CSP `img-src 'self' data:` already allows it, no bytes fixture needed), an inert `)`, a `[](https://example.com)`, and an `#### Deep header` + `##### Deeper` line, plus a pipe table with a `[cell link](https://example.com)` inside a cell.
|
||||||
|
3. `tests/e2e/test_markdown_inline_links_images.py` (new — the phase-138 E2E skeleton verbatim in structure: module-level fixture constants, `_import_fixtures` + `_run_in_thread` + `_reset_db` (truncate `chunks, documents, query_log, steering_notes, kb_overview` then re-import, `summary.added == 1`), `login(page, app_url, next="/")`, the mock LLM trigger answering with the byte-stable inline-markdown answer):
|
||||||
|
- `test_chat_inline_markdown_renders` — the brain bubble contains `a.md-link` elements for the absolute (with `target="_blank"`, `rel` containing `noopener noreferrer`, `href` preserving `&`), relative and `mailto:` links; NO `a.md-link` for the `javascript:` probe (its text stays visible); `img.md-img` for the data-URL image with a non-empty `alt` and `loading="lazy"`; the linked image is an `a.md-link > img.md-img` with no stray bracket in its text; `h5`/`h6` elements exist; `table.md-table a.md-link` exists (cell-level transform); the turn is NOT deflected.
|
||||||
|
- `test_viewer_inline_markdown_renders` — the fixture document opens from the Sources page in the document modal (admin, `openDocumentModal`) and renders the same `a.md-link` / `img.md-img` / `h5` / `h6` shapes (the shared renderer contract across surfaces).
|
||||||
|
- `test_restore_keeps_inline_markdown` — reload the chat page: the localStorage restore path re-runs `renderMarkdown` and the restored brain bubble keeps its links, image and headings.
|
||||||
|
- `test_no_horizontal_overflow_at_mobile` — at `viewport={"width": 375, "height": 720}`, `document.documentElement.scrollWidth <= clientWidth` with the wide image rendered (the `.md-img` `max-width: 100%` contract).
|
||||||
|
4. Run the gates, in order, and fix what fails (do not baseline, do not skip):
|
||||||
|
- `uv run pytest`
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
|
||||||
|
- `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` (in isolation, DB up: `podman compose up -d db`)
|
||||||
|
- `uv run ruff check . && uv run pyright`
|
||||||
|
5. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase's test gate. The unit file must fail if any of tasks 01–03's rules are reordered or removed, and must fail if task 04's CSS introduces a hard-coded colour.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code changes in this phase — the number must not move).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green, including the new unit file (and green with node absent — the skip path)
|
||||||
|
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` green in isolation
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] one atomic commit, `--no-gpg-sign`: `feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer`, containing `frontend/`, `tests/` and `.agents/`
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Phase 181 — SVG documents render as pictures, not as XML
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Need SVG render support on the front end when citing and svg source. Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
|
||||||
|
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
|
||||||
|
**Context:** **The root cause is ingestion, not the viewer.** `app/config.py::image_extensions` (L436) defaults to `png,jpg,jpeg,webp,gif,bmp` and `.env` mirrors it (`BOR_IMAGE_EXTENSIONS`), so an `.svg` can only enter the KB through `BOR_IMPORT_EXTENSIONS` — i.e. as a **text** document whose `content` is the raw XML, which `frontend/assets/document.js::renderDocument` then paints in `<pre class="doc-raw">`. That is exactly the "xml instead of a preview" the owner sees. The rest of the image machinery is already there and needs no change: `app/rag/retriever.py::source_ref_with_image` (L1010-1022) puts `image_url = /api/documents/<id>/image` on a cited image doc, `app/schemas.py::SourceRef.image_url` (L154-181) carries it (omitted when `None`), `app/api/docs.py::get_document_content` (L219-318) returns `is_image` + `image_url` + `format`, `app/api/docs.py::get_document_image` (L319-378) serves the stored bytes with `IMAGE_MIMES.get(suffix, IMAGE_FALLBACK_MIME)`, `frontend/assets/app.js::appendSources` (L2386+) renders a `.source-image` figure per `image_url` ref, `frontend/assets/document.js` renders the `.doc-image` `<img>` block for `doc.is_image` (L453-470, L560+), `frontend/assets/sources.js` (L2328) renders the KB-tree thumbnail. `app/rag/importer.py::_index_file` already prefers the image branch over the text path (L1417-1425) and `_index_image_file` (L3532+) owns bytes-digest → `_markdown_image_context` (phase 125) → `_describe_or_skip` → copy to `image_dir/<doc-id><ext>` → `is_image`/`image_path`/`content`/`summary`. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159, dotted keys) is the one map for both the vision data-URL mime and the route's `Content-Type`; `describe_image` is a CHAT-model vision call (phase 122 LOCKED A3). `app/core/security_headers.py::CSP` (L67-68) + its middleware (which PRESERVES a CSP an inner layer set — L109-110) is the injection boundary. `app/rag/md_images.py::extract_image_refs` (the phase-125 `` scanner) already treats an `.svg` path like any other image. Test patterns: `tests/unit/test_image_documents.py` (toggle / ingest / describe-seam), `tests/integration/test_docs_api.py` (the image route), `tests/e2e/test_image_documents.py` + `tests/e2e/test_answer_images.py` (image E2E), `tests/unit/test_markdown_blockquotes.py` (source-pin + node harness).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An `.svg` in a source tree becomes an **image document**: its own text (`<title>`/`<desc>`/`<text>`) is what gets indexed and retrieved, its bytes are served as `image/svg+xml` under a sandboxed CSP, and every citation surface — the chat figure, the document modal, the `/document.html` viewer, the Sources thumbnail — paints the rendered vector graphic instead of XML. Plus the phase-180 renderer, an SVG referenced as `` inside an answer or a document renders as a real image.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **D1 — SVG is a *vector* image document; the vision call is bypassed (tasks 01–02, locked A1).** OpenAI-compatible vision endpoints do not accept `image/svg+xml`, and adding a rasteriser dependency is out of scope. So an `.svg` takes the phase-122 image branch but its indexed text comes from the file itself: a new pure module `app/rag/svg_text.py` (the `app/rag/md_images.py` precedent — stdlib only, no I/O, fully unit-testable) returns the SVG's own words. `content`/`summary` follow the phase-122 image convention (`summary = content`) and the normal chunk pipeline embeds them; the phase-125 `image_context` (the `` paragraphs around a reference to the file) still prepends, unchanged.
|
||||||
|
- **D2 — the text of a diagram is its `<title>`, `<desc>` and `<text>` (task 01).** Matched by LOCAL name (namespace-agnostic: plain `<title>`, `svg:title`, and namespaced variants all count), in DOCUMENT order, whitespace-normalised, de-duplicated case-insensitively, joined with `\n`. Parsing is `xml.etree.ElementTree` — never a regex over untrusted markup — guarded against the billion-laughs/XE class by (i) refusing any input containing a `<!DOCTYPE` or `<!ENTITY` declaration (→ extraction failure, no parse attempt) and (ii) a `MAX_SVG_BYTES = 2 MiB` input cap (over-cap → extraction failure). XML entity refs inside text (`&`) are decoded by the parser, which is the correct direction (the value is re-escaped on the way out by the renderer).
|
||||||
|
- **D3 — a textless SVG still gets honest, retrievable text (task 01, locked A2).** Many real diagrams carry no words at all (pure `<path>`s). Skipping them (the phase-122 "undescribed image is unsearchable noise" posture) would make them invisible and never citable — the opposite of what the owner asked. So `extract_svg_text` falls back to a structural line built ONLY from facts in the file: `Vector diagram "<file stem>" — <N> shapes (<the three most common element local names with counts>), viewBox <w>×<h>` (each clause omitted when the file does not state it, the stem always present). No invented prose, no model call.
|
||||||
|
- **D4 — ingestion mechanics (task 02).** `image_extensions` default gains `svg` (`.env.example` documents it; a live `.env` keeps its own value — the phase does not edit `.env`). `app/rag/importer.py::_index_image_file` branches on `full_path.suffix.lower() == ".svg"`: vector files take `extract_svg_text` INSTEAD of `_describe_or_skip`; everything else (copy to `image_dir`, digest rules, `is_image`/`image_path`, title = file stem, counters, the `on_chunk` hook) is unchanged and shared. Fail-soft stays LOCKED A3-shaped: extraction returning `None` (over-cap, DOCTYPE-bearing, or unparseable XML) → `summary.images_failed += 1`, a warning, no row, no orphan copy. **One extra guard is required for existing libraries:** `_index_file`'s fast path (L1403, `doc.file_hash == file_digest`) returns BEFORE the image branch, so a row that was indexed as TEXT when `svg` lived only in `BOR_IMPORT_EXTENSIONS` would stay a text doc forever even after the switch. Add a narrow mismatch guard — take the fast path only when the file's kind matches the row's (`settings.images` ∧ the extension is in `image_extension_set` ∧ `doc.is_image is False` ⇒ fall through to the image branch). Everything else keeps the phase-147 fast path byte-for-byte.
|
||||||
|
- **D5 — serving the bytes safely (task 03).** `IMAGE_MIMES[".svg"] = "image/svg+xml"` in `app/rag/summarizer.py` (one map, one truth — the vector branch never reaches `image_data_url`, so the vision path is unaffected). `get_document_image` keeps its public, id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest) and its `Cache-Control: private, max-age=3600`, and adds ONE response header for `image/svg+xml` only: `Content-Security-Policy: default-src 'none'; sandbox` (the `SecurityHeadersMiddleware` preserves an inner CSP, so the route's value wins) — an `<img>` can never execute SVG script, but a user who opens the URL directly must not get a script-capable same-origin document either. `X-Content-Type-Options: nosniff` is already global.
|
||||||
|
- **D6 — the frontend (task 04).** The modal, the viewer, the citation figure and the Sources thumbnail already key off `is_image`/`image_url`, so they get the SVG for free — the task is the sizing + honesty work: `.doc-image-img--vector` (applied in `document.js` where `doc.format === "svg"`) with `width: 100%` so a `viewBox`-only SVG (no intrinsic size, the CSS default replaced-element box is 300×150) fills the `.doc-image` card instead of sitting small in it; the onerror `.doc-image-unavailable` fallback stays as-is; a legacy SVG row still stored as TEXT keeps rendering its XML in `.doc-raw` (honest — there are no bytes behind it) until the next sync converts it.
|
||||||
|
- **D7 — the chat figure is a 96px-capped thumbnail** (`.source-image-img`, `max-height: 96px; max-width: 100%; height: auto`), so a size-less SVG inside it stays inside the cap; `SourceRef` carries no format field and NONE is added (the wire surface stays byte-identical for every text doc — phase-83/122 contract). Clicking the figure/chip opens the modal, which is the full-size preview (`doc.format` known there). Documented in the module comments.
|
||||||
|
- **NOT touched:** the CSP constant in `security_headers.py`, `app/api/chat_images.py::_CHAT_IMAGE_FILENAME_RE` (question attachments are raster-only by design — a vector question image is out of scope), any rasteriser dependency, `describe_image`'s prompt, the phase-124 answer-image path, the `SourceRef` schema, `.env` (user-owned), the PDF embedded-image pass (`pdf.py` — PDFs never embed SVG for this app's extractor).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `180_markdown_inline_links_images` (todo) — its `` transform is what makes `` inside an answer or a document render; this phase supplies the route that answers that URL. No code-level dependency beyond that (the ingestion, the route and the viewer work without it).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_svg_text_extraction.md` — the new pure `app/rag/svg_text.py` module: `<title>`/`<desc>`/`<text>` extraction with the XE/size guards and the structural fallback.
|
||||||
|
2. `02_svg_index_as_image.md` — `svg` in `image_extensions`, the vector branch in `_index_image_file`, and the fast-path kind-mismatch guard that converts existing text-SVG rows.
|
||||||
|
3. `03_svg_bytes_route_csp.md` — `IMAGE_MIMES[".svg"]` and the sandboxed `image/svg+xml` response from `GET /api/documents/{id}/image`.
|
||||||
|
4. `04_svg_preview_render.md` — the vector sizing class + CSS in the viewer, and the figure/modal/thumbnail verification (the "preview instead of XML" contract).
|
||||||
|
5. `05_unit_integration_e2e.md` — `tests/unit/test_svg_text_extraction.py`, `tests/integration/test_svg_document_image.py`, the phase's dedicated Playwright suite `tests/e2e/test_svg_document_preview.py`, the gates, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- **Unit:** `tests/unit/test_svg_text_extraction.py` (new, pure-function, no DB): title+desc+text in document order, namespace-prefixed variants, whitespace normalisation, de-duplication, `&` decoded, the `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback for a textless SVG (stem + shape counts + viewBox, and each clause absent when unknown). Importer-level: the vector branch indexes an SVG with NO vision call (the `_describe_or_skip` seam patched to FAIL the test if called), `content == extracted text`, `summary == content`, the copy lands at `image_dir/<id>.svg`, `is_image=True`, the `images_failed` fail-soft path, and the fast-path conversion of a pre-existing text row (same bytes, `file_hash` unchanged).
|
||||||
|
- **Integration:** `tests/integration/test_svg_document_image.py` (new): `GET /api/documents/{id}/image` for an SVG row → `200`, `Content-Type: image/svg+xml`, the sandbox CSP header present, `Cache-Control: private, max-age=3600`; a raster row's response headers UNCHANGED (no CSP key added); the four 404 shapes (unknown id, unparseable id, text doc, missing file on disk) unchanged.
|
||||||
|
- **E2E (AGENTS rule 4):** `tests/e2e/test_svg_document_preview.py`, run **in isolation** (`uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov`, DB up).
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — `app/rag/svg_text.py` is new pure code and must be covered near-completely; the importer/route deltas are covered by the tests above.
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] an `.svg` in a source tree imports as `is_image` with `image_path` set, no vision call made, and its indexed text drawn from its own `<title>`/`<desc>`/`<text>`
|
||||||
|
- [ ] `GET /api/documents/<id>/image` serves `image/svg+xml` with `Content-Security-Policy: default-src 'none'; sandbox`
|
||||||
|
- [ ] clicking an SVG citation chip/figure in the chat opens the modal showing a rendered `<img>` — no XML text anywhere in the modal body
|
||||||
|
- [ ] the `/document.html` viewer renders the SVG full-width in its `.doc-image` card (not the 300×150 default box) with the extracted text below it
|
||||||
|
- [ ] a `` reference inside a markdown document paints the graphic (the phase-180 renderer + this route — the E2E's two-stage seeding, `test_markdown_image_ref_renders`)
|
||||||
|
- [ ] every raster/PDF image behaviour is unchanged (the phase-122/123/124/133/140 suites pass unedited)
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — an SVG's indexed text is extracted from the file itself (`<title>`/`<desc>`/`<text>`), the vision describe call is bypassed for `.svg`; no rasteriser dependency is added (owner-confirmed 2026-10-14: "SVG as image document").**
|
||||||
|
- **A2 — a textless SVG is indexed with a structural metadata line built only from facts in the file (stem, shape counts, viewBox) rather than skipped, so it stays retrievable and previewable (assistant decision made explicit here; it relaxes the phase-122 fail-soft skip for the vector case only — a genuinely unparseable or over-cap file still skips).**
|
||||||
|
- **A3 — SVG bytes render ONLY through `<img>`, never inlined into the DOM, and the bytes route carries `default-src 'none'; sandbox` so a direct navigation cannot execute script (owner-confirmed 2026-10-14, roadmap confirmation).**
|
||||||
|
- **A7 — `svg` joins the `BOR_IMAGE_EXTENSIONS` DEFAULT; the `BOR_IMAGES` toggle remains the single knob (phase 122 LOCKED A3). With `BOR_IMAGES=0` an `.svg` stays a text document (raw XML in the viewer) and the walk never admits it as an image — no special-casing of the off state (owner-confirmed 2026-10-14, roadmap confirmation).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(kb): index SVGs as image documents and preview them instead of raw XML"
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 01 — `app/rag/svg_text.py`: the words inside a vector drawing
|
||||||
|
|
||||||
|
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source" (the indexed-text half: an SVG document must be retrievable without a vision call)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A new pure module turns SVG bytes into the text that makes the document searchable — its own `<title>`/`<desc>`/`<text>` words, or an honest structural line when it has none — with no third-party dependency, no I/O, and no network.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/svg_text.py` (new — the `app/rag/md_images.py` module shape: module docstring citing this phase + the LOCKED decisions, `from __future__ import annotations`, stdlib only, full type hints, pure functions):
|
||||||
|
- `MAX_SVG_BYTES: int = 2 * 1024 * 1024` and `MAX_SVG_TEXT_CHARS: int = 4000` (module constants, documented — the cap binds on the EXTRACTED text, never silently mid-word: truncate at the last newline/period before the cap).
|
||||||
|
- `@dataclass(frozen=True) class SvgText:` — `text: str` (the searchable text) and `structural: bool` (`True` when the fallback line was used, so the importer can log/count it).
|
||||||
|
- `def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None`.
|
||||||
|
2. `app/rag/svg_text.py` — the guards, BEFORE any parse (LOCKED A3's sibling: untrusted XML):
|
||||||
|
- empty/whitespace-only input → `None`;
|
||||||
|
- `len(data) > MAX_SVG_BYTES` → `None` (the billion-laughs/memory guard, alongside the next one);
|
||||||
|
- the lower-cased bytes containing `<!doctype` or `<!entity` → `None` (a document type declaration is refused outright, which is what makes stdlib `ElementTree` safe here — no external entities, no entity expansion);
|
||||||
|
- `ElementTree.fromstring` raising `ParseError` → `None` (never re-raise: the caller's fail-soft is a skip, not a crash).
|
||||||
|
3. `app/rag/svg_text.py` — extraction: a `local_name(tag)` helper that strips the `{namespace}` prefix, then walk `root.iter()` (document order) and collect, for local name `title` / `desc` → the element's `.text` block (`"".join(el.itertext())` so nested markup is carried), and for `text` → `" ".join(el.itertext())` (this is what covers `<tspan>` children). Normalise each item with `" ".join(item.split())`, drop empties, de-duplicate case-insensitively keeping the FIRST occurrence, cap at `MAX_SVG_TEXT_CHARS`, and return `SvgText("\n".join(items), structural=False)` when at least one item survived (parser-decoded entities like `&` come out decoded — the renderer re-escapes on display).
|
||||||
|
4. `app/rag/svg_text.py` — the structural fallback (LOCKED A2), used only when the walk produced nothing: `Vector diagram "<stem>" — <N> shapes (<top-3 local names by count>), viewBox <w>×<h>` built ONLY from the file — `N` counts descendant elements excluding `svg`/`title`/`desc`; the `shapes` clause is omitted when `N == 0` and the noun pluralises normally (`1 shape`, `2 shapes`); the `viewBox` clause is omitted when the root has no parseable `viewBox` (four whitespace/comma-separated numbers; the last two are `w`/`h`) and when present reports `w×h` with integral values formatted without a trailing `.0`; the stem is `""` → the clause `Vector diagram` alone (no empty quotes). The em-dash form matches the app's own doc-comment typography and is the string the tests pin.
|
||||||
|
5. `app/rag/svg_text.py` — module docstring: why the vision call is bypassed (LOCKED A1: vision endpoints reject `image/svg+xml`; no rasteriser dependency), the guard list, the document-order/dedupe rules, and the fallback contract (LOCKED A2: facts from the file, never invented prose).
|
||||||
|
- ASSUMPTION: `<image>` elements embedding a raster inside the SVG are NOT followed (their `href` is not text; the byte-embedded raster belongs to the phase-151 PDF path, not here).
|
||||||
|
- ASSUMPTION: `aria-label`/`role` attributes are not extracted — `<title>`/`<desc>` are the SVG-spec carriers and the fallback covers the rest.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_svg_text_extraction.py` starts here (completed in task 05) — every branch above is a pure-function test with a literal SVG string, no DB, no LLM, no filesystem: document order across `title`/`desc`/`text`, `tspan` joining, namespace-prefixed tags (`<svg:title>`, `xlink:` attributes ignored), whitespace normalisation, case-insensitive dedupe, `&` decoded, the `<!DOCTYPE`/`<!ENTITY` refusals (both a `<!DOCTYPE svg [<!ENTITY x "boom">]>` wrapper and a bare `<!ENTITY`), the over-cap refusal, malformed XML → `None`, and the fallback (shape counts + viewBox present/absent/malformed, empty stem).
|
||||||
|
- Coverage: **>90%** on this module (target: near-100% — it is pure).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run python -c "from app.rag.svg_text import extract_svg_text; print(extract_svg_text(b'<svg viewBox=\"0 0 8 6\"><title>Top</title><text>a</text></svg>', stem=\"topo\"))"` reports the two extracted items in order with `structural=False`
|
||||||
|
- [ ] a `<svg>` with only `<path>`s yields `Vector diagram "topo" — 1 shape (path)`-shaped text with `structural=True`
|
||||||
|
- [ ] `extract_svg_text(b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>')` returns `None`
|
||||||
|
- [ ] `uv run pytest tests/unit/test_svg_text_extraction.py -v` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 02 — SVG joins the image set and indexes as a vector document
|
||||||
|
|
||||||
|
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source. Also would be nice to click an svg and see a preview … rather than the xml"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`svg` is an image extension, and `_index_image_file` indexes an `.svg` from its bytes + extracted text instead of a vision call — so the row arrives with `is_image=True`, `image_path` set and searchable content. A pre-existing text-indexed SVG row is converted on the next sync.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` (L436) — `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` → `"png,jpg,jpeg,webp,gif,bmp,svg"`; extend the field's doc-comment: SVG is the one VECTOR member (LOCKED A1 — no vision call, `app.rag.svg_text` supplies its text) and `BOR_IMAGES` remains the single knob (LOCKED A7). The existing `_image_extensions_known` validator (L659-673) needs no change (`svg` is a well-formed token); `image_extension_set` (L899-909) yields `.svg`.
|
||||||
|
2. `.env.example` (the `# BOR_IMAGE_EXTENSIONS=…` line ~L167) and `README.md` (L794 prose, the L1239 settings table row) — add `svg` to the documented default, one sentence on the vector behaviour (indexed from its own `<title>`/`<desc>`/`<text>`; no vision call) and one on the `BOR_IMAGES=0` consequence (LOCKED A7: an `.svg` stays a text doc — raw XML in the viewer). Never touch the live `.env` (user-owned).
|
||||||
|
3. `app/rag/importer.py::_index_image_file` (L3532+) — the vector branch, at the `_describe_or_skip` call site (L3632) ONLY:
|
||||||
|
```
|
||||||
|
if full_path.suffix.lower() == ".svg":
|
||||||
|
extracted = extract_svg_text(full_path.read_bytes(), stem=full_path.stem)
|
||||||
|
content = None if extracted is None else extracted.text
|
||||||
|
else:
|
||||||
|
content = await _describe_or_skip(...) # unchanged, byte-identical
|
||||||
|
```
|
||||||
|
Keep everything after it exactly as it is — the `images_failed += 1` + warning + `return` fail-soft when `content is None` (LOCKED A2: only a genuinely unusable file skips; the structural fallback means a textless SVG does NOT), the `image_context` prepend (`f"{image_context}\n\n{content}"`, phase 125 — unchanged), the digest over the raw bytes, the `image_dir/<doc-id>.svg` copy written only after success, `title = full_path.stem`, `is_image=True`, `image_path`, `summary = content`, the chunk pipeline and the `on_chunk` hook.
|
||||||
|
4. `app/rag/importer.py` — import `extract_svg_text` next to the existing `app.rag.summarizer` imports (L356-359) and update the module docstring's image paragraph: the image branch is raster (vision-described) OR vector (`svg`, text-extracted, no model call).
|
||||||
|
5. `app/rag/importer.py` — a log-only `svg_indexed: int = 0` counter on `ImportSummary` (docstring: log-only, the phase-133 stats convention — the sync `detail` API shape is untouched) incremented on a successful vector index, plus one `logger.info("import: svg indexed source=%s path=%s structural=%s chars=%d", …)` line so a run's SVG handling is auditable (AGENTS rule 10).
|
||||||
|
6. `app/rag/importer.py::_index_file` — the fast-path kind guard (design D4). At the `if doc is not None and doc.file_hash == file_digest:` fast path (L1403), take it only when the row's KIND already matches how the file would be indexed now: compute the would-be image membership once (`settings.images` ∧ `match_extension(full_path, settings.image_extension_set) is not None`) and require `doc.is_image` to agree. That converts a row imported as TEXT (before `svg` joined the image set) — same bytes, same `file_hash` — into the image branch instead of returning early, while every correctly-classified file keeps the phase-147 fast path byte-for-byte. Add the explanatory comment (this is the migration; no alembic migration is needed — `is_image` flips through the normal index write).
|
||||||
|
- The reverse direction (a raster row when `BOR_IMAGES` is off) must NOT be forced through the image path (the walk never admits it and the phase-122 prune guard protects it) — the guard only applies when `settings.images` is on.
|
||||||
|
- ASSUMPTION: no dedicated backfill script (the phase-118/138 backfill pattern) — the next scheduled sync converts every SVG through the guard in step 6, which is idempotent and needs no new entrypoint.
|
||||||
|
- ASSUMPTION: an SVG in BOTH `BOR_IMPORT_EXTENSIONS` and `BOR_IMAGE_EXTENSIONS` becomes an image document — `_index_file` already prefers the image branch (L1417-1425) over the text path, so no walk-precedence change is made.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_image_documents.py` extended, or a new `tests/unit/test_svg_image_ingest.py` if the file's scope reads better — the executor's call, stated in the commit body): the vector branch indexes an SVG while the `_describe_or_skip` seam is patched to raise (proving NO vision call), `content` equals the extracted text, `summary == content`, one `is_summary` chunk, `is_image=True`, `image_path` ends with `.svg` inside `image_dir`, `svg_indexed == 1`; extraction failure (over-cap bytes) → `images_failed == 1`, no row, no copy; the `image_context` enrichment prepends for an SVG exactly as for a raster; the fast-path guard — a row with `is_image=False` and a matching `file_hash` is RE-indexed as an image on the next run, while an already-correct row (raster `is_image=True`, same hash) still takes the fast path (assert `_describe_or_skip` is NOT called).
|
||||||
|
- Regression: `tests/unit/test_image_documents.py`, `tests/integration/test_importer.py`, `test_import_docs_git.py`, `test_importer_dates.py` stay green — no raster/PDF behaviour changes.
|
||||||
|
- Coverage: **>90%** on the changed lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `Settings().image_extension_set` contains `.svg` with no env override
|
||||||
|
- [ ] importing a tree containing one `.svg` creates a row with `is_image=True`, an `image_path` under `image_dir`, and content from the SVG's own text — with the vision seam never invoked
|
||||||
|
- [ ] a second sync over the unchanged file reports it `unchanged` (the guard fires once, the fast path resumes)
|
||||||
|
- [ ] a pre-existing text-SVG row flips to `is_image=True` on the next sync (fixture test)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 03 — Serve the bytes as sandboxed `image/svg+xml`
|
||||||
|
|
||||||
|
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "…see a preview … rather than the xml" (the transport half: something must answer `/api/documents/<id>/image` with SVG the browser will paint, safely)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`GET /api/documents/{doc_id}/image` serves an SVG row as `image/svg+xml` behind a `default-src 'none'; sandbox` CSP, while every raster response keeps its current bytes, headers and 404 semantics unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159) — add `".svg": "image/svg+xml"`. Extend the map's doc-comment: SVG is the one vector entry, and the vector import branch (task 02) never routes through `image_data_url`, so this entry exists for the BYTES route only (`app/api/docs.py::get_document_image` reads the same map — one map, one truth).
|
||||||
|
2. `app/api/docs.py::get_document_image` (L319-378) — keep the lookup, the public id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest), the four 404 shapes (unknown id, unparseable id, text doc, missing file) and the `Cache-Control: private, max-age=3600`. Add ONE conditional header next to `media_type`:
|
||||||
|
```
|
||||||
|
headers = {"Cache-Control": "private, max-age=3600"}
|
||||||
|
if media_type == "image/svg+xml":
|
||||||
|
headers["Content-Security-Policy"] = "default-src 'none'; sandbox"
|
||||||
|
```
|
||||||
|
(LOCKED A3.) Raster responses gain NO new header — their header set is byte-identical.
|
||||||
|
3. `app/api/docs.py` — docstring on the route: why SVG needs the extra header — an `<img>` can never execute SVG script, but the URL is directly navigable, and without the policy a stored SVG would be a script-capable same-origin document; `sandbox` blocks scripting AND makes the response an opaque origin (no same-origin fetch/form capability), `default-src 'none'` blocks every subresource. Note that `app/core/security_headers.py::SecurityHeadersMiddleware` PRESERVES an inner CSP (L109-110), so this route's value is the one the browser sees, and that the middleware's global `X-Content-Type-Options: nosniff` still applies.
|
||||||
|
4. `README.md` — the security-headers section (where phase 82/123's CSP is described) gains one line: SVG document bytes are served sandboxed, with the one-sentence rationale.
|
||||||
|
- ASSUMPTION: no `Content-Disposition: attachment` — the graphic must render inline; `sandbox` is the stronger and less intrusive control.
|
||||||
|
- ASSUMPTION: the CSP is scoped to `image/svg+xml` (derived from the extension map, never from a client value) rather than applied to all image types — a raster needs no policy change, and adding one would risk the phase-123/124 image surfaces.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `tests/integration/test_svg_document_image.py` (new, task 05 completes it): an SVG row → `200`, `content-type == "image/svg+xml"`, `content-security-policy == "default-src 'none'; sandbox"`, `x-content-type-options == nosniff`, `cache-control == "private, max-age=3600"`, body bytes byte-equal to the stored copy. A raster (`.png`) row → NO `content-security-policy` key in the response headers and the existing headers unchanged. The 404 shapes: unknown uuid, a non-uuid string (`404`, not 422), a text doc, an image row whose copy was deleted from disk.
|
||||||
|
- Unit: `IMAGE_MIMES` contains `.svg` and the existing entries are untouched (the map is asserted by identity, not membership, so a silent removal fails).
|
||||||
|
- Coverage: **>90%** on the changed lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `curl -sD- -o/dev/null http://127.0.0.1:8000/api/documents/<svg-doc-id>/image` shows `content-type: image/svg+xml` and `content-security-policy: default-src 'none'; sandbox`
|
||||||
|
- [ ] the same request for a PNG row shows no `content-security-policy` header (and `content-type: image/png`)
|
||||||
|
- [ ] `uv run pytest tests/integration/test_svg_document_image.py tests/integration/test_docs_api.py -v` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 04 — The preview instead of the XML (viewer, figure, modal, thumbnail)
|
||||||
|
|
||||||
|
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An SVG document paints as a graphic at readable size in the document modal and the `/document.html` viewer, keeps its compact figure in the chat citation row, and keeps its Sources thumbnail — no XML text anywhere on those surfaces.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document.js::docImageBlock` (L571-591) — one addition: when `doc.format === "svg"`, set the img's class to `doc-image-img doc-image-img--vector` (keep the existing base class so every current rule still applies). Rationale to put in the comment: an SVG carrying only a `viewBox` has NO intrinsic size, so the replaced-element default box (300×150) would sit small inside the full-width `.doc-image` card; `doc.format` is the only place in the app that knows the type (the bytes URL is extension-less and `SourceRef` carries no format field by design — design D7). Do NOT inline the SVG into the DOM (LOCKED A3): `img.src = doc.image_url` stays `createElement` + property assignment, which is already the file's XSS contract.
|
||||||
|
2. `frontend/assets/styles.css` (next to `.doc-image-img`, L5623-5630) — `.doc-image-img--vector { width: 100%; height: auto; object-fit: contain; }` with the house comment (the no-intrinsic-size explanation, why `object-fit: contain` protects `max-height: 70vh` from squashing the drawing, static content/no animation). Palette-neutral: no new colour.
|
||||||
|
3. `frontend/assets/app.js::appendSourceImageFigure` (~L2485+) — NO structural change: the figure already keys off `image_url`, already carries the summary caption and the chip's navigation (the click → `openDocumentModal` path is the "preview in the RAG tab" the owner asked for). Add the comment recording the deliberate non-change: a size-less SVG inside the 96px-capped `.source-image-img` scales inside the cap, and the full-size preview is the modal (no `format` field is added to `SourceRef` — the wire contract stays byte-identical for text docs).
|
||||||
|
4. `frontend/assets/sources.js` (the `d.is_image` thumbnail branch, ~L2328) — verify the KB-tree thumbnail renders for an SVG row through the same `image_url` and add the same one-line comment; only touch the code if the E2E proves a real gap (report the finding in the commit body either way).
|
||||||
|
5. Legacy honesty — do NOT special-case it: a row still indexed as TEXT (an `.svg` under `BOR_IMAGES=0`, or not yet re-synced) keeps rendering its XML in `.doc-raw`, which is the truth about that row (task 02's fast-path guard converts it on the next sync). Comment this in `docImageBlock` so nobody "fixes" it by sniffing the content string.
|
||||||
|
- ASSUMPTION: no lightbox/zoom affordance is added — the modal at full width IS the preview the owner asked for; zoom is a separate feature request.
|
||||||
|
- ASSUMPTION: no "download the SVG" control — the bytes URL is directly openable and the existing "Copy path" control already covers provenance.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_document_viewer.py` extended — `docImageBlock` for `format === "svg"` yields `class="doc-image-img doc-image-img--vector"` with `src` = `image_url` and `alt` = summary (title fallback); for `format === "png"` the class string is unchanged (the base class only); the CSS pins `.doc-image-img--vector` exists, references no colour literal, and keeps `object-fit: contain`.
|
||||||
|
- E2E (task 05): the modal + viewer assertion that an SVG row renders an `<img>` with `naturalWidth > 0` and that the modal body contains no `<` XML text and no `<pre class="doc-raw">`.
|
||||||
|
- Coverage: n/a (frontend) — the unit pins + the E2E are the gate.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] clicking an SVG source chip in a chat answer opens the modal showing the rendered graphic (no XML, no `.doc-raw` in the modal body)
|
||||||
|
- [ ] `/document.html?source=…&path=…svg` shows the graphic spanning the `.doc-image` card (width ≥ 600px at a 1280px viewport for a `viewBox`-only SVG) with the extracted text below it
|
||||||
|
- [ ] the chat citation figure and the Sources thumbnail show the graphic; the figure's caption is the extracted text
|
||||||
|
- [ ] a raster image document's viewer DOM is unchanged (the phase-122/133 suites pass unedited)
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check .` clean
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 05 — Unit + integration + the dedicated Playwright suite, the gates, the commit
|
||||||
|
|
||||||
|
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` (the acceptance evidence for tasks 01–04)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the whole SVG path — extraction, image-document ingestion, the sandboxed bytes route, and the preview at every surface — with the phase's own test files, run the AGENTS gates, and land the single atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_svg_text_extraction.py` — completes task 01's pure-function matrix (document order, `tspan` joining, namespaced tags, whitespace/dedupe rules, entity decoding, `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback with its plural + clause-omission rules, `MAX_*` constants pinned).
|
||||||
|
2. `tests/unit/test_svg_image_ingest.py` (or the extension of `tests/unit/test_image_documents.py` chosen in task 02) — the vector branch with the `_describe_or_skip` seam patched to RAISE (the "no vision call" proof), `content`/`summary`/chunk shape, `is_image`/`image_path` under `image_dir`, `svg_indexed`, the extraction-failure skip (`images_failed`, no row, no copy), the phase-125 `image_context` prepend for an SVG, and the `_index_file` fast-path kind guard in BOTH directions (text row converts; correctly-classified row stays `unchanged`).
|
||||||
|
3. `tests/integration/test_svg_document_image.py` — task 03's contract (SVG → `image/svg+xml` + the sandbox CSP + `nosniff` + `Cache-Control` + exact bytes; PNG → NO CSP key; the four 404 shapes) plus `GET /api/documents/content` for an SVG row: `is_image: true`, `image_url` present and of the form `/api/documents/<uuid>/image`, `format: "svg"`, `content` = the extracted text. Add the wire-contract pin that ties the phases together: that `image_url` shape is accepted by phase 180's `allowImageUrl` (assert the literal path passes the renderer's allowlist regex — the `` synergy).
|
||||||
|
4. `tests/fixtures/svg_docs/` (new dedicated tree — never the shared fixture set) — `diagrams/topology.svg` (a hand-written multi-kilobyte diagram: a `viewBox` and NO `width`/`height`, a `<title>Homelab network topology</title>`, a `<desc>` sentence, two `<text>` labels, and a dozen `<path>`/`<rect>` shapes so retrieval has real words) + `notes/infra.md` (prose that names the diagram so the grounded question has a text co-citation) + `diagrams/blank.svg` (pure paths — the structural-fallback subject).
|
||||||
|
5. `tests/e2e/test_svg_document_preview.py` (new; the `tests/e2e/test_image_documents.py` module-app pattern — its own app instance with `BOR_IMAGES=true` and `BOR_IMAGE_DIR` pinned to a suite tmp dir so a local `.env` cannot leak either way, the phase-138 `_reset_db` truncate-then-`import_sources` seeding, `login(...)`, mock LLM, `E2E_REAL_LLM` skipped):
|
||||||
|
- `test_viewer_renders_svg_not_xml` — open the Sources row for `diagrams/topology.svg` (admin): the modal shows `img.doc-image-img--vector` with `naturalWidth > 0` and a non-empty `alt`; the modal body contains no `<pre class="doc-raw">` and no `<svg` XML text; the extracted `<title>` text appears as the description below the graphic; the "Full page" link's `/document.html` view shows the same graphic at ≥600px width at a 1280px viewport (the vector sizing contract).
|
||||||
|
- `test_chat_figure_and_modal_preview` — the grounded question whose answer cites the SVG doc shows `.source-image` whose `<img>` loads (`naturalWidth > 0`) with the caption from the extracted text; clicking the figure opens the modal with the full graphic (the "preview in the RAG tab" contract) and Escape closes it.
|
||||||
|
- `test_blank_svg_is_indexed_and_searchable` — `diagrams/blank.svg` exists as a document with the structural `Vector diagram "blank" — …` content (the LOCKED A2 fallback), listed on the Sources page with its thumbnail.
|
||||||
|
- `test_markdown_image_ref_renders` (two-stage seeding — copy the fixture tree to the suite's tmp sources dir, import it, read the SVG doc's id from the DB, then write `notes/inline.md` containing `` and re-import): the viewer for that document shows `img.md-img` pointing at the bytes route and loading (`naturalWidth > 0`) — the phase-180 renderer + this route end to end.
|
||||||
|
- `test_raster_and_pdf_surfaces_unchanged` — the smoke guard: a PNG doc in the same tree still renders through `.doc-image-img` (no `--vector` class) and the CSP header is absent on its bytes route.
|
||||||
|
6. Run the gates in order, fixing what fails (no baselining, no skipping):
|
||||||
|
- `uv run pytest`
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
|
||||||
|
- `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` (in isolation, DB up)
|
||||||
|
- `uv run ruff check . && uv run pyright`
|
||||||
|
- `uv run python -m scripts.api_payload_audit` — the phase-167 byte-budget line for the new SVG route is INFO-sized (an SVG fixture stays well under 256 KB).
|
||||||
|
7. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase's gate. If a test cannot be made deterministic, fix the product, not the assertion (the AGENTS rule 9 posture) — and record anything genuinely untestable in the commit body.
|
||||||
|
- Coverage: **>90%** on `app/`; `app/rag/svg_text.py` should sit at ≥95% (pure functions, no excuses).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green (new unit + integration files included)
|
||||||
|
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` green in isolation
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] `uv run python -m scripts.api_payload_audit` clean for the SVG route
|
||||||
|
- [ ] one atomic commit, `--no-gpg-sign`: `feat(kb): index SVGs as image documents and preview them instead of raw XML`, containing `app/`, `frontend/`, `tests/`, `.env.example` and `.agents/`
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
# Phase 182 — Verbatim file display: a `show_text` tool + `bor` locator fences the server resolves
|
||||||
|
|
||||||
|
**Story:** n/a (owner chat 2026-10-14 — "show me my pi.dev config" / "show me the quadlet for gufo" answers are re-typed from context and silently add/remove characters, so the answer text cannot be copy-pasted; the owner has to open the source bubble and ignore the written code block)
|
||||||
|
**Context:** `app/rag/agent.py` — the harness-aligned tool surface `AGENT_TOOLS` (L332: `ls` / `read` / `grep`) plus the 4th, DISPLAY tool `SHOW_IMAGE_TOOL` (L501) composed by `agent_tools(settings)` (L538), the shared identity resolver `_resolve_path` (L1357), the line-range validator `_read_range_args` (L1474) and the `OFFSET_PAST_END` (L671) / `BAD_RANGE_ARGS` (L628) teachings, the refusal block L571–L690 (`MISSING_SHOW_ARGS` L571, `NOT_AN_IMAGE_DOC` L581, `ALREADY_SHOWING` L585), `AgentHolder` (L1600–L1680: `shown_images`, `read_coverage`, `read_truncations`), `_execute_tool` (L1681, the `show_image` branch L2023–L2068 is the shape this phase mirrors); `app/rag/prompts.py` — `TOOLS_SECTION` (L293) and the appended-only-when-offered `SHOW_IMAGE_CLAUSE` (L379) + `build_high_prompt(show_image=…)` (L759); `app/api/chat.py` — the prompt/tool call site `show_image=settings.answer_images` (L529), the per-turn visible-content counter `content_chars` (L934), the `done`-frame builder (L1320–L1365, `answer_ref`/`answer_image` L1343–L1355), the pinned per-turn log line (L1272–L1291); `app/schemas.py` — `SourceRef` (L169), `ChatToolResultEvent` (L279), `ChatDoneEvent.answer_image` + its omission serializer (L305–L380), `ChatMessageRecord.answer_image` + `_drop_image_when_absent` (L1463, L1478); `app/config.py` — `read_max_chars` (L296) + its validator (L686), `answer_images: bool = False` (L539), `log_large_response_bytes` (L76); `app/core/settings_catalog.py` — `read_max_chars` (L267) / `answer_images` (L296) rows (the completeness pin in `tests/unit/test_settings_catalog.py` makes a new `Settings` field a build failure until it has a row); `frontend/assets/markdown.js` — `renderMarkdown(md)` (L43), fence-protection pass (L44–L48: fences → tables → escape+inline → blockquotes → paragraphs → restore, escape-first, no CDN); `frontend/assets/app.js` — the per-delta whole-bubble re-render `renderMarkdown(acc)` (L4322), the `done` handler (L4325–L4460), `appendAnswerImage` (L2598), the restore renderer (L2890–L2960), `copyShareLinkWithFallback` (L3250, the http-origin clipboard degradation), `rememberBrainTurn` save point (L4390–L4420); `frontend/assets/shared.js` — `addAnswerImage` (L573) + the record renderer (L619–L660); `frontend/index.html` L1696–L1701 and `frontend/shared.html` L197–L199 (classic `brand.js` + `markdown.js`, then the module script); `frontend/assets/styles.css` — `.answer-image` (L1028–L1059), the bounded-scroll-panel and `.cell-clip` conventions (PLAN §7.1, phases 156/167); `tests/e2e/mock_llm.py` — the scripted-trigger registry (L1162–L1520, `SHOW_IMAGE_TRIGGER` L2606, `_SHOW_CONFIRM_MARKER`, the "verified: no existing E2E question or fixture file contains the phrase" discipline); `tests/e2e/test_answer_images.py` (the two-app ON/OFF pattern), `tests/unit/test_frontend_*.py` (the JS/CSS text-pin convention).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
When the owner asks *"show me the quadlet for gufo"*, the exact bytes of the indexed file reach the chat answer — **typed by the server, never by the model** — in a panel with a path + line-range caption, a `VERBATIM` badge, line numbers, a working Copy button and a link to the source. The model only *names* the location (`show_text(path, offset?, limit?)`, or an inline ` ```bor <path> <range> ` locator fence where it wants the block to sit); the server resolves the slice from Postgres and the client places it. This kills the class of bug where the answer's code block silently differs from the file, and it removes the user-facing line-number problem entirely: the numbers shown are the server's resolved extent, not the model's claim.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **D1 — the two mechanisms, one authority (A1).** (a) a 5th agent tool `show_text(path, offset?, limit?)`, the display sibling of `show_image`; (b) an **inline locator fence** the model may write in its answer — ` ```bor ` + the combined `source/path` + an optional range — so a resolved block can sit anywhere in the prose. **Both are resolution requests, never content requests**: the only party that supplies bytes is the server. `show_text` is the authority for teaching (a refusal teaches the model to self-correct before the user sees anything); the fence is placement sugar and ALSO resolves standalone (a fence with no prior `show_text` call still renders, because resolution is a plain `(source, path)` lookup, not a replay of a tool call).
|
||||||
|
- **D2 — server resolves, client places (A4/A7).** The client cannot resolve (no new route, A4), so: `app/api/chat.py` accumulates the turn's **clean visible answer text** (next to the existing `content_chars` counter, L934 — the post-scaffold-filter text only, never `thinking`), and at `done`-build time passes it through `app.rag.verbatim.resolve_answer_blocks(db, answer, holder.shown_texts, settings)`; the result rides the `done` frame as the ADDITIVE `answer_text` array. `markdown.js` turns a `bor` fence into `<div class="verbatim" data-bor-locator="<canonical>">` and **drops the model's fence body**; `app.js`/`shared.js` hydrate each placeholder by matching `data-bor-locator` against the record's `answer_text[].locator` (canonical-form match → unique path-only match → unresolved). A block the tool registered that the answer never locates renders APPENDED, at the `answer_image` position (below the answer, above the source clusters).
|
||||||
|
- **D3 — the model's bytes are NEVER painted (A7, the phase's core contract).** Inside a `bor` fence the model's body is discarded at render time — while streaming the placeholder reads `Retrieving the exact text…` (`role="status"`, polite by definition), and a fence the model has not finished typing shows that SAME pending state rather than raw backticks, at `done` it becomes the resolved panel or, when it cannot be resolved, the honest state `I could not verify this against the knowledge base — open <source>` (persona: radically honest, never a silent substitution and never the model's possibly-corrupt text). The E2E proves it with a mock that deliberately types a CORRUPTED body and asserts the panel carries the DB bytes. `renderMarkdown` gains a SECOND, OPTIONAL argument (`{ verbatim: true }`); default OFF keeps every existing call site — the document viewer (`document.js` L494), the thinking blocks, the user bubbles — byte-identical, and a document that legitimately contains a ` ```bor ` fence still renders its own text.
|
||||||
|
- **D4 — line ranges are the server's (A2).** `offset` (1-based) / `limit` are OPTIONAL and validated by the existing `_read_range_args` (L1474): omitted → the WHOLE document (the taught default for "show me X"); `offset` alone → to the end; `limit` overshoot → CLAMPED to the last line; `offset` past the last line → the `OFFSET_PAST_END`-style refusal naming the document's true line count (counts in nothing, consumes one round, invisible to the user). Slicing uses `content.splitlines()` with the SAME 1-based indexing as `read` (L1836) and `grep`, so every line number in the app agrees. The char cap (`settings.show_text_max_chars`) cuts the SLICE at a line boundary, appends the shared `TRUNCATION_MARKER` (`app/rag/retriever.py`) and shrinks `end_line` to the last fully-shown line — the caption never lies.
|
||||||
|
- **D5 — what the model gets back (A3).** The tool result is a confirmation + the truth + an instruction, then the exact slice:
|
||||||
|
`Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.` / `Document {source}/{path}:` / `date: …` / `{origin lines}` / `lines: {first}-{end} of {total_lines}` / the slice. The `Document …` + `date:` + `lines:` header contract is the one `read` established (the E2E mock's `_READ_RESULT_PREFIX` pin keeps parsing it). Returning the text is deliberate (A21: LLM calls are free) — the model's commentary can then never disagree with the panel.
|
||||||
|
- **D6 — new module, no import cycle (D2).** `app/rag/verbatim.py` owns `VerbatimBlock` (frozen dataclass: `source, path, title, start_line, end_line, total_lines, text, truncated, locator`), the pure half (`canonical_locator`, `slice_lines`, `fences_in`, `cap_slice`) and one DB function (`document_by_identity(db, combined)`, a bare `select(Document).where(source==, path==)` — the same `(source, path)` identity contract as `agent._resolve_path`, WITHOUT its did-you-mean suggestions). `agent.py` imports the pure half, so `verbatim.py` must never import `agent.py` (documented at the module top — protocol D2). Path traversal is a non-issue by construction: a traversal string is simply a non-existent row (404-shaped refusal, no filesystem access anywhere).
|
||||||
|
- **D7 — canonical locator (the matching key).** `source/path` for a whole document, `source/path#12-40` for a range. The tolerant input grammar (both the fence parser and the tool's `path` argument) accepts `source/path`, `source/path 12-40`, `source/path:12-40`, `source/path#12-40`, `source/path#L12-L40` and `source/path 12` (single line); anything unparseable is not a `bor` locator at all and renders as an ordinary code block (the body is then the model's own, exactly as today, with a Copy button and no `VERBATIM` badge). Resolution normalizes to the canonical form, so the client's match is a string compare.
|
||||||
|
- **D8 — caps, kill switch, observability (A8).** `answer_text: bool = True` (`BOR_ANSWER_TEXT`, the master switch — `false` = the tool is not offered, the prompt clause is absent, resolution does not run, no `answer_text` key: every frame and every byte-identity contract is exactly pre-phase), `show_text_max_chars: int = 12_000` (per block; `0` = inert, negative fails startup naming the field — the `agent_max_rounds`/`read_max_chars` pattern), `show_text_max_blocks: int = 5` (per answer, in answer order; `0` = inert, negative fails startup). Worst-case frame growth 5 × 12 000 = 60 KB against the 256 KB budget, which the phase-167 byte-budget line (`api … bytes=…`, `grep ' :: api '`) makes visible. The §9 per-turn line gains ONE additive trailing field `verbatim_blocks=N` (the blocks actually sent, tool-registered + fence-resolved, deduped) — `tests/unit/test_chat_gate.py`'s pinned line is updated in the same task, never silently.
|
||||||
|
- **D9 — resolution scope (the honesty posture).** Resolution runs on GROUNDED turns only (the agent ran). A `bor` fence hallucinated into a DEFLECTED answer has no registered blocks and the gate said "I haven't done anything like that" — it renders the unresolved state rather than leaking KB text through a turn that just disclaimed it. `done.sources` / `related` semantics are NOT changed (no regression to the citation surface): the panel itself carries the path, the line range, a Copy button and a link to the document viewer (the git origin link stays the viewer's job, phase 145's `origin_for_source` — the panel does not duplicate it) — the citation affordance sits at the point of use.
|
||||||
|
- **NOT touched:** `show_image` / `SHOW_IMAGE_TOOL` / `answer_image` (image display is out of scope, A6 — verified, not modified), the `ls`/`read`/`grep` contracts, the SSE event-type LIST (A15 stays at seven types — this phase rides `done`, no 8th event), `documents`/`chunks` schema (no migration: `saved_chats.messages` is raw JSONB, the phase-50/160 precedent), `document.js` (the viewer renders documents, not answers), `app/core/response_budget.py`, `scripts/*`, `frontend/assets/router.js`.
|
||||||
|
|
||||||
|
## Locked decisions (owner-confirmed 2026-10-14, from the phase-authoring interview)
|
||||||
|
|
||||||
|
- **A1 — BOTH mechanisms in this phase:** the `show_text` tool AND the inline ` ```bor ` locator fence. (The owner chose "A + B in one phase" over the tool alone.)
|
||||||
|
- **A2 — whole-file default + server clamp:** `offset`/`limit` optional; omitted = the whole document; overshoot clamps; past-the-end refuses to the model with the true total; the caption always shows the SERVER's resolved extent. No `expect`-anchor snapping (considered and deferred — the clamp plus the model's own `grep`/`read` line numbers cover it).
|
||||||
|
- **A3 — the tool returns the verbatim slice** plus the "do not retype" instruction (not a terse confirmation, not the read_coverage-conditional variant).
|
||||||
|
- **A4 — the bytes ride INLINE** in `done.answer_text` and persist in the `bor.chat.v1` brain record: no new bytes route, no new SSE event type, no new anonymous surface (a shared page renders from the share's own record, the same trust model as the phase-123 chat image).
|
||||||
|
- **A5 — Copy buttons on EVERY fenced code block in an answer** (live, restored and shared), but the `VERBATIM` badge + caption belong ONLY to a server-resolved block. The clipboard goes through the phase-55 degradation (a non-secure origin gets the inline select-to-copy fallback) — never a bare `navigator.clipboard` assumption.
|
||||||
|
- **A6 — images are out of scope:** `show_image` keeps images; `show_text` on an `is_image` document is REFUSED with a pointer to `show_image`. The phase's E2E asserts the answer figure is unchanged.
|
||||||
|
- **A7 — the model's fence body never reaches the DOM** (see D3). Unresolvable = honest, visible, non-copyable-as-truth state.
|
||||||
|
- **A8 — settings:** `BOR_ANSWER_TEXT` default `true` (master switch), `BOR_SHOW_TEXT_MAX_CHARS` default `12000`, `BOR_SHOW_TEXT_MAX_BLOCKS` default `5`; `0` = inert, negative = loud startup failure.
|
||||||
|
- **A9 — additive wire only:** `answer_text` is dropped when empty (absent, never `null`) — every frame without blocks, every pre-phase record and the whole default-off configuration stay byte-identical. No PLAN §2/§4 anchor changes beyond recording these revisions.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `124_answer_images` (complete) — the display-tool precedent this phase mirrors end to end (`SHOW_IMAGE_TOOL` → `holder.shown_images` → `done.answer_image` → `appendAnswerImage` → persistence → shared page).
|
||||||
|
- `131_read_line_ranges` / `141_read_range_dedupe_fix` (complete) — `offset`/`limit` + `_read_range_args` + the `lines: N-M of T` header contract and `splitlines()` indexing.
|
||||||
|
- `145_git_source_links` (complete) — the document viewer's `origin_for_source` link, which the panel links TO rather than duplicating.
|
||||||
|
- `167_api_payload_diet` (complete) — the byte-budget line that will show the `done`-frame growth.
|
||||||
|
- `180_markdown_inline_links_images` (todo, already authored) — independent of this phase; both touch `markdown.js` and `styles.css`, so run it FIRST (or rebase) to avoid a renderer conflict.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_settings_and_kill_switch.md` — the three `Settings` fields + validators + the three `CATALOG` rows + `.env.example` (+ their unit/integration pins). Inert: nothing reads them yet.
|
||||||
|
2. `02_verbatim_resolver_module.md` — `app/rag/verbatim.py`: the locator grammar, the canonical normalizer, `splitlines()` slicing, the line-boundary cap and `resolve_answer_blocks` (+ the resolver unit suite).
|
||||||
|
3. `03_show_text_tool.md` — `SHOW_TEXT_TOOL`, the `agent_tools` gate, the `_execute_tool` branch, the refusal set, `AgentHolder.shown_texts` (+ agent unit suite).
|
||||||
|
4. `04_prompt_teaching_clause.md` — `SHOW_TEXT_CLAUSE` inside `<tools>`, `build_high_prompt(show_text=…)`, the `app/api/chat.py` call site (+ byte-identity pins when off).
|
||||||
|
5. `05_done_frame_and_persistence.md` — the `VerbatimBlockRef` schema, `ChatDoneEvent.answer_text`, `ChatMessageRecord.answer_text`, the visible-text accumulator + resolution + `verbatim_blocks=N` in the per-turn line (+ integration/frame tests).
|
||||||
|
6. `06_markdown_locator_and_panel_css.md` — `renderMarkdown(md, opts)`'s opt-in `verbatim` pass → `.verbatim[data-bor-locator]` (body dropped; ordinary fences keep rendering exactly as today), and the `styles.css` panel/copy-button CSS contract. Inert until task 07 opts in and wraps `<pre>` nodes at the DOM level.
|
||||||
|
7. `07_hydration_and_copy_buttons.md` — `frontend/assets/verbatim.js` (+ `index.html`/`shared.html` script tags): hydrate-at-`done`, restore, shared page, error/settled paths, `rememberBrainTurn` persistence, copy buttons on every answer code block.
|
||||||
|
8. `08_e2e_and_docs_pass.md` — the mock triggers (`show_text` call, corrupt-body fence, fence-only, off-app), `tests/e2e/test_verbatim_show_text.py` in isolation, the regression set, the full gates, the README/AGENTS full consistency pass, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit:** `tests/unit/test_config.py` (the three defaults; env overrides; `0` accepted as inert; each negative value fails validation naming the field); `tests/unit/test_settings_catalog.py` (the completeness pin green with the three new rows); `tests/unit/test_verbatim_resolver.py` (NEW — the locator grammar incl. every tolerant form + rejects, canonical normalization, whole-file/offset/limit/clamp/past-end, the line-boundary cap + `TRUNCATION_MARKER` + shrunk `end_line`, `fences_in` ordering + dedupe, tool-then-fence dedupe, `max_blocks` cut, an `is_image` doc refused, an unknown path unresolved, a fenced `bor` line inside an ordinary fence, an unclosed fence, an empty answer); `tests/unit/test_agent.py` (the `show_text` branch: success shape + headers, `shown_texts`, refusal set — missing args, non-string, unknown path did-you-mean, image doc, bad range args, offset past end, block cap, duplicate identical show; every refusal counts in nothing and executes nothing; `agent_tools` offers 4 vs 5 tools by flag); `tests/unit/test_prompts.py` (the clause is INSIDE `<tools>` before `</tools>`, the off-build is byte-identical to pre-phase, `TOOLS_SECTION` itself unchanged); `tests/unit/test_chat_gate.py` (the per-turn line's new `verbatim_blocks=N`, and `0` on a turn with none); `tests/unit/test_frontend_verbatim_markdown.py` + `tests/unit/test_frontend_verbatim_blocks.py` + a CSS pin module (the `test_frontend_*` text-pin convention: the `verbatim` option, the dropped body, `data-bor-locator`, the `.verbatim`/`.code-block`/`.verbatim-copy` markers, the new `<script>` tags in `index.html` + `shared.html` (plain relative `src` — no `cache-busting` token in the path, the `?v=` rewrite is the middleware's job, phase 160 A7), `shared.js` parity, no inline `onclick` — CSP `default-src 'self'`).
|
||||||
|
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` (all three rows: `bool` + two `int`s, correct groups/labels, live apply without a restart, reset-to-env); the chat API suite — a grounded turn with a `show_text` call and/or a fence yields `done.answer_text` with the SERVER's text and extent; a turn with none has NO `answer_text` key (byte-identity); `answer_text=false` yields the pre-phase frame byte-identically; the saved-chat round trip preserves `answer_text` (`extra="forbid"` accepts it — the phase-113 lesson) and `GET /api/chats/{id}` returns it.
|
||||||
|
- **E2E (AGENTS rule 4, task 08):** `tests/e2e/test_verbatim_show_text.py`, run **in isolation** — the panel's text equals the fixture file's bytes (the mock's corrupted fence body never appears anywhere in the DOM), the caption reads `source/path · lines 12–40 of N` from the SERVER's extent, `VERBATIM` badge + line numbers + Copy button copy EXACTLY the panel's bytes (clipboard read with granted permissions, or the fallback field), the panel's open link targets `/document?source=…&path=…`, an unresolvable locator renders the honest state with no model bytes, restored-after-reload and the anonymous shared page render the same panel, every ordinary code block has a Copy button and no badge, and the `show_image` figure still renders (A6 regression).
|
||||||
|
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (setup, commands, configuration, features, conventions). This phase's user-visible additions: the `show_text` capability + the `bor` locator syntax in the feature list, the three `BOR_*` settings in the env table, and (in `AGENTS.md` rule 5's UI list) the verbatim-panel contract "an answer's code block is either verbatim-and-badged or plainly the model's — never a silently-substituted hybrid".
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `ask "show me the quadlet for <x>"` on a running app (real or mock LLM) renders a panel whose text is byte-identical to `documents.content`'s resolved slice, with a caption whose numbers came from the server, and a Copy button whose clipboard content equals that slice exactly.
|
||||||
|
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` prints `True 12000 5`; each negative value refuses startup naming its field.
|
||||||
|
- [ ] `GET /api/admin/settings` lists `answer_text`, `show_text_max_chars`, `show_text_max_blocks`; `PUT`/`DELETE` round-trip per the overlay contract.
|
||||||
|
- [ ] A grounded turn's `done` frame carries `answer_text[]` with `source,path,title,start_line,end_line,total_lines,text,truncated,locator`; a turn without verbatim content carries NO key; with `BOR_ANSWER_TEXT=false` the frame is byte-identical to pre-phase and only FOUR tools are offered.
|
||||||
|
- [ ] The per-turn log line ends with `verbatim_blocks=N` (`grep` it on a shown turn and on a plain turn).
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_verbatim_show_text.py -v --no-cov` green **in isolation**; `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_markdown_tables.py` (and the phase-180 suite if it landed first) green in isolation.
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||||
|
- [ ] no behavior change in completed phases (image answers, `read`/`grep`/`ls`, deflected byte-identity, the document viewer, saved-chat byte-identity).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(agent): display file contents verbatim with a show_text tool and server-resolved locator fences`); the phase dir moves to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 01 — Three settings and the kill switch (inert)
|
||||||
|
|
||||||
|
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Add the three `Settings` fields the phase runs on — the master `answer_text` switch and the two budgets — with their admin-panel rows and `.env.example` entries. Nothing reads them yet: after this task the app behaves byte-identically to pre-phase.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `app/config.py` — in the agent-tools neighborhood of `read_max_chars` (L296), add:
|
||||||
|
- `answer_text: bool = True` — the master switch for verbatim display (`BOR_ANSWER_TEXT`). Docstring: names the env var, the default `True`, and what `False` means (the `show_text` tool is not offered, the `<tools>` clause is absent, locator resolution does not run, the `done` frame carries no `answer_text` key — every byte-identity contract is exactly pre-phase). Model the docstring on `answer_images` (L539) but note the DEFAULT IS TRUE here (the owner chose the fix on by default; the switch is an escape hatch, not an opt-in).
|
||||||
|
- `show_text_max_chars: int = 12_000` (`BOR_SHOW_TEXT_MAX_CHARS`) — the per-block character budget for one displayed slice; `0` = inert (no block displays), a negative value is a typo and fails startup naming the field (the `_read_max_chars_non_negative` validator at L686 is the exact pattern).
|
||||||
|
- `show_text_max_blocks: int = 5` (`BOR_SHOW_TEXT_MAX_BLOCKS`) — the per-answer cap on displayed blocks; `0` = inert, negative fails startup (the `agent_max_rounds` validator at L678 is the pattern).
|
||||||
|
Add the two `@field_validator`s next to `_read_max_chars_non_negative`, each raising a `ValueError` that names the field (house rule: fail loud, never half-configured).
|
||||||
|
2. `app/core/settings_catalog.py` — three `SettingMeta` rows beside `read_max_chars` (L267): `answer_text` → `"Answer text"`, `show_text_max_chars` → `"Show text max chars"`, `show_text_max_blocks` → `"Show text max blocks"`, all in the `Agent tools` group (the `read_max_chars` group). The completeness pin in `tests/unit/test_settings_catalog.py` (`set(CATALOG) == set(Settings.model_fields)`) is the enforcement — without the rows the build fails.
|
||||||
|
3. `.env.example` — three lines in the agent-tools block, each with a one-line comment: `BOR_ANSWER_TEXT=true`, `BOR_SHOW_TEXT_MAX_CHARS=12000`, `BOR_SHOW_TEXT_MAX_BLOCKS=5`. State the `0`/negative semantics in the comments (the surrounding file's style).
|
||||||
|
4. No consumer changes. Do NOT touch `app/rag/agent.py`, `app/rag/prompts.py`, `app/api/chat.py`, `app/schemas.py` or any frontend file in this task.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit:** `tests/unit/test_config.py` — defaults (`True`, `12000`, `5`); each env override applies; `BOR_SHOW_TEXT_MAX_CHARS=0` and `BOR_SHOW_TEXT_MAX_BLOCKS=0` are accepted (inert, not an error); each negative value raises a validation error whose message names the field (the `read_max_chars` test is the template). `tests/unit/test_settings_catalog.py` — the completeness pin green, the three rows carry the right type/group/label.
|
||||||
|
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the three rows appear in `GET /api/admin/settings` (`bool` + two `int`, group `Agent tools`); `PUT` stores each raw value and it resolves live without a restart; `DELETE` resets each to the env default; the `0` values are storable.
|
||||||
|
- **Coverage:** **>90%** on this task's new/modified code (the validators are the only new logic — cover both branches of each).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` → `True 12000 5`.
|
||||||
|
- [ ] `BOR_SHOW_TEXT_MAX_CHARS=-1 uv run python -c "import app.config"` refuses with a message naming `show_text_max_chars` (same for `show_text_max_blocks`).
|
||||||
|
- [ ] The three rows are visible in the admin Settings panel's `Agent tools` group (or asserted by the integration test if the browser is not running).
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the env table) — the phase's final pass re-checks both project-wide.
|
||||||
|
- [ ] no behavior change in completed work (no consumer reads the new fields yet).
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Task 02 — `app/rag/verbatim.py`: the locator grammar, the slicer, the resolver
|
||||||
|
|
||||||
|
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The single source of truth for "which exact lines of which indexed document, and what are their bytes". Pure functions for the locator grammar and the slicing, one bare DB lookup, and `resolve_answer_blocks` which the API layer calls at `done` time. Inert: nothing calls it yet (tasks 03 and 05 wire it in).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. Create `app/rag/verbatim.py`. Module docstring states the contract (PLAN §6's tool surface extension, phase 182 D1–D7): *the model names a location, this module supplies the bytes*; and the import rule — **this module must never import `app.rag.agent`** (agent.py imports the pure half of this module; a reverse import cycles). Allowed imports: `sqlalchemy`, `app.models.Document`, `app.rag.retriever.TRUNCATION_MARKER`, `app.config.Settings`, stdlib.
|
||||||
|
2. `LOCATOR_LANG = "bor"` and `@dataclass(frozen=True) class VerbatimBlock` — `source: str`, `path: str`, `title: str`, `start_line: int`, `end_line: int`, `total_lines: int`, `text: str`, `truncated: bool`, `locator: str` (the canonical form, built by the factory, never passed in by a caller). The dataclass docstring pins: `start_line`/`end_line` are 1-based and INCLUSIVE and describe exactly the lines `text` contains after any cap; `text` never carries a trailing newline (lines are joined with `"\n"`).
|
||||||
|
3. `canonical_locator(raw: str) -> str | None` — the tolerant parser of D7. Accepts, in order: a combined `source/path` (the first `/` splits it, the `read`/`grep` identity contract) followed by an optional separator + range — separators ` `, `:`, `#`, `#L`; ranges `12-40`, `12–40` (en dash), `L12-L40`, or a bare `12` (one line, end == start). Returns `source/path` for a whole document and `source/path#12-40` for a range (always the canonical `#` + plain digits, never the en dash, never the `L`). Returns `None` (NOT a locator) for: an empty/whitespace input, a value with no `/` in the path part, a non-numeric or non-positive bound, `start > end`, trailing junk after the range, or a leading `bor` token left in by the caller. No DB, no normalization of case (paths are case-sensitive row values).
|
||||||
|
4. `fences_in(answer: str) -> list[str]` — every `bor` fence's RAW locator string, in document order, deduped keeping the first occurrence. Rules: the opening line is a fence whose info string is exactly `` ```bor `` + whitespace + a non-empty remainder (a bare `` ```bor `` with no path is NOT a locator — it is an ordinary code block and must not be swallowed); the fence closes at the next line that is `` ``` `` (trimmed); an UNCLOSED fence contributes nothing (defensive — a half-streamed answer); the body is read and DISCARDED (this module never looks at what the model typed); nested `` ``` `` inside the body is not supported and is a documented limitation (the fence ends at the first closer, the same rule `markdown.js` applies).
|
||||||
|
5. `@dataclass(frozen=True) class SliceResult` — `text: str`, `first_line: int`, `last_line: int`, `total_lines: int`, `past_end: bool` — and `slice_lines(content: str, offset: int | None = None, limit: int | None = None) -> SliceResult`. Indexing is `content.splitlines()` with the SAME 1-based semantics as `read` (`app/rag/agent.py` L1836) and `grep`: `offset` omitted → 1; `limit` omitted → to the end; `end = min(offset + limit - 1, total_lines)` (an overshooting `limit` CLAMPS); `offset > total_lines` → `past_end=True` with empty text and the true `total_lines` (the caller builds the teaching refusal); `total_lines == 0` (an empty document) → `past_end=True`. Never raises for a range problem — the refusal is the caller's, the total always travels with the result.
|
||||||
|
6. `cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]` — the character budget applied at a LINE boundary, returning `(text_to_show, last_line_shown, truncated)`. `cap <= 0` raises `ValueError` (a mis-wired caller must be loud — the `show_text_max_chars == 0` kill switch is checked by `resolve_answer_blocks`, never by reaching here). When `len(text) <= cap`: `(text, last_line_of(text), False)`. Otherwise cut to the last whole line that fits within `cap` and return `(cut + "\n" + TRUNCATION_MARKER, first_line + n_lines_in_cut - 1, True)`; a first line LONGER than `cap` cuts that one line mid-way (the only sub-line cut, documented) and still ends with the marker. The marker is the shared `app.rag.retriever.TRUNCATION_MARKER` — never a second copy of the text.
|
||||||
|
7. `document_by_identity(db: Session, combined: str) -> Document | None` — split at the FIRST `/` (source names are directory basenames and never contain one); `select(Document).where(Document.source == source, Document.path == path)` → the row or `None`. Docstring: this is the same `(source, path)` identity contract as `app.rag.agent._resolve_path` REIMPLEMENTED as a bare lookup — no did-you-mean suggestions here (the tool branch already has `_resolve_path` for teaching; the fence resolver only needs yes/no), and the duplication exists to keep the import direction one-way. A traversal string is simply a row that does not exist.
|
||||||
|
8. `block_for(db, combined: str, *, offset: int | None = None, limit: int | None = None, cap: int, doc: Document | None = None) -> VerbatimBlock | None` — resolve (or take the caller's already-resolved `doc`), refuse `None` for: no document, `doc.is_image` (image display belongs to `show_image`, locked A6), an empty/past-end slice; otherwise `slice_lines` → `cap_slice` → the block with `locator = f"{source}/{path}"` (+ `f"#{first}-{last}"` when the request named a range or the cap cut it — the canonical form task 07 matches on). `title` is the row's title.
|
||||||
|
9. `resolve_answer_blocks(db, answer: str, shown: Sequence[VerbatimBlock], settings: Settings) -> list[VerbatimBlock]`:
|
||||||
|
- returns `[]` when `not settings.answer_text`, or `settings.show_text_max_blocks == 0`, or `settings.show_text_max_chars == 0` (the kill switches, checked HERE so no caller has to remember);
|
||||||
|
- for each `fences_in(answer)` raw string in order: `canonical_locator` → skip when `None`; `block_for(...)` → skip when `None` (unknown path, image doc, past end) — the caller sees simply "not among the blocks", which is what task 07 renders as the unresolved state;
|
||||||
|
- dedupe by canonical locator, the FENCE winning over a same-locator entry from *`shown`* (the fence controls placement);
|
||||||
|
- append the `shown` blocks the answer never located, in their original call order;
|
||||||
|
- truncate the list to `settings.show_text_max_blocks`.
|
||||||
|
Docstring pins the ordering rule (answer order first, then the unlocated tool blocks) — task 07's DOM order and the E2E assertions depend on it.
|
||||||
|
10. Update the module-level `__all__` and keep every public name docstring'd in the house style (contract + phase reference).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit:** create `tests/unit/test_verbatim_resolver.py`.
|
||||||
|
- `canonical_locator`: `homelab/a.md` → itself; `homelab/a.md 12-40` / `:12-40` / `#12-40` / `#L12-L40` / `12–40` (en dash) → `homelab/a.md#12-40`; `homelab/a.md 12` → `#12-12`; `None` for `a.md` (no source), `""`, `" "`, `homelab/a.md 0-5`, `homelab/a.md 40-12`, `homelab/a.md x-y`, `homelab/a.md 12-40 extra`, `"bor homelab/a.md"`.
|
||||||
|
- `slice_lines`: whole file, offset-only, limit-only, both, overshooting limit clamps `last_line` to the total, `offset` past the end → `past_end` + true total, empty content → `past_end`, and the `splitlines()` agreement check (a fixture whose line `N` is asserted by content, mirroring what `grep` would report).
|
||||||
|
- `cap_slice`: under cap (untouched, `truncated=False`), cut at a line boundary (text ends with the shared `TRUNCATION_MARKER`, `last` = the last whole line shown, and the marker line is not counted as content), a single over-long first line, `cap=0` raises `ValueError`.
|
||||||
|
- `document_by_identity`: hit, miss, traversal-shaped miss, first-slash splitting when the path itself contains slashes.
|
||||||
|
- `block_for`: whole-document block, ranged block, canonical locator with/without a range, image doc → `None`, unknown path → `None`, past-end → `None`, capped block's `end_line` equals the lines actually shown.
|
||||||
|
- `resolve_answer_blocks`: no fences + no shown → `[]`; fence-only (no tool call) resolves; tool-only appends; fence + identical shown block → ONE block (fence wins, appears at the fence position); two fences keep answer order; `show_text_max_blocks=1` truncates; every kill-switch combination returns `[]`; an unresolvable fence is simply absent; a duplicated fence appears once.
|
||||||
|
- Session-fixture style follows the existing agent/resolver unit suites (the same in-memory/session fixture the `tests/unit/test_agent.py` helpers use).
|
||||||
|
- **Coverage:** **>90%** on this task's new/modified code (every branch above is a test).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/unit/test_verbatim_resolver.py -v` green.
|
||||||
|
- [ ] `uv run python -c "import app.rag.verbatim as v; print(v.canonical_locator('homelab/a.md#L12-L40'))"` prints `homelab/a.md#12-40`.
|
||||||
|
- [ ] `grep -n "import" app/rag/verbatim.py` shows NO import of `app.rag.agent` (the one-way rule).
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (nothing user-visible yet — the phase's final pass re-checks both project-wide).
|
||||||
|
- [ ] no behavior change in completed work (nothing calls this module yet).
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# Task 03 — The `show_text` agent tool: name a location, the server shows the bytes
|
||||||
|
|
||||||
|
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The 5th agent tool: `show_text(path, offset?, limit?)` displays an indexed document (or a line range of one) to the user verbatim. The handler resolves the slice through task 02, records it for the `done` frame, and hands the SAME bytes back to the model with an explicit "do not retype" instruction.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `app/rag/agent.py` — add `SHOW_TEXT_TOOL` right after `SHOW_IMAGE_TOOL` (L501), same OpenAI function shape:
|
||||||
|
- `name: "show_text"`.
|
||||||
|
- `description` (this IS the behavior contract — the phase-124 discipline): "Display part of a document to the user EXACTLY as stored — the user sees the server's bytes and can copy them. Call it when the user wants to see, copy or run a file (a quadlet, a config, a script): they copy from what is displayed, so do NOT retype the content yourself after calling it — describe it instead. `path` is the combined `source/path` string exactly as shown in `ls`. `offset` (1-based first line) and `limit` (line count) are optional — omit them to display the whole document; use them when `grep`/`read` told you which lines matter. The displayed line numbers come from the server, not from your arguments: an overshooting `limit` is clamped and an `offset` past the end is refused with the true line count. Only text documents — for an image use `show_image`. Call one tool at a time — wait for this result before your next call."
|
||||||
|
- `parameters`: `path` (string, required, the `read` "a bare document path will not resolve" teaching), `offset` and `limit` (integers, optional, the `read` wording reused).
|
||||||
|
2. `agent_tools(settings)` (L538) — append `SHOW_TEXT_TOOL` when `settings.answer_text`, keeping `AGENT_TOOLS` byte-identical and the list fresh per call (the `show_image` precedent; the offer order is `ls`, `read`, `grep`, `show_image`, `show_text`). Update the docstring (four base + two display tools, each gated by its own flag).
|
||||||
|
3. `AgentHolder` (L1600–L1680) — add `shown_texts: list[VerbatimBlock] = field(default_factory=list)` with a docstring paragraph in the `shown_images` style: the blocks in CALL order, capped by `settings.show_text_max_blocks`, read by `app.api.chat` after the stream for the `done` frame's `answer_text` (the `shown_images`/`read_docs` read-after-stream precedent); refused shows append nothing.
|
||||||
|
4. Refusal constants beside the `show_image` ones (L571–L590), each single-braced and pinned in the house style:
|
||||||
|
- `MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."`
|
||||||
|
- `SHOW_TEXT_NOT_A_TEXT_DOC` — one `{arg}` field: the doc is a standalone image; `show_image` displays images. (LOCKED A6 — never silently converted.)
|
||||||
|
- `SHOW_TEXT_BLOCK_CAP` — one `{max}` field: that many blocks are already displayed this answer — describe them instead of showing more.
|
||||||
|
- `SHOW_TEXT_ALREADY_SHOWN` — one `{locator}` field: that exact range is already on screen — say so instead of showing it again.
|
||||||
|
Reuse `BAD_RANGE_ARGS` (L628) and `OFFSET_PAST_END` (L671) verbatim for the range problems (the same teaching, the same echoed values — one copy of each string in the codebase).
|
||||||
|
5. `_execute_tool` (L1681) — a `show_text` branch placed next to the `show_image` branch (L2023), in this order:
|
||||||
|
1. `raw_path = call.arguments.get("path")`; non-string/blank → `MISSING_SHOW_ARGS`-shaped refusal (`MISSING_SHOW_TEXT_ARGS`).
|
||||||
|
2. `settings.answer_text` false (defensive — the tool was not offered) → the same refusal copy as the round-cap-off posture: state that the tool is unavailable; counts in nothing.
|
||||||
|
3. `doc, _source, _path = _resolve_path(db, arg)`; `doc is None` → `_no_document_refusal(db, arg)` (the phase-72 did-you-mean teaching, reused unchanged).
|
||||||
|
4. `doc.is_image` → `SHOW_TEXT_NOT_A_TEXT_DOC`.
|
||||||
|
5. `ok, err = _read_range_args(offset, limit)` — reuse the existing validator unchanged; `not ok` → `err`.
|
||||||
|
6. `len(holder.shown_texts) >= settings.show_text_max_blocks` → `SHOW_TEXT_BLOCK_CAP` (checked BEFORE the work; `0` therefore disables the tool's effect while keeping it offered — pinned in the tests).
|
||||||
|
7. `block_for(db, arg, offset=offset, limit=limit, cap=settings.show_text_max_chars, doc=doc)` → `None` (empty document / `offset` past the end) → `OFFSET_PAST_END.format(total_lines=...)` (compute the total with `len(doc.content.splitlines())`, the same expression `read` uses); a returned block whose canonical `locator` already appears in `holder.shown_texts` → `SHOW_TEXT_ALREADY_SHOWN`.
|
||||||
|
8. Success: `holder.shown_texts.append(block)`; `holder.tool_calls += 1` (executed-only counting, the house rule); return the result string in this EXACT shape (the `read`/`show_image` header contract the E2E mock's `_READ_RESULT_PREFIX` parses):
|
||||||
|
```
|
||||||
|
Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.
|
||||||
|
Document {source}/{path}:
|
||||||
|
date: {YYYY-MM-DD}
|
||||||
|
{origin lines — the same _origin_lines(origin) helper the read branch uses}
|
||||||
|
lines: {first}-{last} of {total_lines}
|
||||||
|
{block.text}
|
||||||
|
```
|
||||||
|
`block.text` already ends with `TRUNCATION_MARKER` when the char cap cut it; when `block.truncated` is true append one line, in the `READ_TRUNCATION_NOTICE` spirit, naming that the rest is NOT displayed and that the user can open the document (no `offset`/`limit` re-show loop is implied — the full text is a click away, and only the displayed slice is what the user copies).
|
||||||
|
- Every refusal path appends nothing, increments nothing, and consumes a round exactly like the existing refusals.
|
||||||
|
6. Module docstring — extend the numbered loop contract (the `show_image` paragraph's neighborhood) with the `show_text` contract and the locked rules A1–A3/A6: the model names a location, the server supplies the bytes, the displayed extent is the server's, images belong to `show_image`, the fence form is resolved at the API layer (task 05), not here.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit (`tests/unit/test_agent.py`, the `show_image` family is the template):** `agent_tools` offers 4 tools with `answer_text=False` (byte-identical to pre-phase, `show_image` last) and 5 with it on (`show_text` present, correct order); a successful whole-document show (result headers in order, `lines: 1-N of N`, the slice verbatim, `shown_texts` length 1, `tool_calls` 1, `read_docs` unchanged); a ranged show (clamped `limit` reflected in the `lines:` header); a capped show (`TRUNCATION_MARKER` present + the extra notice line); every refusal — missing `path`, non-string `path`, unknown path (did-you-mean copy, no `shown_texts`), image doc, bad range args, `offset` past the end (the true total named), block cap (`max_blocks=1`, second call refused), duplicate identical show; each refusal asserts `tool_calls` and `shown_texts` untouched; `max_blocks=0` → the first call is refused with the cap copy.
|
||||||
|
- **Integration:** none required in this task (the frame wiring is task 05) — but `tests/integration/` must stay green, and any test that pins the exact tool-offer array is updated here, deliberately, with a comment.
|
||||||
|
- **Coverage:** **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/unit/test_agent.py -v -k show_text` green, including all nine refusal cases and the counting assertions.
|
||||||
|
- [ ] With `answer_text=True`, a scripted agent turn returns a result whose 2nd line starts `Document ` and whose `lines:` header matches the slice actually returned.
|
||||||
|
- [ ] `grep -n "SHOW_TEXT_" app/rag/agent.py` shows the four new constants, each used by exactly one branch.
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the tool surface list in README's agent-tools section) — the phase's final pass re-checks both project-wide.
|
||||||
|
- [ ] no behavior change in completed work (`ls`/`read`/`grep`/`show_image` results byte-identical; the pre-phase four-tool offer byte-identical when the flag is off).
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 04 — Teach the model to point instead of type: the `<tools>` clause
|
||||||
|
|
||||||
|
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The HIGH prompt teaches two things it has never said: **the user copies from what is displayed, not from your prose**, and the exact ` ```bor ` locator-fence syntax for placing a displayed block. Gated by `settings.answer_text`, byte-identical when off.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `app/rag/prompts.py` — add `SHOW_TEXT_CLAUSE: str` right after `SHOW_IMAGE_CLAUSE` (L379), with a docstring comment in the same style (why it exists, what each sentence pins, and that `TOOLS_SECTION` stays byte-identical). The clause text must contain, in this order:
|
||||||
|
- the display contract: "`show_text` displays part of a document to the user EXACTLY as stored — the user copies from what is displayed, not from your prose.";
|
||||||
|
- when to call it: "Call it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file — instead of writing the content into your answer yourself.";
|
||||||
|
- the do-not-retype rule (the phase's reason for existing): "After calling it, do NOT repeat the displayed lines in your answer — describe them, explain them, point at them. Anything you type yourself may differ from the file by a character, and the user cannot tell.";
|
||||||
|
- the placement syntax: "To put a displayed block exactly where you want it in your answer, write a locator line on its own: three backticks followed by `bor`, a space, the combined `source/path`, and optionally the line range (` 12-40`). Leave its body EMPTY — the app replaces it with the exact stored text. Never type the file's content inside a `bor` block.";
|
||||||
|
- the honesty-about-numbers rule: "Never guess line numbers — use the numbers `grep` and `read` reported. When you are not sure which lines, OMIT the range and display the whole document: the numbers you display are the server's, never yours.";
|
||||||
|
- the failure rule: "If `show_text` refuses, say what you could not display — do not reconstruct it from memory.";
|
||||||
|
- the image pointer: "For an image use `show_image` instead."
|
||||||
|
2. `build_high_prompt(...)` (L759) — add `show_text: bool = False` after `show_image`; when true, the clause rides INSIDE `<tools>` AFTER the `SHOW_IMAGE_CLAUSE` (when that is present) and before the closing `</tools>` tag, using the existing "section text minus the final `</tools>` + clause + closing tag" mechanism (L782–L791 documents it). Both clauses compose independently: `show_image` only → exactly the phase-124 output; `show_text` only → the three-tool copy + the text clause; both → image clause then text clause; neither → byte-identical to pre-phase. Update the docstring (a *show_text* paragraph mirroring the *show_image* one, incl. the note that the deflection prompt NEVER carries it — a deflected turn has no tools and D9 forbids resolving locators on it).
|
||||||
|
3. `app/api/chat.py` (the grounded `TurnPlan` builder, L516–L529) — pass `show_text=settings.answer_text` next to `show_image=settings.answer_images`, with the same style of phase comment.
|
||||||
|
4. Do NOT change `TOOLS_SECTION`, `build_deflect_prompt`, or any mock in `tests/e2e/mock_llm.py` (the new scripted triggers belong to task 08).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit (`tests/unit/test_prompts.py` + the `tests/unit/test_prompt_lock.py` pins):** the off-build (`show_text=False`, the default) is byte-identical to the pre-phase string for every combination of the other arguments; the on-build contains the clause INSIDE `<tools>` (index of the clause text > index of `<tools>`, `< ` index of `</tools>`); `show_image=True, show_text=True` yields the image clause before the text clause; `TOOLS_SECTION` itself unchanged (the existing lock test still passes untouched); the clause names `show_text`, `bor`, the do-not-retype rule and the "omit the range" rule (the E2E and the persona depend on those exact ideas); the deflect prompt never contains `show_text` or `bor`.
|
||||||
|
- **Integration:** the grounded-turn integration tests stay green (the prompt string is not pinned byte-for-byte there beyond the `<tools>` marker); add one assertion that a grounded plan built with `answer_text=True` carries the clause and one that `answer_text=False` does not.
|
||||||
|
- **Coverage:** **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green.
|
||||||
|
- [ ] `uv run python -c "from app.rag.prompts import build_high_prompt as b; print('show_text' in b([], show_text=True))"` prints `True`, and the same call without the flag prints `False`.
|
||||||
|
- [ ] A grounded request against a running app shows the clause in the model request (log/assert via the E2E mock's request capture in task 08 — here, the unit proof is enough).
|
||||||
|
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the prompt/tool teaching summary) — the phase's final pass re-checks both project-wide.
|
||||||
|
- [ ] no behavior change in completed phases (deflected prompt byte-identical; the phase-124 clause position untouched).
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 05 — The `done` frame's `answer_text`, the visible-text accumulator, the log field
|
||||||
|
|
||||||
|
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Put the resolved bytes on the wire and in the persisted record: the turn's clean visible answer text is accumulated during streaming, resolved through `app.rag.verbatim` at `done`-build time, and sent as the ADDITIVE `answer_text` array; the per-turn log line gains `verbatim_blocks=N`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
|
||||||
|
1. `app/schemas.py` — `class VerbatimBlockRef(BaseModel)` beside `SourceRef` (L169): `source: str = Field(max_length=120)`, `path: str = Field(max_length=1000)`, `title: str = Field(max_length=500)` (the `SourceRef` caps, the same identity), `start_line: int = Field(ge=0)`, `end_line: int = Field(ge=0)`, `total_lines: int = Field(ge=0)`, `text: str = Field(max_length=600_000)` (the phase-83 answer-text cap — generous above the 12 000 default, so an operator raising `BOR_SHOW_TEXT_MAX_CHARS` does not hit the wire limit), `truncated: bool = False`, `locator: str = Field(max_length=2100)`. Docstring: the phase-182 display block; `text` is SERVER-supplied bytes from `documents.content`, never model output; 1-based inclusive lines.
|
||||||
|
2. `ChatDoneEvent` (L305) — `answer_text: list[VerbatimBlockRef] | None = None` + docstring in the `answer_image` style, and extend its `model_serializer` to POP the key when `None` **or empty** (absent, never `null`, never `[]`: every frame without blocks stays byte-identical to pre-phase — the pin that proves it lives in the integration suite). Note the two deliberate differences from `answer_image` in the docstring: it is a LIST (several blocks per answer, D1), and it carries the TEXT itself (A4 — no bytes route, no new anonymous surface, a shared page renders from its own record).
|
||||||
|
3. `ChatMessageRecord` (L1400–L1510) — `answer_text: list[VerbatimBlockRef] | None = None` on the BRAIN record, popped when `None`/empty in the existing `_drop_image_when_absent` serializer (extend its docstring). Required so `extra="forbid"` does not 422 the done-time auto-save of a turn with blocks (the phase-113 lesson, applied up front) and so the restore path + the shared page re-render the panels. No migration (`saved_chats.messages` is raw JSONB, the phase-50/160 precedent).
|
||||||
|
4. `app/api/chat.py`:
|
||||||
|
1. Next to the visible-content counter (`content_chars`, L934) add `visible_parts: list[str]` and append EVERY visible (post-scaffold-filter) content piece in the piece loop — the same pieces counted into `content_chars`, never `thinking` (D3: locators only ever come from the answer). Bounded by the model's output-token budget; note that in the comment.
|
||||||
|
2. In the `done` builder (L1320–L1365), after `answer_ref`: `blocks = resolve_answer_blocks(db, "".join(visible_parts), holder.shown_texts, settings) if not plan.deflected else []` (D9 — a deflected turn resolves NOTHING: no fence leaks KB text through a turn that just disclaimed it; the deflected path has no `holder`, so the guard is load-bearing). Convert with `VerbatimBlockRef(**dataclasses.asdict(b))` in order, pass `answer_text=blocks or None`.
|
||||||
|
3. Per-turn log line (L1272–L1291): append `verbatim_blocks=%d` after `scaffold_stripped=%d` (the count of blocks SENT, i.e. after dedupe + `show_text_max_blocks`; `0` on every turn without them, including deflected ones) and the module docstring's line-format block with the new field. This is the §9 additive revision — update `tests/unit/test_chat_gate.py`'s pinned line in the same commit, never silently.
|
||||||
|
4. `holder.shown_texts` is read AFTER the stream (the `shown_images` precedent) — do not reorder the generator's finally/cancel semantics (a cancelled turn still writes neither line nor `query_log` row).
|
||||||
|
5. Nothing in `app/schemas.py`'s existing validators changes; no new SSE event type (A15 stays at seven).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- **Unit:** `tests/unit/test_chat_gate.py` (or the module that owns the per-turn line pins) — the line ends `… scaffold_stripped=N verbatim_blocks=0` on a plain grounded turn, `verbatim_blocks=2` on a turn whose holder has two blocks, `verbatim_blocks=0` on a deflected turn (and the deflected frame still carries no `answer_text`).
|
||||||
|
- **Integration:** the chat API/streaming suite — a scripted grounded turn that calls `show_text` yields a `done` frame whose `answer_text[0]` has the SERVER's `text`, the clamped extent, the canonical `locator`, and `truncated` false under the cap; a turn whose ANSWER contains a ` ```bor ` fence (no tool call) resolves the same block from the fence; fence + identical tool call produce exactly ONE entry; three blocks with `show_text_max_blocks=2` send two in answer order; a turn with no blocks serializes WITHOUT the `answer_text` key (byte-identity pin, exact-dict compare); `answer_text=False` yields the pre-phase frame byte-for-byte AND `verbatim_blocks=0`; a deflected frame carries no `answer_text` even when its (mocked) text contains a fence. The saved-chat suite — `POST /api/chats` with a brain record carrying `answer_text` is accepted (`extra="forbid"`), `GET /api/chats/{id}` returns it unchanged, and a record without it round-trips byte-identically to pre-phase.
|
||||||
|
- **Coverage:** **>90%** on this task's new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] A scripted SSE turn with a `show_text` call returns `answer_text: [{source,path,title,start_line,end_line,total_lines,text,truncated,locator}]` and the text equals `documents.content` for that range exactly.
|
||||||
|
- [ ] `grep ' :: api ' ` on that request shows the `done`-bearing response's byte line (the phase-167 audit) — record the observed size in the phase's task notes if it approaches the 256 KB budget.
|
||||||
|
- [ ] The per-turn log line shows `verbatim_blocks=N`; a plain turn shows `0`.
|
||||||
|
- [ ] `uv run pytest` green (incl. the updated line pins), coverage >90%, ruff + pyright clean.
|
||||||
|
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the SSE `done` frame contract in README's API section; AGENTS rule 10's log-line list) — the phase's final pass re-checks both project-wide.
|
||||||
|
- [ ] no behavior change in completed phases (`answer_image`, `sources`, `related`, `budget_exhausted`, the deflected byte-identity, the cancelled-turn contract).
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user