Compare commits
79
Commits
152d4f2b88
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a45f744604 | ||
|
|
ae080ac7f3 | ||
|
|
fb17d15fbe | ||
|
|
4f20c256d7 | ||
|
|
c6ffa050a6 | ||
|
|
47456a44df | ||
|
|
140a343b92 | ||
|
|
f20d40e4fe | ||
|
|
c054b47df5 | ||
|
|
644f7de696 | ||
|
|
632fb88e43 | ||
|
|
065304b52f | ||
|
|
7b5ea7c2ee | ||
|
|
b34b8ca672 | ||
|
|
814a400e6d | ||
|
|
6880bd0866 | ||
|
|
3d82a72b6f | ||
|
|
2b57180998 | ||
|
|
76887639f3 | ||
|
|
50658580e9 | ||
|
|
dad54a2854 | ||
|
|
149e753952 | ||
|
|
c6223c93dc | ||
|
|
36eddce672 | ||
|
|
dd119d534a | ||
|
|
2a5df7cb31 | ||
|
|
bd271a6f52 | ||
|
|
375169bbfc | ||
|
|
938d6cf09d | ||
|
|
da441ea799 | ||
|
|
a86caec821 | ||
|
|
f27106a638 | ||
|
|
b21b05eaee | ||
|
|
e9ae6e2ba3 | ||
|
|
c4f098d0c0 | ||
|
|
1200437bf8 | ||
|
|
abe7388dda | ||
|
|
542e568d25 | ||
|
|
a88a5a3713 | ||
|
|
05f7d0645e | ||
|
|
8fac4e64d4 | ||
|
|
a12fdcdf0a | ||
|
|
a2b2f283fd | ||
|
|
2ac37455f0 | ||
|
|
ab9391d019 | ||
|
|
b0c7aed613 | ||
|
|
8d70aa946c | ||
|
|
dcb1062b83 | ||
|
|
34d60efbc1 | ||
|
|
6ba46e735a | ||
|
|
74eb2e265c | ||
|
|
855fa27a30 | ||
|
|
eebc824636 | ||
|
|
690fd98036 | ||
|
|
051f62deb7 | ||
|
|
8172956c15 | ||
|
|
cb8767512a | ||
|
|
4d7d0f6297 | ||
|
|
aa297ea9b1 | ||
|
|
31bd903a93 | ||
|
|
3f752fb77f | ||
|
|
1513981c1c | ||
|
|
742eef955b | ||
|
|
c275735db4 | ||
|
|
966c4b0540 | ||
|
|
12fa69b7ea | ||
|
|
c7cfcd26b5 | ||
|
|
edf6048a0d | ||
|
|
9d4c813cae | ||
|
|
dd524246ec | ||
|
|
b233c8d714 | ||
|
|
863519e980 | ||
|
|
ab3f5f685b | ||
|
|
32a3ec4f5e | ||
|
|
486ceb4019 | ||
|
|
ba4841a811 | ||
|
|
aa65d2d691 | ||
|
|
a8c0402ad2 | ||
|
|
de49a03c4d |
+41
-4
@@ -84,6 +84,25 @@ A15 — `thinking`, `tool`, unlimited tool rounds under
|
|||||||
never-truncated contract is unchanged; the `read`-tool cap is
|
never-truncated contract is unchanged; the `read`-tool cap is
|
||||||
the only exception, owner-permitted 2026-09-10.
|
the only exception, owner-permitted 2026-09-10.
|
||||||
|
|
||||||
|
Phase 182 (verbatim text display, owner chat 2026-10-14) adds
|
||||||
|
**`show_text(path, offset?, limit?)` as the fifth agent tool** — offered
|
||||||
|
only when `BOR_ANSWER_TEXT=1` (the default), gated exactly like
|
||||||
|
`show_image`, with its own `<tools>` clause and its own kill switch — and
|
||||||
|
the **additive `done.answer_text`** field above. The new settings are
|
||||||
|
`BOR_ANSWER_TEXT` (master switch, ON), `BOR_SHOW_TEXT_MAX_CHARS` (per-block
|
||||||
|
budget, 12 000) and `BOR_SHOW_TEXT_MAX_BLOCKS` (per-answer budget, 5), all
|
||||||
|
following the `agent_max_rounds` pattern (`0` inert, negative refuses
|
||||||
|
startup). The principle in one sentence: **the model points, the server
|
||||||
|
supplies the bytes** — a ` ```bor source/path 12-40 ` fence or a
|
||||||
|
`show_text` call names a location (canonical locator `source/path` or
|
||||||
|
`source/path#start-end`, 1-based and inclusive), and the displayed text is
|
||||||
|
sliced from `documents.content` server-side, so what the user copies is the
|
||||||
|
knowledge base, never a re-typed approximation. **A1–A6 and A15 are
|
||||||
|
otherwise
|
||||||
|
UNCHANGED:** no new SSE event type, no new table or column, no new model,
|
||||||
|
endpoint or transport, no change to retrieval, the honesty gate, the
|
||||||
|
persona text or the embedding dimension.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. High-Level Architecture
|
## 3. High-Level Architecture
|
||||||
@@ -129,7 +148,9 @@ POST /api/chat {message, history?} (auth: require_user)
|
|||||||
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
||||||
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
||||||
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
||||||
│ + <tools> → agent loop (ls/read/grep; round-capped
|
│ + <tools> → agent loop (ls/read/grep, plus show_image when
|
||||||
|
│ BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
|
||||||
|
│ "show the user an artifact" tools, gated independently; round-capped
|
||||||
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
|
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
|
||||||
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
|
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
|
||||||
byte-identical direct-stream path) → turbo streamed
|
byte-identical direct-stream path) → turbo streamed
|
||||||
@@ -139,8 +160,20 @@ POST /api/chat {message, history?} (auth: require_user)
|
|||||||
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
|
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
|
||||||
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
|
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
|
||||||
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
|
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
|
||||||
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
|
suggestions[], budget_exhausted?, answer_text[]?}` (terminal). **Phase 182
|
||||||
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
|
extends the `done` frame ADDITIVELY** — `answer_text` is the list of
|
||||||
|
server-resolved verbatim blocks (`locator`, `source`, `path`, `start`,
|
||||||
|
`end`, `total_lines`, `text`, `format`, `truncated`) the answer displayed;
|
||||||
|
there is **no eighth event type**, the key is OMITTED when no block was
|
||||||
|
displayed, and with `BOR_ANSWER_TEXT=0` it is absent on every turn (the
|
||||||
|
frame is then byte-identical to pre-phase). The same array persists on the
|
||||||
|
saved chat record, so restore and the anonymous shared page re-render the
|
||||||
|
panels from the server's bytes. Failure: `error` (terminal — no `done`, no
|
||||||
|
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
|
||||||
|
`tools=None` request never emits a `tool` frame; a round-capped turn
|
||||||
|
whose final reply carries no visible content answers with the budget
|
||||||
|
fallback instead of erroring (phase 160, recorded revision of the
|
||||||
|
phase-71 cap-path terminal, owner-confirmed 2026-10-05).
|
||||||
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
|
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
|
||||||
still counted). LLM retries (A15 extension): retry count and delay are
|
still counted). LLM retries (A15 extension): retry count and delay are
|
||||||
configurable (defaults: 3 retries, 5 s flat delay), **only before a
|
configurable (defaults: 3 retries, 5 s flat delay), **only before a
|
||||||
@@ -268,8 +301,12 @@ scripts.…`, and tests all honor it.
|
|||||||
```
|
```
|
||||||
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
|
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
|
||||||
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
|
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
|
||||||
thinking_chars=… tool_calls=N total_ms=… retries=N scaffold_stripped=N
|
thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
|
||||||
|
retries=N scaffold_stripped=N verbatim_blocks=N
|
||||||
```
|
```
|
||||||
|
(`verbatim_blocks=` — phase 182 — is the count of verbatim display blocks
|
||||||
|
actually SENT on the `done` frame, `0` on every turn without them,
|
||||||
|
deflected included.)
|
||||||
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
|
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
|
||||||
executed only; a cancelled turn writes neither line nor `query_log`
|
executed only; a cancelled turn writes neither line nor `query_log`
|
||||||
row.)
|
row.)
|
||||||
|
|||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Phase 150 — retire `span.sync-result`: `#sync-progress` is the single sync text surface
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L5 — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||||
|
**Context:** the Sources page (`frontend/index.html`, the `view-rag` section). Phase 149 added `#sync-progress` (L493–495 — `<div class="sync-progress" id="sync-progress" hidden aria-hidden="true">` + the inner `<span class="sync-progress-text" id="sync-progress-text">`), the bordered panel under the button row that carries the full untruncated live label while a run is in flight. The pre-149 announcer survived beside it: `<span class="sync-result" id="sync-result" role="status" aria-live="polite">` (L511) — today the SOLE aria-live region, which receives (a) the same live label while a run runs (a byte-identical duplicate of the box), (b) the settle counts line (`fmtSyncResult(detail)` — "N added · M updated · …"), (c) the "Sync cancelled" line, (d) the cancel-request failure copy, (e) `""` on the sync-failure / idle / upload-settle paths (plus the settle label "Synced HH:MM" and the idle "Sync sources", written through the shared writer). The single writer is `frontend/assets/sources.js` `setSyncLabel(mainText, countText, chunkText, running)` (L809–843): it writes the button parts, the `title`, the box text + `hidden = !running`, the button `aria-label`, AND `syncResult.textContent = full` (L834). The `syncResult` element handle is L750; the other write sites are L1092 (`applySyncSuccess` — the counts line), L1107 (`applySyncIdle` — clear), L1125 (`applySyncCancelled` — "Sync cancelled"), L1213/L1224 (the upload settles — clear), L1291/L1298/L1310 (the cancel paths — clear + "Couldn't cancel the sync — try again."). CSS: `.sync-progress` L2252, `.sync-progress-text` L2259, `.sync-result` L2284–2290, the mobile `.sync-result` override at L5639, and the "the sync-result shape" comments at L2938 / L3659 (those rules belong to OTHER elements — the upload success line and the action feedback line — and keep their own classes).
|
||||||
|
**Locked (owner decision, chat — this phase):** `#sync-progress` becomes the **single** sync text surface AND the **sole** aria-live region: `role="status" aria-live="polite"` moves onto the box, `aria-hidden` is dropped, and `span.sync-result` is **deleted** from the template. The box keeps the phase-149 running behavior AND takes over the announcer's content: it is visible while a run is in flight (the live label) and stays visible after a settle to carry the result line (the counts / "Sync cancelled" / cancel-failure copy), hiding only when its text is empty (idle, upload settles, failures — exactly the states where `#sync-result` is empty today). Every string that reaches `#sync-result` today reaches the box instead — no copy changes.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One element, one live region: the sync/upload live label, the settle result line, the cancel line and the click copy all render in `#sync-progress`, which announces them exactly once; `#sync-result` no longer exists in the DOM, the JS, or the CSS.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `149_sync_progress_box` (complete) — the box, its single-writer wiring, and its CSS are what this phase folds the announcer into. Pipeline order only: no phase is pending in `todo/`.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The element (task 01):** `#sync-progress` keeps its id, class, and inner `#sync-progress-text` span (the textContent target — the house rule: text only, never HTML) and gains `role="status" aria-live="polite`; `aria-hidden="true"` is REMOVED (an aria-live region that is aria-hidden announces nothing — the two attributes are mutually exclusive by intent). It stays ship-hidden (`hidden`) — the anonymous-safe/idle state. `span.sync-result` (L511) and its explanatory comment block (L496–510) are deleted; the box's comment (L483–492) is rewritten to say it is now the SOLE announcer (the phase-149 "presentational (aria-hidden)" sentence is retired — leave a one-line note that phase 150 folded the announcer in, the file's pinned-comment convention).
|
||||||
|
- **The writer (task 02):** introduce ONE function that owns the box — `setSyncProgress(text)` — which sets `syncProgressText.textContent = text` and `syncProgress.hidden = (text === "")` (the global `[hidden]` rule does the display). Two content modes, decided at the callsites (the same "the callsites are the only decision sites" rule phase 149 established):
|
||||||
|
* **running mode** — `setSyncLabel(main, count, chunk, running)` keeps writing the box with `full` **only when `running === true`** (exactly today's reveal rule). When `running === false` it does NOT touch the box: the idle/settle button labels ("Sync sources", "Synced HH:MM") must not fill a big panel (today they land on the small `#sync-result` line where they read as a duplicate of the button — that artifact is retired, not migrated).
|
||||||
|
* **result mode** — the settle paths call `setSyncProgress(text)` explicitly: `applySyncSuccess` → `fmtSyncResult(status.detail)` (the counts line), `applySyncCancelled` → `"Sync cancelled"`, `applySyncFailure` → `""` (the banner + modal carry the error), `applySyncIdle` → `""`, the two upload settles → `""` (the upload result lives on the Sources page — phase 90 A3), `requestCancel`'s network/non-2xx branches → `"Couldn't cancel the sync — try again."` (string unchanged). `requestCancel`'s opening clear (L1291, which today blanks the announcer while the run is still in flight) becomes a NO-OP on the box — the live label stays up until the poll's `cancelled` branch settles it (blanking would hide the running surface for up to one 2 s poll).
|
||||||
|
The `syncResult` handle (L750) and every `if (syncResult)` guard are deleted. There is no 202/409 click copy: `startSync`'s 202/409 branch enters the running state (L1255–1262), so the box reveals the bare "Syncing…" label through `setSyncLabel` as today.
|
||||||
|
- **The CSS (task 03):** delete the `.sync-result` rule (L2284–2290) and the mobile `.sync-result` override (L5639); the two "the sync-result shape" comments (L2938, L3659) are reworded to reference `.sync-progress-text` (they describe the mono/soft-ink language those other elements share — the rules themselves and their selectors are untouched). `.sync-progress` needs NO new rule for the settled state (the panel already reads as a calm box; the global `[hidden] { display: none !important }` rule keeps it gone when empty — verify it still applies, the box sets no `display` of its own). Contrast stays AA: `.sync-progress-text` uses `--ink-soft` on `--surface` (8.2:1 built-in dark, AA in both built-ins).
|
||||||
|
- **The test migration (tasks 04/05):** mechanical selector migration — `#sync-result` → `#sync-progress` (assert on the box's text, i.e. `#sync-progress-text` for textContent, `#sync-progress` for `hidden`/attributes). Settled-state copy assertions must stay byte-identical (the counts line string itself does not change).
|
||||||
|
- **ASSUMPTION A1 (owner-locked):** the box stays visible after settle to hold the counts line and hides only when its text is empty. The alternative (box running-only, result line elsewhere) was rejected.
|
||||||
|
- **ASSUMPTION A2:** the settle label "Synced HH:MM" is no longer written to the live region (it stays on the button, visible). The announcer hears the counts line that superseded it in the same tick a moment ago — the merge makes one surface, and a panel reading "Synced HH:MM" under a button reading "Synced HH:MM" is the duplicate this retirement exists to remove.
|
||||||
|
- **ASSUMPTION A3:** the box's settled appearance needs no new CSS (the same bordered panel reads fine for a result line); only the running-only min-height stays, so the panel does not jump between modes.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_html_single_surface.md` — delete `span#sync-result`, move `role=status aria-live=polite` onto `#sync-progress`, drop `aria-hidden`, rewrite the two comment blocks.
|
||||||
|
2. `02_js_single_writer.md` — `sources.js`: add `setSyncProgress`, route all 9 announcer sites through it, delete the `syncResult` handle, change the box visibility rule to text-driven.
|
||||||
|
3. `03_css_retire_sync_result.md` — `styles.css`: remove `.sync-result` + its mobile override, reword the two shape comments.
|
||||||
|
4. `04_unit_test_migration.md` — `tests/unit/test_sync_button.py` (20 refs) + `tests/unit/test_frontend_sync_upload.py` (26 refs) migrated to the box.
|
||||||
|
5. `05_e2e_sync_family.md` — the 12 sync-family E2E files' selectors migrated + the dedicated `tests/e2e/test_sync_progress_box.py` extended with the single-live-region + settled-line assertions.
|
||||||
|
6. `06_e2e_image_pdf_family.md` — the 6 image/PDF E2E files migrated (their sync-done barriers move to the box) + the a11y/no-CDN/full-gate pass and the phase commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the two frontend-parsing suites (`test_sync_button.py`, `test_frontend_sync_upload.py`) pin the writer contract on the box — running label visible + announced, settle counts line, "Sync cancelled", the idle/upload clears, the "Couldn't cancel" copy — and assert `#sync-result` is ABSENT from the template (the retirement is pinned, not implied).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` in isolation (the slow-LLM proxy recipe already in that file) proves in-flight → settle → idle on the real sync path; every pre-existing sync/image/PDF E2E that touched the announcer passes after migration (12 sync-family files + 6 image/PDF files).
|
||||||
|
- Coverage: **>90%** on `app/` (this phase touches no `app/` logic — the floor must simply not regress).
|
||||||
|
- Accessibility gate (AGENTS.md rule 5): exactly ONE `aria-live` region in the sync area; the box is a `status` region (polite — no alert spam); the button keeps its accessible name in both states; focus-visible and contrast unchanged.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "sync-result\|syncResult" frontend/ tests/` returns NOTHING (the element, its class, and its JS handle are gone from source and tests)
|
||||||
|
- [ ] `#sync-progress` carries `role="status" aria-live="polite"` and no `aria-hidden`; it is the only `aria-live` element in the sync area
|
||||||
|
- [ ] in flight: the box shows the full untruncated live label (wrapping, never clipped); at settle: the box shows the counts line byte-identical to the pre-phase `#sync-result` string ("N added · …") and "Sync cancelled" for a cancel; at idle / upload-settle / sync-failure: the box is `hidden` and empty
|
||||||
|
- [ ] the button is still the icon-only spinner while running and restores "Sync sources" on settle (phase 149 behavior unchanged)
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] no behavior change in completed phases (the cancel control, the 409 double-trigger, the error banner/modal, the upload run's status, the sync status API)
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — `index.html`: delete `span#sync-result`, make `#sync-progress` the live region
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The template ships ONE sync text surface: `#sync-progress`, which is now the `role="status"` `aria-live="polite"` announcer; `span.sync-result` no longer exists in the DOM.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` (the `view-rag` Sources section, L483–511):
|
||||||
|
- `#sync-progress` (L493) becomes
|
||||||
|
`<div class="sync-progress" id="sync-progress" role="status" aria-live="polite" hidden>`
|
||||||
|
— `aria-hidden="true"` is REMOVED (an aria-live region must not be aria-hidden), `role`/`aria-live` are ADDED, `hidden` stays (ship-hidden = the idle state, anonymous-safe). The inner `<span class="sync-progress-text" id="sync-progress-text"></span>` (L494) is unchanged (it stays the textContent target).
|
||||||
|
- DELETE the `#sync-result` span (L511) and its comment block (L496–510, the "#sync-result is the aria-live announcer …" paragraph).
|
||||||
|
- REWRITE the box's own comment (L483–492) so it states the phase-150 contract: the box carries the full untruncated live label while a run is in flight AND the settle result line after it; it is the SOLE aria-live announcer for the sync area (one live region, no double announce); the button is the spinner while running (`.is-running` hides the label parts). Keep the file's comment voice (phase-prefixed, rationale-first).
|
||||||
|
2. Nothing else in the template changes: `#sync-btn` and its label parts, `#sync-cancel-btn`, `#sync-error-banner` / `#sync-error-text`, the sync modal, and the Sources gate are untouched.
|
||||||
|
3. No CDN/external references are introduced (AGENTS.md rule 6 — the page stays fully self-hosted).
|
||||||
|
4. ASSUMPTION: the box keeps its current position (between `.page-head` and the error banner) — the owner asked to remove the span, not to move the box.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: this task's own gate is the template parse — the pre-existing suites that read `index.html` still find `#sync-progress` (with the new attributes) and no longer find `#sync-result`. Tasks 04/05 do the full migration; if a suite fails ONLY on the missing `#sync-result`, note it for task 04 rather than editing test expectations here beyond what the parse requires.
|
||||||
|
- Coverage: n/a (template task — `app/` coverage floor unchanged).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -n "sync-result" frontend/index.html` returns nothing
|
||||||
|
- [ ] `#sync-progress` has `role="status"`, `aria-live="polite"`, `hidden`, and NO `aria-hidden`
|
||||||
|
- [ ] the sync area contains exactly one `aria-live` element
|
||||||
|
- [ ] `uv run pytest tests/unit -q` is green or failing ONLY on the `#sync-result` expectations tasks 04/05 own
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Task 02 — `sources.js`: one box writer (`setSyncProgress`), the announcer handle deleted
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every string that used to land on `#sync-result` lands on `#sync-progress` instead, written by one function; the `syncResult` handle is gone and the box's visibility is owned by that single writer.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/sources.js` — add the box writer next to `setSyncLabel` (L809):
|
||||||
|
```js
|
||||||
|
function setSyncProgress(text) {
|
||||||
|
if (!syncProgressText || !syncProgress) return;
|
||||||
|
syncProgressText.textContent = text;
|
||||||
|
syncProgress.hidden = text === "";
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(textContent only — the house rule, never HTML; the `hidden` attribute drives `display:none` through the global `[hidden]` rule.)
|
||||||
|
2. `setSyncLabel` (L809–843): replace the phase-149 pair `syncProgressText.textContent = full;` + `syncProgress.hidden = !running;` (L838–839) with `if (running) setSyncProgress(full);` — the running label still flows through the single writer, and a non-running call leaves the box alone (the idle/settle button labels "Sync sources" / "Synced HH:MM" must NOT fill the panel; the panel is the live label + the result line, nothing else). The `title`, the `aria-label` (`running ? full : "Sync sources"`), and the label parts are untouched. Delete `if (syncResult) syncResult.textContent = full;` (L834) — the box IS the announcer now.
|
||||||
|
3. Delete the element handle `const syncResult = root.querySelector("#sync-result");` (L750) and every `if (syncResult)` guard. Migrate the eight remaining announcer sites to `setSyncProgress(...)`, at the SAME position in the same order:
|
||||||
|
- `applySyncSuccess` (L1092) → `setSyncProgress(fmtSyncResult(status.detail))` (the counts line — the string builder is unchanged);
|
||||||
|
- `applySyncFailure` (L1107) → `setSyncProgress("")` (the banner + modal carry the error; the box hides);
|
||||||
|
- `applySyncCancelled` (L1125) → `setSyncProgress("Sync cancelled")`;
|
||||||
|
- `applySyncIdle` (L~1130) → `setSyncProgress("")` (the true idle — retry-ready, nothing to announce);
|
||||||
|
- the upload-success settle (L1213) and the upload-failed settle (L1224) → `setSyncProgress("")` (the upload's result line lives on the Sources page — phase 90 A3);
|
||||||
|
- `requestCancel`'s network-failure branch (L1298) and its non-2xx tail (L1310) → `setSyncProgress("Couldn't cancel the sync — try again.")` (string byte-identical);
|
||||||
|
- `requestCancel`'s opening clear (L1291) becomes a **no-op on the box** — the run is still in flight at that moment, so blanking would hide the running surface for up to one 2 s poll; the poll's `cancelled` branch settles it. Leave a short comment saying why the clear was dropped (the phase-134 "never stale" contract is kept by the control's immediate disable).
|
||||||
|
4. Update the module/function comments that name `#sync-result` as the announcer (the file cites it in ~15 comment lines — L160, 321, 375, 670, 677, 700, 706, 723, 744, 789, 859, 894, 949, 1006, 1120, 1209, 1256, 1274, 1286): rename the references to `#sync-progress` and keep the rationale (one live region, the untruncated label, A4). Do not change any behavior while doing it.
|
||||||
|
5. ASSUMPTION A2 (owner-visible): the settle label "Synced HH:MM" is no longer written to the live region — it stays on the button; the counts line is the announced result.
|
||||||
|
6. ASSUMPTION: the re-attach path (`initSyncButton`, the page-load mid-run entry) needs no new code — it goes through `enterSyncRunningState` → `setSyncLabel(..., true)`, which reveals the box (the never-stale contract holds).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: task 04 migrates the two frontend suites; this task's own gate is `uv run pytest tests/unit -q` failing only on the expectations task 04 owns, and `grep -n "syncResult" frontend/assets/sources.js` returning nothing.
|
||||||
|
- Coverage: n/a (frontend task).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -n "syncResult\|sync-result" frontend/assets/sources.js` returns nothing
|
||||||
|
- [ ] `setSyncProgress` is the only writer of the box's text/visibility; `setSyncLabel` writes it only when `running`
|
||||||
|
- [ ] the counts line, "Sync cancelled", and "Couldn't cancel the sync — try again." all render in `#sync-progress`; the idle/upload-settle/sync-failure states leave it hidden and empty
|
||||||
|
- [ ] the running label, the button `title`, and the button `aria-label` behavior are byte-identical to pre-phase
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 03 — `styles.css`: retire `.sync-result`, keep the box correct in both modes
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
No CSS rule targets the deleted element, and `#sync-progress` reads correctly as both the running label and the settled result line.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css`:
|
||||||
|
- DELETE the `.sync-result` rule (L2284–2290 — `display:block; color:var(--ink-soft); font-family:var(--mono); font-size:0.8rem; white-space:nowrap; padding-block:0.5rem;`) and its "Last-result announcer" comment.
|
||||||
|
- DELETE the mobile `.sync-result` override inside the phone media query (L5639) — read it first and confirm it only tweaks the deleted element (if it also styles a surviving selector, keep that part).
|
||||||
|
- REWORD the two comments that reference "the sync-result shape" (L2938 — the upload success line `role=status`; L3659 — the action feedback line `role=status`) to name `.sync-progress-text` as the shared mono/soft-ink language. The RULES themselves (their own selectors) are untouched — only the cross-reference text changes.
|
||||||
|
- `.sync-progress` (L2252) / `.sync-progress-text` (L2259) need no new rule: the panel already carries the border/surface/radius/padding and the `min-height: 4rem`; the box sets no `display` of its own, so the global `[hidden] { display: none !important }` rule keeps it gone when empty (verify by reading that global rule — if it is scoped to something narrower, add `.sync-progress[hidden] { display: none; }`).
|
||||||
|
- Update the `.sync-progress-text` comments that say "the .sync-result mono language / size / ink" (L2261–2266) so they no longer cite a dead class (keep the contrast note: `--ink-soft` on `--surface` is a pinned AA pair).
|
||||||
|
2. No new colors, no new fonts, no external assets (AGENTS.md rule 6). Contrast stays ≥4.5:1 in both built-in themes (the ink/surface pairs are unchanged).
|
||||||
|
3. ASSUMPTION A3: the settled result line uses the same panel styling as the running label (no `.is-settled` variant) — the owner asked for one surface, not two looks.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/E2E: the migrated suites (tasks 04/05) assert the box's visibility states; the theme suites (`tests/unit/test_theme*.py`, the custom-theme contrast gate) pass unchanged — no theme token changed.
|
||||||
|
- Coverage: n/a (CSS task).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -n "sync-result" frontend/assets/styles.css` returns nothing
|
||||||
|
- [ ] `#sync-progress` is `display:none` while `hidden` and a bordered panel when shown, at desktop and 360 px
|
||||||
|
- [ ] the upload success line and the action feedback line still render exactly as before (their own rules untouched, only comments reworded)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 04 — unit suites: migrate the `#sync-result` source pins to the box
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The source-level suites that pin the sync UI no longer reference `#sync-result` / `.sync-result`, and they pin the new contract instead: one live region, box-driven.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_sync_button.py` (20 references) — the pins are regex/string reads of `frontend/index.html`, `frontend/assets/sources.js`, `frontend/assets/styles.css`:
|
||||||
|
- L232–241 (`test …sync-result…`): the "polite live region right after the button" pin moves to `#sync-progress` — assert the div carries `role="status"` and `aria-live="polite"`, does NOT carry `aria-hidden`, and sits after the button row; rename the test accordingly;
|
||||||
|
- ADD a pin that the retired element is gone: `assert 'id="sync-result"' not in index_html` (the retirement is pinned, not implied);
|
||||||
|
- L747–751 (the `.sync-result` CSS block pin): retarget to `.sync-progress-text` (assert the mono family, the `--ink-soft` color, the `overflow-wrap: anywhere`) and assert `.sync-result` has no rule in `styles.css`;
|
||||||
|
- L1984 (the DOM-order pin `main_end < cancel_start < sync-result`): re-anchor on `id="sync-progress"`;
|
||||||
|
- L371, 384, 465, 652, 2067 and the module docstring (L11, 43, 56): reword the "title + #sync-result" language to "title + #sync-progress" (the A4 untruncated-label contract is unchanged — it just names the surviving surface);
|
||||||
|
- the cancel-request pin (~L2067) that asserted the announcer clears on click: update to the new contract — the box is NOT blanked at cancel-request time (the live label stays until the poll settles), the control disables immediately.
|
||||||
|
2. `tests/unit/test_frontend_sync_upload.py` (26 references) — same migration: the `fmtSyncLabel` / `enterSyncRunningState` pins that assert the full untruncated label reaches the announcer now assert it reaches `#sync-progress` (via `setSyncProgress` / the `running` branch of `setSyncLabel`); the A3 pin ("the settle never renders upload counts into #sync-result") becomes "the upload settles write an empty string to the box" (`setSyncProgress("")`); the docstring references (L9, 12, 61, 79) are reworded.
|
||||||
|
3. Keep every assertion's INTENT: same strings, same states, one different selector. Do not weaken a pin to make it pass — if a behavior genuinely changed (the "Synced HH:MM" announcer write, the cancel-request clear), pin the NEW behavior explicitly and say why in the test docstring.
|
||||||
|
4. ASSUMPTION: no new test file is created for the unit layer — the retirement is a migration of existing pins plus the two "element is gone" assertions.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/unit/test_sync_button.py tests/unit/test_frontend_sync_upload.py -v` green.
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage stays >90% (no `app/` change in this phase).
|
||||||
|
- Coverage: n/a for frontend source; the two suites must not lose assertions (compare the collected test count before/after — it must not drop).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "sync-result\|syncResult" tests/unit/` returns nothing
|
||||||
|
- [ ] both suites assert the box is the `role="status" aria-live="polite"` region and that `#sync-result` is absent from the template and from `styles.css`
|
||||||
|
- [ ] test count is not lower than before the migration; both files green
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 05 — E2E migration (sync family) + the dedicated `test_sync_progress_box.py` assertions
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every sync-flow Playwright suite drives and asserts the box instead of the deleted span, and the phase's dedicated suite proves the merged contract end to end (one live region, running label + settled result line).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. Migrate the sync-family suites (the `sync-result` reference count in parentheses) — replace the selector `#sync-result` with the box (`#sync-progress` for visibility/attributes, `#sync-progress-text` for `text_content()`), keeping every expected STRING byte-identical:
|
||||||
|
- `tests/e2e/test_sync_button.py` (2), `tests/e2e/test_sync_cancel.py` (2 — the "Sync cancelled" line and the cancel lifecycle), `tests/e2e/test_sync_model_down.py` (2), `tests/e2e/test_sync_upload_progress.py` (2), `tests/e2e/test_upload_no_scan.py` (2), `tests/e2e/test_sync_summary_visibility.py` (3), `tests/e2e/test_sync_changed_only.py` (3), `tests/e2e/test_sync_label_mobile.py` (1), `tests/e2e/test_git_source_dates.py` (1), `tests/e2e/test_git_source_branch.py` (4), `tests/e2e/test_force_push_resync.py` (4), `tests/e2e/test_sync_chunk_progress.py` (8 — the "· chunk n/m" announcer pins).
|
||||||
|
- Where a suite asserted "the announcer is empty" it now asserts `expect(page.locator("#sync-progress")).to_be_hidden()`; where it asserted announcer text it asserts the box's text. A suite that asserted the announcer was VISIBLE while running keeps that assertion on the box (it is visible while running — phase 149).
|
||||||
|
2. `tests/e2e/test_sync_progress_box.py` (11 references) — the dedicated suite for this phase (A16: one file per story, run in isolation). Extend it, keeping the existing running-state proofs:
|
||||||
|
- **template pin:** exactly one `[aria-live]` element in the sync area, and it is `#sync-progress`; `#sync-result` is absent (`page.locator("#sync-result").count() == 0`);
|
||||||
|
- **running:** unchanged (icon-only button + the full untruncated label in the box, the long `source/relative/path` wrapping at 360 px and desktop);
|
||||||
|
- **settled success:** the box is VISIBLE and its text equals the counts line the pre-phase announcer produced (the `fmtSyncResult` shape — "N added · …"), the button reads "Synced HH:MM";
|
||||||
|
- **settled cancel:** the box reads "Sync cancelled" (the phase-134 neutral result — no banner, no modal);
|
||||||
|
- **settled failure:** the box is hidden/empty and the error banner + modal carry the error (unchanged);
|
||||||
|
- **idle re-attach:** a page load with no run leaves the box hidden.
|
||||||
|
3. Keep the suites MOCK-ONLY (the `tests/e2e/slow_llm.py` delay-proxy recipe already in the file) and run each migrated file in isolation while editing it: `uv run pytest tests/e2e/test_<name>.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||||
|
4. ASSUMPTION: no new E2E file — the phase's behavioral proof belongs to the existing `test_sync_progress_box.py` (the phase-149 dedicated suite is the box's suite; this phase extends it rather than forking a near-duplicate).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green IN ISOLATION (the phase's gate file).
|
||||||
|
- Every migrated sync-family file green in isolation; then the full `uv run pytest` green.
|
||||||
|
- Coverage: >90% on `app/` (unchanged — no `app/` edit).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rln "sync-result" tests/e2e/` returns nothing after task 06 also lands (this task clears the sync family)
|
||||||
|
- [ ] the dedicated suite proves: one live region, running label in the box, counts line visible at settle, "Sync cancelled" on cancel, hidden on failure/idle
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 06 — E2E migration (image / PDF family) + the no-CDN and a11y gate pass
|
||||||
|
|
||||||
|
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The remaining suites that touch the announcer — the image and PDF stories, which wait on sync result text — drive the box, and the whole gate passes with the span gone.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. Migrate the image/PDF-family suites (reference count in parentheses), same rule as task 05 — selector changes only, expected strings byte-identical:
|
||||||
|
- `tests/e2e/test_image_documents.py` (2), `tests/e2e/test_image_anchor_context.py` (2), `tests/e2e/test_answer_images.py` (2), `tests/e2e/test_pdf_support.py` (2), `tests/e2e/test_pdf_image_context.py` (2), `tests/e2e/test_pictorial_pdf.py` (2).
|
||||||
|
- These use the announcer mostly as a SYNC-DONE barrier (wait for the counts line before querying the KB). Keep them as barriers, just on the box: wait for `#sync-progress-text` to contain the expected result text (or for the box to become visible with the counts line) — do NOT replace them with `wait_for_timeout`.
|
||||||
|
2. Confirm the no-CDN integration test still passes (AGENTS.md rule 6 — the index page has no external `<script src=>`/`<link href=>`): the template edit added no external reference.
|
||||||
|
3. Accessibility pass on the merged surface (AGENTS.md rule 5, PLAN §7): exactly one `aria-live` region in the sync area; `role="status"` (polite — the failure path keeps its `role="alert"` banner); the button keeps an accessible name in both states (`aria-label` full label while running, "Sync sources" otherwise); `:focus-visible` on the button + Cancel untouched; `--ink-soft` on `--surface` contrast ≥4.5:1 in both built-in themes (unchanged tokens).
|
||||||
|
4. Full gate: `uv run pytest` (unit + integration), `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright` clean, and the phase's E2E file in isolation.
|
||||||
|
5. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit, `--no-gpg-sign`, e.g.
|
||||||
|
`feat(ui): retire #sync-result; #sync-progress is the single sync surface + live region`
|
||||||
|
(includes `.agents/phases/todo/150_drop_sync_result/` moved to `.agents/phases/complete/`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation (the phase gate file).
|
||||||
|
- The six migrated image/PDF suites green in isolation, then the full suite green.
|
||||||
|
- Coverage: `app/` >90% (no `app/` change).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "sync-result\|syncResult" frontend/ tests/ app/` returns NOTHING anywhere in the project
|
||||||
|
- [ ] the six image/PDF suites still gate on a real sync-done signal (no sleeps introduced)
|
||||||
|
- [ ] one `aria-live` region in the sync area; contrast/focus-visible unchanged; no external CDN references
|
||||||
|
- [ ] `uv run pytest` green, `uv run ruff check . && uv run pyright` clean, coverage >90%
|
||||||
|
- [ ] the phase directory is in `.agents/phases/complete/` and committed with `--no-gpg-sign`
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Phase 151 — no more blank white image documents from PDFs (dedupe + soft masks + a blank gate)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||||
|
**Context:** `app/rag/pdf.py` — `extract_pdf_images()` (the module's image extractor: `page.get_image_info(xrefs=True)` per page via `_page_image_entries`, then `doc.extract_image(xref)` per entry, one `PdfImage(page, index, width, height, data, mime)` per **occurrence**) and its consumer `app/rag/importer.py` `_index_pdf_embedded_images()` (L1924 — the per-PDF embedded-image pass: `_pdf_image_context_inputs` once, then `extract_pdf_images(full_path)` and one `_index_one_embedded_image()` (L2030) per image: mime→ext gate through `_MIME_TO_EXT` (L1340 — png/jpg/gif/webp/bmp/tiff; anything else maps to `.bin` and is skipped), sha256-digest skip, one vision describe call (`_describe_or_skip`, L2907), the `image_dir` copy, chunk + embed, one transaction per image). The `images` toggle (`BOR_IMAGES`, `settings.images`, default OFF) gates the whole pass (LOCKED A3). `ImportSummary` (L390–475) carries the log-only counters (`images_failed`, `pdfs_screenshotted`, `pdfs_pictorial`, `pdf_pictorial_pages`, `image_context_backfilled`) — the phase-133 stats log-only convention: counters land in the `import: summary …` line, never in the sync `detail` API shape.
|
||||||
|
|
||||||
|
**Reproduced on the owner's file (`test_sources/podman-systemd.pdf`, 50 pages) — three distinct defects:**
|
||||||
|
1. **Duplicates:** `get_image_info` reports one entry per *placement*, so the file yields **46 occurrences of only 15 unique xrefs** (each image is placed 2–4 times). Every occurrence becomes its own document (`…#page=N#img=i.<ext>`), so the same image is vision-described and embedded up to 4×.
|
||||||
|
2. **Soft masks ignored:** every one of the 15 images carries an `/SMask` (`extract_image` reports `smask=<xref>`, `get_image_info` reports `has-mask=True`). `extract_image()` returns the **base** sample stream only — the mask (which carries the visible shape) is dropped, so the stored bytes are not what a PDF viewer shows.
|
||||||
|
3. **Decorative underlays indexed:** those images are table-shading underlays — `DeviceGray`, `mean=238/255`, **8 unique sample values**, **99.5% of samples within ±8 of the mean** (verified: composited with their mask they are still a near-white block; the table text a viewer shows is real PDF text drawn on top). They are indexed as documents, vision-described, and returned as image sources.
|
||||||
|
|
||||||
|
Measured comparison (the same probe, `fitz.Pixmap(...).samples`): the podman underlays `uniq=8`, near-uniform(±8) `0.95–0.995`; `test_sources/pdf_horse_test.pdf`'s photo `uniq=256`, `0.14`; `test_sources/players_handbook.pdf` page art `uniq≥188`, `0.10–0.59`. The gate separates them cleanly.
|
||||||
|
|
||||||
|
**Locked (owner decisions, chat):** blank/decorative images are **skipped entirely** (never indexed, never described — not stored-and-flagged); JPEG2000 (`jpx`) images stay **skipped** as they are today (the unsupported-extension gate keeps running on the RAW embedded format, so compositing to PNG must not newly admit the 586 `jpx` images in `players_handbook.pdf`).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A PDF's embedded-image pass yields one document per genuinely distinct, non-decorative image, with the bytes a viewer would show (soft mask applied) — no blank white documents, no triplicate images, and far fewer vision calls per PDF.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `150_drop_sync_result` (todo) — pipeline order only (that phase touches the Sources-page sync UI; this phase touches `app/rag/pdf.py` + the importer's image pass). No shared files.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Dedupe (task 01):** in `extract_pdf_images`, keep the FIRST occurrence of each distinct image and skip the repeats — key = the `xref` when non-zero, else the sha256 of the final `data` (a digest set also catches byte-identical images stored under different xrefs). The kept `PdfImage` carries the first occurrence's `page`/`index` (the synthetic path `…#page=N#img=i.<ext>` is therefore stable for the first placement). One INFO line per PDF when duplicates were dropped (`pdf: deduped N duplicate image occurrence(s) in <path>`), DEBUG per image.
|
||||||
|
- **Soft masks (task 02):** when the info/`extract_image` dict reports a non-zero `smask`, build the VISIBLE image instead of the raw base: `base = fitz.Pixmap(doc, xref)`, `mask = fitz.Pixmap(doc, smask_xref)`, `visible = fitz.Pixmap(base, mask)` (PyMuPDF's alpha-combining constructor) → `data = visible.tobytes("png")`, `mime = "image/png"`. Fail-soft: any exception (or a `Pixmap` with a colorspace PyMuPDF cannot combine) falls back to the raw `extract_image` bytes + the raw mime and logs a warning — the image is still indexed, exactly as today. **Ordering rule (owner-locked A4):** the raw-format gate runs FIRST — an image whose raw format is unsupported (`jpx` → `.bin`) is left exactly as today (raw bytes, raw mime, skipped downstream by the importer's ext gate); compositing happens only for images whose raw mime already maps through `_MIME_TO_EXT`. A `jpx` image with an smask therefore stays skipped.
|
||||||
|
- **Blank gate (task 03):** a NEW pure predicate in `app/rag/pdf.py`, `is_blank_image(samples: bytes, *, mean_min: int = 235, tolerance: int = 12, fraction: float = 0.98) -> bool` — pure (no PDF, no I/O — the `is_usable_pdf_text` / `pictorial_pages` house style, so it is unit-testable with hand-built sample buffers): blank when `len(samples)` is non-empty AND `mean(samples) >= mean_min` AND at least `fraction` of the samples lie within `±tolerance` of the mean. Applied in `extract_pdf_images` to the samples of the image that will actually be stored (the composited pixmap when a mask was applied, else the base pixmap): a blank image is **skipped** (not returned), with one INFO line per skipped image (`pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.0f uniform=%.3f`) and one summary line per PDF. Fail-soft: a pixmap that cannot be built skips the gate (the image is kept — never lose a real image to a decode problem).
|
||||||
|
- **The knob (task 04):** `Settings.pdf_image_skip_blank: bool = True` (`BOR_PDF_IMAGE_SKIP_BLANK`) — the kill switch, exactly the `pdf_pictorial` (L433) pattern: a `bool` with a `#:` docstring naming the env var, read by the importer and passed down, `False` restores the pre-phase behavior (every image kept, gate off). Dedupe and mask compositing are NOT knobed (they are correctness fixes). `.env.example` gains the documented entry next to the other PDF/image vars; `README.md` gains one line in the PDF/image settings section.
|
||||||
|
- **Counters (task 03):** two log-only `ImportSummary` fields — `pdf_images_deduped` and `pdf_images_blank_skipped` — appended to `ImportSummary.log()`'s line (the phase-133 stats log-only convention: the sync `detail` API shape is untouched). They count occurrences dropped per run across all PDFs.
|
||||||
|
- **ASSUMPTION A1 (owner-locked):** blank images are skipped entirely (no row, no copy, no vision call) rather than indexed-and-flagged.
|
||||||
|
- **ASSUMPTION A2 (owner-locked):** thresholds `mean ≥ 235`, `≥98%` of samples within `±12` — pinned by unit tests against the three real PDFs in `test_sources/` (podman underlays skipped; the horse photo and the handbook page art kept).
|
||||||
|
- **ASSUMPTION A3:** the dedupe changes the synthetic path for repeat occurrences, so image docs created pre-phase for a 2nd/3rd placement are pruned on the next sync and the first placement's doc is kept (idempotent from then on). No migration code.
|
||||||
|
- **ASSUMPTION A4 (owner-locked):** `jpx` (JPEG2000) images stay unsupported/skipped; compositing never newly admits them.
|
||||||
|
- **ASSUMPTION A5:** the gate lives in the parser (`extract_pdf_images`), not the importer, so every caller of the parser gets the fix and the parser stays the single place that decides what "an embedded image" is.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_dedupe_image_occurrences.md` — one `PdfImage` per distinct image (xref, else digest); duplicate-drop log + counter.
|
||||||
|
2. `02_composite_soft_masks.md` — `/SMask` composited to an alpha PNG (raw-format gate first, fail-soft to raw bytes).
|
||||||
|
3. `03_blank_image_gate.md` — the pure `is_blank_image()` predicate + the skip in the extractor + the two log-only counters.
|
||||||
|
4. `04_settings_knob.md` — `BOR_PDF_IMAGE_SKIP_BLANK` (default on) threaded parser → importer, `.env.example` + README.
|
||||||
|
5. `05_unit_tests.md` — PyMuPDF-built fixtures for dedupe/mask/blank + the real-PDF regression pins.
|
||||||
|
6. `06_e2e_pdf_image_extraction.md` — the dedicated Playwright suite (upload a PDF with a duplicate placement + a blank underlay + a real image → exactly one image doc, no blank docs).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_pdf.py` + a new `tests/unit/test_pdf_image_filter.py`): dedupe (N placements of one image → 1 result, first occurrence's page/index kept, counter), digest-level dedupe across xrefs, smask compositing (an RGBA image round-trips with alpha; the raw-format gate leaves `jpx`-style unsupported formats untouched; the fail-soft path returns raw bytes), `is_blank_image` as a pure function (uniform white → blank; noise → not blank; dark-uniform → NOT blank because the mean gate is about near-white underlays; empty samples → not blank), the knob off → every image kept.
|
||||||
|
- Real-file pins (skip-if-missing, the `HANDBOOK`/`HORSE_PDF` pattern already in `tests/unit/test_pdf.py` L1734–1737): `podman-systemd.pdf` → 15 unique images in, **0 image docs out** (all are decorative underlays); `pdf_horse_test.pdf` → its photo kept; `players_handbook.pdf` → its images stay skipped by the ext gate (unchanged, `jpx`).
|
||||||
|
- Integration: the importer's embedded-image pass indexes exactly the extractor's returned list (a blank/duplicate image never reaches `_index_one_embedded_image` — assert with a fake `Embedder` counting vision calls, the existing image-test idiom).
|
||||||
|
- Coverage: **>90%** on new/modified code (`app/rag/pdf.py`, the importer's pass, `app/config.py`).
|
||||||
|
- E2E: one dedicated Playwright file, run in isolation (`uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] syncing a PDF like `podman-systemd.pdf` creates **no** blank white image documents and **no** duplicate image documents (verified by the E2E and the real-file unit pins)
|
||||||
|
- [ ] an image with an `/SMask` is stored as the composited image (alpha PNG), not the blank base; an unsupported raw format (`jpx`) is still skipped
|
||||||
|
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` restores the pre-phase behavior (every extracted image indexed) — pinned by a unit test
|
||||||
|
- [ ] `import: summary … pdf_images_deduped=N pdf_images_blank_skipped=M …` appears in the sync log; the sync `detail` API shape is unchanged
|
||||||
|
- [ ] no behavior change in completed phases (the phase-137 document-context block, the phase-125 markdown enrichment, the digest skip, the `images` toggle semantics, the phase-133 fallback, the phase-140 pictorial pass)
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 01 — one `PdfImage` per distinct image (duplicate placements dropped)
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`extract_pdf_images()` returns one entry per DISTINCT image, so a PDF that places the same image 4× produces one document instead of four.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/pdf.py` — `extract_pdf_images()`: keep the per-page/occurrence walk (it is what gives `page`/`index`), but skip anything already seen:
|
||||||
|
- maintain `seen_xrefs: set[int]` and `seen_digests: set[str]` (sha256 hex of the final `data`) outside the page loop;
|
||||||
|
- after `extract_image` succeeds and `data` is non-empty: `if xref in seen_xrefs: dropped += 1; continue`, then `digest = hashlib.sha256(data).hexdigest()`; `if digest in seen_digests: dropped += 1; continue`; record both before appending (the digest set catches byte-identical images stored under different xrefs);
|
||||||
|
- the FIRST occurrence wins: the appended `PdfImage` keeps that occurrence's `page` (1-based) and `index` (0-based on the page) — the importer's synthetic path `…#page=N#img=i.<ext>` therefore stays stable for the first placement (ASSUMPTION A3: repeat-placement docs created pre-phase are pruned on the next sync, no migration code);
|
||||||
|
- when `dropped`: one INFO line after the page loop — `pdf: dropped %d duplicate image occurrence(s) from %s (%d unique)` — and a DEBUG line per drop (`pdf: duplicate image xref=%d page=%d index=%d of %s — skipped`);
|
||||||
|
- `hashlib` is already imported in this module? If not, add `import hashlib` (check first — `app/rag/pdf.py` currently imports `logging`, `re`, `collections.abc`, `dataclasses`, `pathlib`, `typing`, `pymupdf`).
|
||||||
|
2. Update the module docstring's `extract_pdf_images` bullet: "one :class:`PdfImage` per DISTINCT embedded image (page order, first placement wins) — repeated placements and byte-identical copies are deduplicated (phase 151 task 01)".
|
||||||
|
3. `app/rag/importer.py` — `_index_pdf_embedded_images()` needs no logic change (it indexes exactly what the extractor returns), but its docstring's "each embedded image … in page order then 0-based index" sentence gains the dedupe note. Add the two `ImportSummary` counters' plumbing HERE only if task 03 has not landed yet — otherwise leave the counters to task 03 (avoid a duplicate field).
|
||||||
|
4. ASSUMPTION: dedupe is per-PDF (the sets are local to one `extract_pdf_images` call), never across PDFs — two PDFs sharing a logo still each get their own image doc (the synthetic path is per-PDF, so cross-PDF dedupe would break the upsert key).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_pdf.py`, the PyMuPDF-built-fixture convention): a PDF with the SAME image inserted on 3 pages (or 3× on one page) → `len(extract_pdf_images(...)) == 1` and it carries the first occurrence's `page`/`index`; a PDF with two genuinely different images → 2; two different xrefs holding byte-identical bytes → 1 (digest path); the INFO line emitted (caplog) only when a duplicate was dropped.
|
||||||
|
- Real-file pin (skip-if-missing): `test_sources/podman-systemd.pdf` → the extractor returns **15** entries (46 placements pre-phase).
|
||||||
|
- Coverage: **>90%** on the new branches (both dedupe paths + the no-duplicate path).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `extract_pdf_images(Path("test_sources/podman-systemd.pdf"))` returns 15 entries (was 46)
|
||||||
|
- [ ] the first occurrence's page/index are preserved; the digest path is covered by a unit test
|
||||||
|
- [ ] the duplicate-drop INFO line appears once per PDF with duplicates
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 02 — apply the `/SMask`: store the image a viewer would show
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An image stored in the PDF with a soft mask is extracted as the composited (masked) image, not as the blank base sample stream `extract_image()` returns today.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/pdf.py` — `extract_pdf_images()`, inside the per-image block, AFTER the raw-format decision and BEFORE appending:
|
||||||
|
- read the mask reference: `smask = int(extracted.get("smask") or 0)` (the `extract_image` dict reports it; `get_image_info`'s `has-mask` is the cross-check, not the source of truth);
|
||||||
|
- **raw-format gate FIRST (owner-locked A4):** compute `mime` from the raw format exactly as today (`_mime_from_image_info({**info, "ext": extracted.get("ext") or ""})`). If that mime is `application/octet-stream` (i.e. a format the importer's `_MIME_TO_EXT` gate will reject — `jpx`/JPEG2000, the 586 images in `players_handbook.pdf`), DO NOT composite: keep the raw bytes + raw mime, so the behavior is byte-identical to pre-phase for unsupported formats (compositing must never newly admit them);
|
||||||
|
- otherwise, when `smask` is non-zero: build the visible image
|
||||||
|
```python
|
||||||
|
base = fitz.Pixmap(doc, xref)
|
||||||
|
mask = fitz.Pixmap(doc, smask)
|
||||||
|
visible = fitz.Pixmap(base, mask) # base + alpha-from-mask
|
||||||
|
data = visible.tobytes("png")
|
||||||
|
mime = "image/png"
|
||||||
|
```
|
||||||
|
wrapped in `try/except Exception` → on failure log a warning (`pdf: could not composite smask=%d for image page=%d index=%d of %s — %s; using raw bytes`) and fall back to the raw `extracted["image"]` bytes + the raw mime (fail-soft: the image is still indexed, exactly as pre-phase);
|
||||||
|
- keep `width`/`height` from `extracted` (unchanged — the composited pixmap has the same dimensions);
|
||||||
|
- a DEBUG line when a mask was applied (`pdf: composited smask=%d for image page=%d index=%d of %s`).
|
||||||
|
2. `app/rag/pdf.py` module docstring: extend the `extract_pdf_images` bullet — masked images are composited to an alpha PNG (phase 151 task 02), unsupported raw formats are left untouched so the importer's extension gate still decides.
|
||||||
|
3. `app/rag/importer.py`: no code change — `_index_one_embedded_image()` already derives the copy extension from `img.mime` via `_MIME_TO_EXT`, and `image/png` → `.png` is in `settings.image_extension_set` by default. Verify the digest-skip consequence in a comment: a masked image's bytes change, so pre-phase rows for masked images re-describe ONCE on the first post-phase sync (the sha256 digest differs) — expected, self-limiting, no migration.
|
||||||
|
4. ASSUMPTION: `fitz.Pixmap(base, mask)` is the correct combinator (verified against `podman-systemd.pdf` xrefs 202/744/1633 — it produces an alpha PNG of the same dimensions); a CMYK/Indexed base that PyMuPDF refuses to combine takes the fail-soft path.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_pdf.py`): build a PDF with an RGBA PNG inserted (`page.insert_image(rect, stream=png_bytes)` — PyMuPDF writes the `/SMask`) and assert (a) `extract_image`'s raw bytes ≠ the extractor's `data`, (b) the extractor's `mime == "image/png"`, (c) re-opening the extractor's bytes as a `fitz.Pixmap` reports `alpha == 1` (the mask survived), (d) a PDF whose image has no mask is returned byte-identical to pre-phase (no regression for the common case); the fail-soft path via a monkeypatched `fitz.Pixmap` that raises → raw bytes returned + the warning logged (caplog).
|
||||||
|
- Real-file pin (skip-if-missing): every image extracted from `test_sources/podman-systemd.pdf` carries an smask pre-phase; post-phase the extractor's output for it is either composited PNG or (blank-gated away by task 03) absent — assert the composited bytes differ from the raw base for at least one xref when the gate is off.
|
||||||
|
- Coverage: **>90%** on the new branches (mask applied / mask failed / unsupported format untouched / no mask).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a masked image extracted through `extract_pdf_images` yields alpha-PNG bytes (`mime="image/png"`), not the blank base stream
|
||||||
|
- [ ] an unsupported raw format (`jpx`) is untouched — raw bytes + raw mime, still skipped downstream
|
||||||
|
- [ ] a compositing failure falls back to the raw bytes with a warning (no image lost, no exception raised)
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Task 03 — the blank/decorative gate: never index a near-white underlay
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A deterministic, pure predicate decides "this image carries no visible content" and the extractor drops those images — no row, no `image_dir` copy, no vision call.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/pdf.py` — add the pure predicate (the `is_usable_pdf_text` / `pictorial_pages` house style: no PDF, no I/O, no LLM):
|
||||||
|
```python
|
||||||
|
BLANK_IMAGE_MEAN = 235 #: near-white floor
|
||||||
|
BLANK_IMAGE_TOLERANCE = 12 #: "same as the mean" band
|
||||||
|
BLANK_IMAGE_FRACTION = 0.98 #: fraction of samples that must sit in the band
|
||||||
|
|
||||||
|
def is_blank_image(samples: bytes, *, mean_min: int = BLANK_IMAGE_MEAN,
|
||||||
|
tolerance: int = BLANK_IMAGE_TOLERANCE,
|
||||||
|
fraction: float = BLANK_IMAGE_FRACTION) -> bool:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
Blank when `samples` is non-empty AND `sum(samples)/len(samples) >= mean_min` AND `sum(1 for s in samples if abs(s - mean) <= tolerance) / len(samples) >= fraction`. Empty samples → `False` (never blank a thing we cannot measure). Document the measured basis in the docstring (podman underlays: mean 238, uniq 8, uniform(±8) 0.95–0.995 → blank; the horse photo: mean 123, uniq 256, uniform 0.14 → kept; handbook page art: mean 24–221, uniq ≥188, uniform 0.10–0.59 → kept). Note explicitly that a DARK uniform block is NOT blank by this rule (the gate targets near-white page underlays — a solid dark block is unusual and cheaper to keep than to explain away).
|
||||||
|
2. `app/rag/pdf.py` — `extract_pdf_images()`: after the dedupe + mask steps, decide blank-ness on the samples of the image that would actually be stored:
|
||||||
|
- build `gate_pixmap` = the composited pixmap when a mask was applied, else `fitz.Pixmap(doc, xref)`; when `gate_pixmap.alpha`, drop the alpha first (`fitz.Pixmap(gate_pixmap, 0)`) so the mean is over COLOR samples only (an alpha channel would drag the mean and produce false blanks);
|
||||||
|
- `if settings-gate and is_blank_image(gate_pixmap.samples): blank_skipped += 1; continue` (the gate is skipped entirely when the caller passes `skip_blank=False` — task 04's knob);
|
||||||
|
- one INFO line per skipped image: `pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.1f uniform=%.3f`, and one INFO per PDF when `blank_skipped` (`pdf: skipped %d blank/decorative image(s) in %s`);
|
||||||
|
- fail-soft: if the pixmap cannot be built for the gate, DO NOT skip the image (keep it, DEBUG why) — a decode problem must never delete a real image;
|
||||||
|
- the extractor signature gains `skip_blank: bool = True` (keyword, default on) so callers/tests can turn it off without settings; the dedupe and mask steps are NOT knobed.
|
||||||
|
3. `app/rag/importer.py` — `ImportSummary` gains two log-only counters (the phase-133 stats log-only convention: the sync `detail` API shape is untouched), each with a `#:` docstring citing this phase: `pdf_images_deduped: int = 0` and `pdf_images_blank_skipped: int = 0`, appended to `ImportSummary.log()`'s format string (`… pdf_images_deduped=%d pdf_images_blank_skipped=%d formats=%s`). The extractor returns the counts alongside the images so the importer can record them — the minimal clean shape is a small result object or a 2-tuple; prefer keeping `extract_pdf_images` returning `list[PdfImage]` for its other callers and add the counters via an optional out-param-free route: give the function a sibling `extract_pdf_images_with_stats(pdf_path, *, skip_blank=True) -> tuple[list[PdfImage], PdfImageStats]` (a `@dataclass(frozen=True) PdfImageStats` with `duplicates`, `blank_skipped`, `total_found`) and make `extract_pdf_images` a thin wrapper returning `stats[0]`. `_index_pdf_embedded_images()` calls the `_with_stats` variant and adds to `summary.pdf_images_deduped` / `summary.pdf_images_blank_skipped`.
|
||||||
|
4. ASSUMPTION: the counters are log-only (no API change) — the owner reads them in the sync log; the UI's sync result line is untouched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_pdf_image_filter.py`, new file): `is_blank_image` as a pure function — all-250 bytes → blank; random noise → not blank; all-0 (dark) → NOT blank; empty → not blank; a 97%-uniform-with-a-few-dark-pixels buffer → not blank (just under the fraction); the boundary values pinned exactly at `mean_min` and `fraction`; the extractor-level behavior with PyMuPDF-built PDFs — a near-white 200×200 image inserted → 0 results + the INFO line + `stats.blank_skipped == 1`; a colorful noise image → 1 result; `skip_blank=False` → both returned.
|
||||||
|
- Integration: the importer's pass with a fake `Embedder` (the existing image-test idiom) — a PDF with one blank + one real image produces exactly ONE image document and ONE vision call; `import: summary … pdf_images_blank_skipped=1` in the log (caplog).
|
||||||
|
- Real-file pin (skip-if-missing): `test_sources/podman-systemd.pdf` → `extract_pdf_images` returns `[]` (all 15 unique images are decorative underlays) with `stats.duplicates == 31` and `stats.blank_skipped == 15`; `test_sources/pdf_horse_test.pdf` → its photo is returned.
|
||||||
|
- Coverage: **>90%** on the new code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `extract_pdf_images(Path("test_sources/podman-systemd.pdf"))` returns no images (pre-phase: 46 blank-white docs)
|
||||||
|
- [ ] `extract_pdf_images(..., skip_blank=False)` still returns them (the kill switch at the function level)
|
||||||
|
- [ ] `import: summary … pdf_images_deduped=… pdf_images_blank_skipped=…` appears in the sync log; the `/api/sync/status` `detail` shape is unchanged
|
||||||
|
- [ ] a dark-uniform image and a real photo are both kept (unit-pinned)
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf_image_filter.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 04 — `BOR_PDF_IMAGE_SKIP_BLANK`: the kill switch, wired end to end
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The blank gate is a documented setting, default ON, with a supported way to turn it off — the `pdf_pictorial` pattern.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add to `Settings`, next to `pdf_pictorial` / `pdf_pictorial_min_chars` (L433–443):
|
||||||
|
```python
|
||||||
|
#: Skip blank/decorative PDF-embedded images (phase 151, task 04,
|
||||||
|
#: ``BOR_PDF_IMAGE_SKIP_BLANK``; ``0``/``false`` = off) …
|
||||||
|
pdf_image_skip_blank: bool = True
|
||||||
|
```
|
||||||
|
The docstring names the env var, states the default, and says what turning it off means (every extracted image is indexed again — the pre-phase behavior; dedupe and soft-mask compositing are unaffected, they are correctness fixes). No validator needed (a plain bool like `pdf_pictorial` / `images`).
|
||||||
|
2. `app/rag/importer.py` — `_index_pdf_embedded_images()`: pass the setting into the extractor call (`extract_pdf_images_with_stats(full_path, skip_blank=settings.pdf_image_skip_blank)`), so the knob is read in exactly one place. Update the pass's docstring bullet list with the gate + the knob.
|
||||||
|
3. `.env.example` — add the entry in the PDF/image block (next to `BOR_PDF_PICTORIAL` / `BOR_IMAGES`), commented in the file's existing voice: what it does, the default, and when to set it to `false` (a KB of scanned/decorative pages where a near-white image still matters).
|
||||||
|
4. `README.md` — one line in the PDF/image settings list (the settings table/section the phase-126/140 entries live in): `BOR_PDF_IMAGE_SKIP_BLANK` (default `true`) — skip blank/decorative images embedded in PDFs.
|
||||||
|
5. ASSUMPTION: no admin-UI toggle (the phase-91 Theme Tab precedent shows knobs live in `.env`; the owner asked for the bug fixed, not a settings page).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `Settings` default is `True`; `BOR_PDF_IMAGE_SKIP_BLANK=false` (env override, the existing config-test idiom) yields `False`; an integration test with the fake `Embedder` proves the off path indexes a blank image again (the inverse of task 03's assertion) — the kill switch is proven, not assumed.
|
||||||
|
- Coverage: **>90%** on the touched lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` restores indexing of blank PDF-embedded images (test-proven)
|
||||||
|
- [ ] the setting is documented in `.env.example` and `README.md`
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 05 — unit + integration tests for dedupe, masks, and the blank gate
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every new branch of the extractor is pinned without a browser: PyMuPDF-built fixtures for the synthetic cases, the three real PDFs in `test_sources/` for the regression proof.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_pdf_image_filter.py` (new file; the `tests/unit/test_pdf.py` conventions — PyMuPDF-built temp PDFs, no binary fixtures committed, `tmp_path`):
|
||||||
|
- **dedupe:** one image inserted on 3 pages → 1 result with the FIRST page/index; two distinct images → 2; byte-identical bytes under two xrefs → 1 (digest path); `stats.duplicates` counts; the INFO line via `caplog`;
|
||||||
|
- **soft mask:** an RGBA PNG inserted (`page.insert_image(rect, stream=…)`) → the extractor's `mime == "image/png"`, its bytes ≠ `doc.extract_image(xref)["image"]`, and `fitz.Pixmap(stream=extracted_bytes).alpha == 1`; an unmasked image → byte-identical to pre-phase; a monkeypatched `fitz.Pixmap` that raises → raw bytes returned + the warning (fail-soft);
|
||||||
|
- **blank gate:** `is_blank_image` as a pure function (uniform white → True; noise → False; uniform dark → False; empty → False; the exact `mean_min`/`fraction` boundaries); extractor-level — a near-white 200×200 image → 0 results + `stats.blank_skipped == 1` + the INFO line; a colorful noise image → 1 result; `skip_blank=False` → both returned;
|
||||||
|
- **unsupported format untouched:** an image whose raw ext is unsupported (build one by monkeypatching `extract_image` to report `ext="jpx"` + a non-zero `smask`) → the extractor returns raw bytes + the unmapped mime (no compositing) — the owner-locked A4 pin.
|
||||||
|
2. `tests/unit/test_pdf.py` — extend the existing extractor tests (they already cover `extract_pdf_images` and the fail-soft contract): the dedupe/mask steps must not break the "unreadable file → []", "no xref → skipped", "empty data → skipped" pins; add the real-file pins with the file's existing skip-if-missing idiom (see `HANDBOOK` / `HORSE_PDF`, L1734–1737):
|
||||||
|
- `test_sources/podman-systemd.pdf` → `extract_pdf_images(...) == []`, `stats.duplicates == 31`, `stats.blank_skipped == 15` (the owner's exact file — the bug's tombstone test);
|
||||||
|
- `test_sources/pdf_horse_test.pdf` → exactly 1 image, not blank;
|
||||||
|
- `test_sources/players_handbook.pdf` → images are returned by the parser but their raw mime is unmapped (`jpx`) → the importer's ext gate still skips them (assert the mime mapping, not the importer).
|
||||||
|
Each guarded by `pytest.mark.skipif(not path.exists(), …)` — `test_sources/*.pdf` are untracked fixtures.
|
||||||
|
3. Integration (`tests/unit/test_image_documents.py` or a new `tests/integration/test_pdf_image_pass.py`, whichever matches the existing image-pass tests' home): with `tests/fakes.py::FakeEmbedder` (the `_VisionMockLLM` idiom — a vision-capable fake that counts `describe_image` calls), a PDF containing one blank underlay + one real image + a duplicate placement indexes EXACTLY ONE image document and makes exactly ONE vision describe call; `summary.pdf_images_deduped` / `summary.pdf_images_blank_skipped` carry the counts; the `import: summary …` line includes them (caplog).
|
||||||
|
4. Run the whole suite after each addition: `uv run pytest -q`, then `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/rag/pdf.py`, `app/rag/importer.py`, `app/config.py`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: **>90%** on the new/modified code paths (dedupe ×2, mask applied/failed/unsupported, blank skip/keep/knob-off, the stats plumbing).
|
||||||
|
- No network, no real LLM (the fakes only); the real-PDF pins are pure CPU (PyMuPDF) and must stay under a few seconds each.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf_image_filter.py tests/unit/test_pdf.py -v` green
|
||||||
|
- [ ] the podman-systemd pin asserts 0 images out of 46 placements (the reported bug, pinned)
|
||||||
|
- [ ] the `skip_blank=False` and unsupported-format (A4) paths are both covered
|
||||||
|
- [ ] `uv run pytest -q` fully green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 06 — the dedicated E2E: a PDF sync produces no blank, no duplicate image docs
|
||||||
|
|
||||||
|
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Through the real product path (upload a PDF → press Sync → inspect the KB), a PDF containing a decorative underlay, a duplicate placement, and one real image yields exactly one image document — the owner's complaint, proven fixed in the browser.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_pdf_image_extraction.py` (new; the `tests/e2e/test_pdf_support.py` / `test_pdf_image_context.py` per-suite-app pattern — a module-scoped app with `BOR_IMAGES=true` + the `pdf` import token, the shared E2E Postgres, the autouse truncate, the mock LLM only):
|
||||||
|
- **fixture PDF built in the suite with PyMuPDF** (the house convention — no binary committed): 2 text pages, page 1 carrying (a) a near-white 200×200 "decorative underlay" image, (b) the SAME real image inserted twice (the duplicate placement), (c) one genuinely distinct colorful image; page 2 carrying one more placement of the real image (a cross-page duplicate — the podman case);
|
||||||
|
- **seed** it through the real admin upload flow + the RAG page's **Sync** button (the same user path the owner used), then wait for the sync to settle on the sync result surface (post-phase-150: `#sync-progress-text`);
|
||||||
|
- **assertions:**
|
||||||
|
* the catalog contains exactly ONE image document for the PDF (the distinct real image, first placement) — count the `is_image` docs whose path starts with the PDF's synthetic path;
|
||||||
|
* NO document's stored image bytes are a blank white image (fetch the served copy / read the row's `image_path` and assert the file's decoded pixels are not near-uniform — the same predicate the app uses, imported from `app.rag.pdf.is_blank_image`, so the test cannot drift from the product);
|
||||||
|
* the synthetic path of the kept doc is the FIRST placement's (`…#page=1#img=<i>.png`) — the dedupe's stability contract;
|
||||||
|
* the mock's vision call count for image description is 1 (the duplicate and the blank never reached the model — the cost the owner paid in "several minutes");
|
||||||
|
* a follow-up re-sync (nothing changed) reports `unchanged` and creates nothing new (idempotence after the fix).
|
||||||
|
- **negative control:** a second module app (or a second seed source) with `BOR_PDF_IMAGE_SKIP_BLANK=false` — the same PDF now yields the blank doc too (the knob is proven live in the product, not just in unit tests). If a second app is too heavy for the file's structure, drive the knob through the same env-fallback mechanism the other suites use for per-suite settings.
|
||||||
|
2. Run it in isolation until green: `uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||||
|
3. Regression run: the pre-existing PDF/image suites must stay green unedited (they pin the behaviors this phase must not disturb): `tests/e2e/test_pdf_support.py`, `test_pdf_image_context.py`, `test_pictorial_pdf.py`, `test_image_documents.py`, `test_image_anchor_context.py`, `test_answer_images.py`.
|
||||||
|
4. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit with `--no-gpg-sign`, e.g.
|
||||||
|
`fix(rag): dedupe PDF images, apply soft masks, skip blank/decorative images`
|
||||||
|
(moves `.agents/phases/todo/151_pdf_image_extraction/` to `complete/`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The dedicated E2E green IN ISOLATION (the phase gate) and the six pre-existing PDF/image E2Es green.
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- Mock-only suite (no real vision calls), deterministic (no sleeps as barriers — wait on the sync result surface).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` green in isolation
|
||||||
|
- [ ] exactly one image document per distinct real image; zero blank-white documents; one vision call
|
||||||
|
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` demonstrably restores the old behavior in the product
|
||||||
|
- [ ] the six pre-existing PDF/image E2Es green unedited; full suite green; coverage >90%
|
||||||
|
- [ ] the phase directory moved to `.agents/phases/complete/` and committed with `--no-gpg-sign`
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Phase 152 — the sync label keeps moving while images are described and embedded
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "The sync status on the RAG page gets \"stuck\" when it's embedding images. For example, when syncing test_sources/podman-systemd.pdf the status was stuck on `Syncing… podman-systemd.unit — Podman documentation/podman-systemd.unit — Podman documentation.pdf (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||||
|
**Context:** the sync status machine `app/api/sync.py` — `SyncStatus` (L276–300: `state`, `current_file`/`files_done`/`files_total` (phase 64), `phase`/`current_summary`/`summaries_done`/`summaries_total` (phase 98), `current_chunk`/`chunks_total` (phase 129)), the `GET /api/sync/status` payload (L330–360), and `_run_sync`'s paired progress closures `_hook` (L553 — names the file) and `_chunk_hook` (L563 — the in-flight file's embed-batch position, fed by `LLMClient.embed`'s `on_progress`). The importer threads the chunk hook as `import_sources(on_chunk_progress=…)` → `_index_file(on_chunk=…)` → `_index_pdf_file(on_chunk=…)` → `_replace_content_chunks(on_chunk=…)`. The importer's own docstring states the gap this phase closes (L684–690): *"Only the content-chunk embeds report … the PDF's embedded-image docs (separate documents, not the walked file) do not report either."*
|
||||||
|
|
||||||
|
**Reproduced:** the owner's `podman-systemd.pdf` is the LAST file of a 2,540-file run. Its text chunks finish (the label's last write: `· chunk 125/126`), then the PDF's vision spans run — `_index_pdf_embedded_images` (46 image occurrences pre-phase-151, each a `describe_image` chat call + an embed + a summary), `_pictorial_pdf_pass` (L1715 — one render + one describe call per sparse page), `_fallback_page_screenshots` (L1635 — one describe call per page of a garbage-text PDF). **None of them reports progress**, so the label sits frozen on the last chunk position for minutes while the run is very much alive. The phase-149 box made the freeze plainly visible (it is now the big surface the user reads).
|
||||||
|
|
||||||
|
**Locked (owner decision, chat):** the fix is a real progress signal for the vision spans (not a spinner/heartbeat that hides the stall), reported as a per-image position alongside the existing chunk position.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
During a PDF's image spans the status reports `image n/m`, the UI renders it, and the label advances once per vision call — a long sync is visibly progressing, and a genuinely stalled run is distinguishable from a busy one.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `151_pdf_image_extraction` (todo) — its dedupe + blank gate shrink the span this phase reports on, and its `extract_pdf_images_with_stats` is the call that establishes the span's `total`. Report order matters: the totals this phase reports are post-dedupe/post-gate counts.
|
||||||
|
- `150_drop_sync_result` (todo) — the label surface is `#sync-progress` (the phase-149 box is the only place the running label renders; the button is the spinner).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The hook (task 02):** `import_sources(..., on_image_progress: Callable[[int, int], None] | None = None)`, threaded exactly like `on_chunk_progress`: `import_sources` → `_index_file(on_image=…)` → `_index_pdf_file(on_image=…)` → the three vision spans (`_index_pdf_embedded_images`, `_pictorial_pdf_pass`, `_fallback_page_screenshots`). Contract (the phase-129 pairing rule, extended): the hook is called **immediately before each vision describe call** with `(position_1based, total)` where `total` is that span's count — the extracted-image count for the embedded pass (post-dedupe, post-blank-gate: the work actually about to happen), the candidate-page count for the pictorial pass, the rendered-page count for the screenshot fallback. The three spans of ONE file are reported as ONE continuous counter: the pass that runs second starts where the first ended (`base = images_done_so_far`), so the label climbs `1/N … N/N` across the file's whole vision span instead of restarting. `None` (every existing caller — the CLI, the chat path) is a zero-cost guarded no-op and the run is byte-identical (the phase-129 rule).
|
||||||
|
- **The status fields (task 01):** `SyncStatus` gains `images_done: int = 0` / `images_total: int = 0`, reported by `GET /api/sync/status` as `images_done` / `images_total`. Clear/reset rules mirror the chunk position (a sub-position of a file): cleared at the `phase = "overview"` / `phase = "summaries"` transitions and in every terminal (success/failed/cancelled), and **reset to 0/0 by the per-file `_hook`** — the pairing rule is "a `progress` call names the file and clears the image position; every image position until the next `progress` call belongs to it". (The chunk position does not need this because every file embeds chunks; a file with no images must not inherit the previous PDF's position.)
|
||||||
|
- **The label (task 03):** `sources.js` `fmtSyncPhaseLabel` appends `· image n/m` when `status.images_total > 0` (the phase-129 chunk-part language: a position reads as data, mono in the box). `setSyncLabel` gains a fifth parameter `imageText = ""` appended to `full` (after the chunk part) — it rides the title, the box, and the running `aria-label`. **No new button span** is added: while running the label parts are hidden (phase 149 makes the button the spinner), so the parts DOM stays as-is and the position lives only in the composed `full` string. The chunk part keeps its own slot (the two positions are distinct and both may show: `… (2540/2540) · chunk 125/126 · image 3/15`).
|
||||||
|
- **ASSUMPTION A1 (owner-locked):** a real per-image position, not an elapsed-time heartbeat.
|
||||||
|
- **ASSUMPTION A2:** counts only, not the image identity — the file label already names the PDF and the synthetic `…#page=N#img=i.png` path is noise in a label.
|
||||||
|
- **ASSUMPTION A3:** the three PDF vision spans share ONE counter per file (continuous, not three restarts) — the user cares that the file is progressing, not which span it is in.
|
||||||
|
- **ASSUMPTION A4:** standalone image files (one image per file) get no sub-position — `1/1` per file would be noise; the file count already covers them.
|
||||||
|
- **ASSUMPTION A5:** no new `phase` value — the image spans are inside `phase: "import"` (the phase machine stays as phase 98 defined it).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_status_image_fields.md` — `SyncStatus.images_done/images_total`, the reset/clear rules, the `/api/sync/status` payload + docstring.
|
||||||
|
2. `02_importer_image_hook.md` — `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter.
|
||||||
|
3. `03_label_and_wiring.md` — the `_image_hook` closure in `_run_sync`, the `sources.js` label part + title/aria, the phase-129-style comment updates.
|
||||||
|
4. `04_unit_tests.md` — the status machine pins (idle 0/0, mid-run position, the per-file reset, every terminal clears) + the importer hook pairing.
|
||||||
|
5. `05_e2e_image_progress.md` — the dedicated Playwright suite: the label advances through a PDF's image span (slow-LLM proxy), never frozen.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_sync_button.py`'s full-status-dict pin gains the two keys (idle `0/0`); the terminal-clears and per-file-reset rules pinned through the runner's own closures (the phase-64/129 idiom); the importer hook pinned with `tests/fakes.py::FakeEmbedder` — a PDF with N images calls the hook N times with `(1,N)…(N,N)`, the three spans continue one counter, `None` costs nothing (the hook-less run is byte-identical).
|
||||||
|
- Integration: a sync run over a fixture PDF reports a monotonically increasing `images_done` across polls (no backwards jumps, never above `images_total`).
|
||||||
|
- Coverage: **>90%** on new/modified code (`app/api/sync.py`, `app/rag/importer.py`, the JS is pinned by the source-level suites + E2E).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov` in isolation (the `tests/e2e/slow_llm.py` delay proxy — every describe call outlives the 2 s poll, so each position is observed deterministically).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] during a PDF's image span the status reports `images_done`/`images_total` climbing 1→N and the UI shows `· image n/m` in the box (and the button `title`/`aria-label`)
|
||||||
|
- [ ] the label is never frozen for more than one vision call's duration on the fixture PDF (the E2E asserts at least 3 distinct positions observed)
|
||||||
|
- [ ] the image position clears at the overview/summaries transitions and in every terminal; a file with no images never shows a stale position
|
||||||
|
- [ ] `GET /api/sync/status` is additive only — every pre-existing key unchanged (the full-dict pin)
|
||||||
|
- [ ] a hook-less caller (the CLI import path) is byte-identical to pre-phase
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 01 — `SyncStatus` gains `images_done` / `images_total`
|
||||||
|
|
||||||
|
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The sync status carries the in-flight file's vision-image position, with the same clear/reset discipline the chunk position already has.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py` — `SyncStatus` (L276–300): add
|
||||||
|
```python
|
||||||
|
# Phase 152 (task 01): the in-flight file's VISION-image position —
|
||||||
|
# the 1-based index of the image about to be described and the
|
||||||
|
# file's total images for this run (the importer's
|
||||||
|
# on_image_progress hook; the three PDF vision spans share ONE
|
||||||
|
# continuous counter per file). 0/0 before the first image, during
|
||||||
|
# the prelude/overview/summaries phases, and in terminal states.
|
||||||
|
images_done: int = 0
|
||||||
|
images_total: int = 0
|
||||||
|
```
|
||||||
|
and extend the dataclass docstring with a phase-152 paragraph in the file's established voice (it is a SUB-position of a file, like the chunk position: the keep-final-counts convention covers FILES only, so this clears with the file).
|
||||||
|
2. Clear/reset rules — exactly the points `current_chunk`/`chunks_total` clear today (grep `_status.current_chunk = 0` — the `phase = "overview"` transition ~L597, the `phase = "summaries"` transitions ~L650/671, the terminal paths ~L713/717/752/755/770/774, and the run start ~L462): set the two new fields to `0` at the SAME sites. In addition, the per-file `_hook` closure (L553) resets them to `0/0` — the pairing rule is "a `progress` call names the file and clears the image position; every image position until the next `progress` call belongs to it" (unlike the chunk position, a file may have NO images at all, and it must not inherit the previous PDF's position).
|
||||||
|
3. `GET /api/sync/status` (the payload dict, L330–360): add `"images_done"` / `"images_total"` beside `current_chunk`/`chunks_total`, with the same style of inline comment; extend the endpoint docstring with the phase-152 sentence (additive only — every pre-existing key keeps its name, type, and meaning).
|
||||||
|
4. Module docstring: add the phase-152 paragraph after the phase-129 chunk-position paragraph (the file documents each phase's contribution in order).
|
||||||
|
5. ASSUMPTION A5: no new `phase` value — the image spans live inside `phase: "import"`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the full idle-dict pin in `tests/unit/test_sync_button.py` gains the two keys (`0`/`0`); a terminal-clears test and a per-file-reset test (drive the runner's closures directly, the phase-64/129 idiom).
|
||||||
|
- Coverage: **>90%** on the new lines.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/sync/status` returns `images_done` / `images_total` (idle `0`/`0`) and every pre-existing key is unchanged
|
||||||
|
- [ ] the fields clear at the overview/summaries transitions and in success/failed/cancelled, and reset per file
|
||||||
|
- [ ] `uv run pytest tests/unit/test_sync_button.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — the importer's `on_image_progress` hook across the PDF vision spans
|
||||||
|
|
||||||
|
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every vision describe call in a file's import reports `(position, total)` before it happens, so the caller can show the span moving.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — thread the callback exactly like `on_chunk_progress` (the phase-129 route):
|
||||||
|
- `import_sources(..., on_image_progress: Callable[[int, int], None] | None = None)` (L636–650) — document it in the docstring next to the `on_chunk_progress` paragraph, and CORRECT that paragraph's closing sentence ("the PDF's embedded-image docs … do not report either") — it no longer holds;
|
||||||
|
- `_index_file(..., on_image: Callable[[int, int], None] | None = None)` (L1050) → `_index_pdf_file(..., on_image=…)` (L1372);
|
||||||
|
- the three spans, each called with the file's continuous counter (ASSUMPTION A3 — one counter per file, spans continue each other):
|
||||||
|
* `_index_pdf_embedded_images(..., on_image=…)` (L1924): `total = len(images)` from `extract_pdf_images_with_stats` (phase 151 — post-dedupe, post-blank-gate: the work actually about to happen); call `on_image(base + i + 1, base + total)` **immediately before** each `_describe_or_skip` (so the label moves at the START of each slow call, not after it), where `base` is the count already reported for this file;
|
||||||
|
* `_pictorial_pdf_pass(..., on_image=…)` (L1715): `total = len(candidate_pages)` (the deterministic sparse-page gate's list, already computed before the loop); same call site rule;
|
||||||
|
* `_fallback_page_screenshots(..., on_image=…)` (L1635): `total = len(rendered pages)`; same rule;
|
||||||
|
- the counter handoff: `_index_pdf_file` keeps a small local `reported = 0` and passes it down / adds each span's returned count (each span returns the number of images it reported — reuse its existing return shape or return the count; the executor picks the minimal clean shape). A span that is skipped (toggle off, no candidates, usable text) reports nothing and does not advance the counter;
|
||||||
|
- `None` stays a zero-cost guarded no-op: every call site is `if on_image is not None:` (the phase-129 rule — a hook-less run is byte-identical, so the CLI and chat paths are untouched).
|
||||||
|
2. Update the module docstring's progress paragraph (L287–291) to name the image hook, and the `_index_pdf_embedded_images` docstring's bullet list.
|
||||||
|
3. ASSUMPTION A4: standalone image files (`_index_image_file`, L2989) get NO sub-position — one image per file, `1/1` would be noise; the file count already covers them.
|
||||||
|
4. ASSUMPTION A2: counts only — the hook carries no image identity (the file label already names the PDF).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_importer.py` / the image-pass tests' `tests/fakes.py::FakeEmbedder` idiom): a fixture PDF with N embedded images → the hook fires N times with `(1,N)…(N,N)`; a PDF with a pictorial span + an embedded span → ONE continuous counter (no restart, never above the combined total); a text-only PDF → the hook never fires; `on_image_progress=None` → the import result is byte-identical to pre-phase (the regression pin).
|
||||||
|
- The hook never raises into the pipeline: a hook that raises is a caller bug (the phase-64 rule — "the hooks in this repo only assign fields", no try/except), so do NOT add defensive handling; keep the closure assignment-only.
|
||||||
|
- Coverage: **>90%** on the new branches (each span, the skip paths, the `None` path).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the hook fires once per vision describe call, 1-based, with the span's total, before the call
|
||||||
|
- [ ] the three PDF spans share one continuous per-file counter
|
||||||
|
- [ ] a hook-less import is byte-identical to pre-phase
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 03 — wire the hook to the status and render `· image n/m` in the label
|
||||||
|
|
||||||
|
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The importer's image hook lands on the status object, and the Sources page renders the position — the label advances once per vision call.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py` — `_run_sync`: add the third paired closure beside `_hook` (L553) and `_chunk_hook` (L563):
|
||||||
|
```python
|
||||||
|
def _image_hook(done: int, total: int) -> None:
|
||||||
|
_status.images_done = done
|
||||||
|
_status.images_total = total
|
||||||
|
```
|
||||||
|
pass it into the import call (L580): `import_sources(..., on_image_progress=_image_hook)`; comment it in the file's established voice (the three hooks are paired: `progress` names the file and resets the image position, `on_chunk_progress` reports the embed-batch position, `on_image_progress` reports the vision-image position).
|
||||||
|
2. `frontend/assets/sources.js`:
|
||||||
|
- `fmtSyncPhaseLabel(status)` (L896–919): the import branch gains `image` — `status.images_total > 0 ? \`· image ${status.images_done}/${status.images_total}\` : ""` — and the `overview` / `summaries` branches return `image: ""` (the position belongs to the import phase only);
|
||||||
|
- `enterSyncRunningState` (L964): destructure `{ main, count, chunk = "", image = "" }` and pass the new argument to the writer;
|
||||||
|
- `setSyncLabel(mainText, countText = "", chunkText = "", running = false, imageText = "")` (L809, post-phase-150): the composed `full` string becomes `main + (count ? " " + count : "") + (chunk ? " " + chunk : "") + (image ? " " + image : "")` — so the box, the button `title`, and the running `aria-label` all carry `Syncing… <file> (2540/2540) · chunk 125/126 · image 3/15` (the phase-149 box is the surface that shows it; the button's label parts stay as they are — no new span, the running button is the spinner);
|
||||||
|
- update the function's docstring + the phase-129/149 comment blocks that enumerate the label parts (the file cites the "· chunk n/m" contract in ~10 comments — name the image part beside it, do not restate the whole contract).
|
||||||
|
3. No CSS change: the box already wraps mono text with `overflow-wrap: anywhere`; the extra part is just more text.
|
||||||
|
4. ASSUMPTION A2/A3: counts only, one continuous per-file counter (the importer owns the continuity — the UI just renders what it is told).
|
||||||
|
5. ASSUMPTION: no new button label span — while running the parts are hidden (phase 149), so the position rides the composed string only.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level pins in `tests/unit/test_sync_button.py` / `test_frontend_sync_upload.py` gain the image part (the `fmtSyncPhaseLabel` contract: `images_total > 0` → the part present, `0` → absent; the `overview`/`summaries` branches never carry it; the composed `full` string pinned byte-for-byte for a status with both positions).
|
||||||
|
- Integration: a mocked sync run over a fixture PDF shows `images_done` increasing across two polls of `GET /api/sync/status` (the phase-129 mid-run pin idiom).
|
||||||
|
- Coverage: **>90%** on the `app/` change.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] while a PDF's images are described, `GET /api/sync/status` reports the climbing position and `#sync-progress-text` shows `· image n/m`
|
||||||
|
- [ ] the chunk part and the image part can show together and each disappears independently
|
||||||
|
- [ ] the prelude / overview / summaries labels are byte-identical to pre-phase
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 04 — unit + integration tests for the image progress contract
|
||||||
|
|
||||||
|
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The new position is pinned at both ends: the importer emits it, the status machine carries and clears it, the label formats it.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_sync_button.py` — the status-machine pins (the phase-64/98/129 idiom: drive the runner's closures, assert the endpoint dict):
|
||||||
|
- idle: the full-dict pin includes `images_done: 0` / `images_total: 0` and every pre-existing key unchanged;
|
||||||
|
- mid-run: after `_image_hook(3, 15)` the endpoint reports `3/15` while `current_file` / the chunk position keep their own values (the three positions are independent);
|
||||||
|
- per-file reset: a `_hook(...)` call for the NEXT file resets the image position to `0/0` (the pairing rule — a file with no images never shows a stale position);
|
||||||
|
- every terminal (success / failed / cancelled) clears the image position while the FILE counts keep their final values (the phase-64 convention, restated for the new sub-position);
|
||||||
|
- the `phase = "overview"` and `phase = "summaries"` transitions clear it (the same sites as the chunk position).
|
||||||
|
2. `tests/unit/test_importer.py` (or the image-pass home that already uses `tests/fakes.py::FakeEmbedder`) — the hook contract:
|
||||||
|
- a fixture PDF with N embedded images → the hook fires exactly N times, `(1,N) … (N,N)`, each call BEFORE the corresponding describe call (assert the interleaving by recording events in the fake);
|
||||||
|
- a PDF with a pictorial span AND an embedded span → one continuous counter across both spans, never exceeding the combined total, never restarting;
|
||||||
|
- a text-only PDF → the hook never fires;
|
||||||
|
- `on_image_progress=None` → the `ImportSummary` and the stored rows are identical to the hook-less baseline (the byte-identical regression pin);
|
||||||
|
- the blank-gate interaction (phase 151): images skipped by the dedupe/blank gate are NOT counted in `total` (the total is the work about to happen).
|
||||||
|
3. `tests/unit/test_sync_button.py` (frontend-source pins) — `fmtSyncPhaseLabel`'s image part: present iff `images_total > 0`; the composed `full` string pinned for a status carrying both positions (`… (2540/2540) · chunk 125/126 · image 3/15`); the `overview`/`summaries` labels unchanged.
|
||||||
|
4. Integration (`tests/integration/`): a full sync run over a fixture KB containing one image-bearing PDF, polling `GET /api/sync/status` from the test while the mock import runs — assert the observed `images_done` sequence is monotonically non-decreasing and ends at `images_total` (no backwards jumps, no overshoot).
|
||||||
|
5. Run the gates: `uv run pytest -q`, then `uv run pytest --cov=app --cov-report=term-missing` (>90%).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Coverage: **>90%** on `app/api/sync.py` + `app/rag/importer.py` new lines (every clear/reset site, each span, the `None` path).
|
||||||
|
- Deterministic: no sleeps, no real LLM (the fakes), no reliance on wall-clock timing.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the status pins cover idle / mid-run / per-file reset / all three terminals / both phase transitions
|
||||||
|
- [ ] the importer hook pins cover the N-image case, the continuous multi-span counter, the no-image case, and the hook-less byte-identical baseline
|
||||||
|
- [ ] the label formatter's image part is pinned byte-for-byte
|
||||||
|
- [ ] `uv run pytest -q` green; coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 05 — the dedicated E2E: the label advances through a PDF's image span
|
||||||
|
|
||||||
|
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
In the browser, during a PDF sync whose images take time to describe, the live label shows the image position MOVING — the owner's "stuck for several minutes" is impossible on the fixture.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_sync_image_progress.py` (new; the `tests/e2e/test_sync_chunk_progress.py` recipe — a module-scoped app on its own port, `BOR_IMAGES=true` + the `pdf` import token, the `tests/e2e/slow_llm.py` delay proxy in front of the mock so EVERY LLM call (vision describes included) takes `SLOW_DELAY_S = 1.5 s` ≫ the UI's 2 s poll, the autouse truncate, mock-only — `E2E_REAL_LLM` unsupported):
|
||||||
|
- **fixture:** a source (local dir or `file://` git repo, the suite's existing idiom) with ONE PyMuPDF-built PDF carrying 4 genuinely distinct, non-blank images (post-phase-151 they all survive the gate) and enough text to produce ≥2 embed batches (so the chunk part and the image part are both exercised);
|
||||||
|
- **the observation loop:** press Sync, then poll the page (not the API) and collect the DISTINCT `#sync-progress-text` values seen while the run is in flight;
|
||||||
|
- **assertions:**
|
||||||
|
* at least three distinct labels containing `· image ` are observed (e.g. `· image 1/4`, `· image 2/4`, `· image 3/4`) — the freeze the owner reported is a single repeated label, so "≥3 distinct image positions" IS the bug's negation;
|
||||||
|
* the observed image positions are non-decreasing and never exceed the total;
|
||||||
|
* the chunk part appeared earlier in the run and the image part appears during the image span (the two positions are distinct);
|
||||||
|
* the button `title` / `aria-label` carry the same full string (the A4 untruncated contract);
|
||||||
|
* at settle the box shows the counts line and the image part is gone (no stale position in a terminal state);
|
||||||
|
* a page reload mid-run re-attaches with the current image position visible (the never-stale contract).
|
||||||
|
2. Keep the suite deterministic: the slow proxy makes each position outlive the poll; no `wait_for_timeout` as a barrier — wait on the observed text values.
|
||||||
|
3. Run in isolation until green: `uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||||
|
4. Regression run of the neighbouring label suites (they must stay green unedited): `tests/e2e/test_sync_progress_box.py`, `test_sync_chunk_progress.py`, `test_sync_label_mobile.py`, `test_pdf_support.py`, `test_image_documents.py`.
|
||||||
|
5. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit with `--no-gpg-sign`, e.g.
|
||||||
|
`feat(sync): report PDF image-describe progress so the sync label never stalls`
|
||||||
|
(moves `.agents/phases/todo/152_sync_progress_image_pass/` to `complete/`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The dedicated E2E green IN ISOLATION (the phase gate) + the four neighbouring suites green.
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- Mock-only, deterministic, no real vision calls.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] ≥3 distinct `· image n/m` label values observed during one sync (the anti-freeze proof)
|
||||||
|
- [ ] positions are monotonic, cleared at settle, and re-attached on a mid-run reload
|
||||||
|
- [ ] the chunk part's behavior is unchanged
|
||||||
|
- [ ] full suite green, coverage >90%, lint + types clean
|
||||||
|
- [ ] the phase directory moved to `.agents/phases/complete/` and committed with `--no-gpg-sign`
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Phase 153 — the document viewer carries its origin, and the path is copyable
|
||||||
|
|
||||||
|
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "document.html's back link always says '← Sources' even when you arrived from Chat" + optimization "a 'copy path' affordance next to the path chip would speed up the copy-paste-to-terminal workflow."
|
||||||
|
**Story:** `.agents/user_stories/document-viewer.md` (phase 10, complete) + `.agents/user_stories/document-back-navigation.md` (phase 13, complete) — this phase extends both: the phase-13 `back` contract exists on the DIRECT chip link (`app.js::documentUrl`, L447–457) but the phase-26 same-page modal's "Full page" escape hatch (`document-modal.js::fullPageUrl`, L94–97) drops it, so a chat-originated document opened through the modal lands on "← Sources" in the standalone viewer.
|
||||||
|
**Context:** `openDocumentModal(source, path, trigger = null)` (document-modal.js L167) — called from the chat view (app.js L1700, L1762, L1804, L1825, L1908 — source chips, nearby-doc links, source-image figures) and from the RAG folder table (sources.js L2023). The "Full page" anchor (L202) and the no-skeleton `window.open` fallback (L172) both use `fullPageUrl`. The standalone back-link resolution (document.js, phase 13) honors a same-origin relative `back` param (`/` prefix, not `//`) and falls back to `/sources.html` otherwise; its label matches the resolved target (Chat / Sources) — already shipped, untouched by this phase. The clipboard idiom is established (app.js L216–221, L2375–2384; history.js L297–301): `navigator.clipboard.writeText` in a try — a non-secure (http) homelab origin rejects the clipboard, so the failure path renders an inline fallback field (an input-like node that selects its content on focus) + a status line.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A document opened from the in-page modal carries its origin to the standalone viewer (chat → back to chat, Sources → back to Sources — the phase-13 resolution, finally fed on every path), and the owner can copy the document path (`source/path`) from both the modal's meta and the standalone titlebar with one click, with the established clipboard-fallback contract.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `13_document_back_navigation` (complete) — the `back` param + same-origin validation + label-match contract this phase feeds.
|
||||||
|
- `10_story_document_viewer` (complete) — the viewer's meta row (source/format chips, mono path) the Copy-path control joins.
|
||||||
|
- `144_modal_back_button` (complete) — the modal's history-entry model; this phase adds no navigation, only a URL param on the escape hatch.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The origin param (task 01):** `openDocumentModal(source, path, trigger = null, back = null)` and `fullPageUrl(source, path, back = null)` — `back` is appended as `&back=` + `encodeURIComponent(back)` ONLY when truthy (a null/absent `back` keeps the URL byte-identical to pre-phase: the Sources path is unchanged). The modal remembers the `back` of the CURRENT open (a module-level variable set at each `openDocumentModal` — the modal is one-at-a-time by its own contract, so a single variable is sound). The five chat-view call sites (app.js) pass `"/"` — chat is the shell's home route, the same value `documentUrl` already uses; the sources.js L2023 call site omits it (null → no param → the viewer's `/sources.html` default). The no-skeleton fallback (L172) uses the same `back`. The app.js L447–457 comment ("this URL is also what the modal's 'Full page' link points at") becomes TRUE — update the comment where it cites the contract, do not restate it.
|
||||||
|
- **Copy path (task 02):** ONE control shape on both surfaces — an icon + "Copy path" text button (visible text, `aria-label` "Copy the document path" is redundant with the text so it carries the plain text; icon `aria-hidden`), placed in the standalone titlebar's meta row after the mono path and in the modal's meta block. The copied string is `source + "/" + path` (ASSUMPTION A1 — the exact `source/path` form the `read` tool, the chips, and the Sources table use; the paste target is the owner's own tooling). Clipboard contract (the share-link-fallback idiom, app.js L216–221): `navigator.clipboard.writeText` in a try → success: a transient status "Path copied." (visible near the control + the page's live region; auto-clears ~4 s, the toast idiom) → failure: an inline fallback field (input-like, prefilled with the path, selects its content on focus — the `.share-link-fallback` pattern) + the status line "Path ready — copy it from the field." (the owner-locked fallback wording, adapted for the path). Per-page duplication of the small helper is the established idiom (history.js L301) — document.js and document-modal.js each carry their own; no new shared module.
|
||||||
|
- **A11y (both tasks):** the standalone page already has the landmarks/label frame (phase 10/13); the new button is a labeled control (WCAG 1.3.1/4.1.2); the status line is `role="status"`; the fallback field is a labeled input-like node (the share-fallback precedent). No new color-alone state (B5).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_modal_back_context.md` — the `back` param through `openDocumentModal`/`fullPageUrl` + the six call-site updates + the comment correction.
|
||||||
|
2. `02_copy_path.md` — the Copy-path control on the standalone titlebar + the modal meta, clipboard idiom + fallback field + status line.
|
||||||
|
3. `03_unit_pins.md` — the URL-contract pins (with/without `back`), the modal param default, the copy-path strings + fallback contract.
|
||||||
|
4. `04_e2e_viewer_context.md` — the dedicated Playwright suite: chat→modal→Full page back-to-chat, Sources→modal→Full page back-to-Sources, Copy path success + fallback.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/` source-level pins — `fullPageUrl` byte-identical without `back`, `&back=%2F` appended with `back = "/"` (encoded); `openDocumentModal`'s default is null (the Sources path unchanged); the copy-path status strings pinned byte-for-byte (success + fallback); the phase-13 resolution pins (a garbage `back` still falls back to `/sources.html` — the regression guard).
|
||||||
|
- Integration: the `/api/documents/content` + `GET /document.html` contract is untouched (no new server surface — the phase is frontend-only; the server pins are regression guards only).
|
||||||
|
- Coverage: **>90%** on modified code (JS pinned by the source-level suites + the E2E).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py` — a citation-bearing answer renders the chips).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the Copy-path affordance is a user-visible feature — name it where the viewer is described).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] chat view → source chip → modal → "Full page": the standalone back link is `href="/"` with the "Chat" label; Sources table → modal → "Full page": `href="/sources.html"` with the "Sources" label (E2E)
|
||||||
|
- [ ] a direct Sources table chip (no `back` param) and the pre-phase URL are byte-identical — no regression on the phase-13/26 paths
|
||||||
|
- [ ] Copy path on both surfaces: clipboard success reads "Path copied."; a rejected clipboard renders the prefilled fallback field + "Path ready — copy it from the field." (E2E pins both branches)
|
||||||
|
- [ ] the copied string is exactly `source/path` (the `read`-tool form)
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 01 — the modal's "Full page" escape hatch carries the origin `back`
|
||||||
|
|
||||||
|
**Phase:** `153_document_viewer_context` · **Story:** `.agents/user_stories/document-back-navigation.md`
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A document opened from the in-page modal reaches the standalone viewer with the same `back` context the direct chip link already carries (phase 13) — chat-originated opens land back on chat, Sources-originated opens keep the `/sources.html` default.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document-modal.js`:
|
||||||
|
- `fullPageUrl(source, path, back = null)` (L94): append `"&back=" + encodeURIComponent(back)` when `back` is truthy — the encoding contract is the one `app.js::documentUrl` (L447–457) already documents (percent-encoded query values; `back` is a same-origin relative URL, never a URL with its own query).
|
||||||
|
- `openDocumentModal(source, path, trigger = null, back = null)` (L167): store `back` in a module-level variable (the modal is one-at-a-time by its own contract — a single variable is sound; name it `currentBack` and reset it at the top of every open); the no-skeleton fallback (L172) and the "Full page" anchor (L202: `openEl.href = fullPageUrl(source, path, currentBack)`) both read it.
|
||||||
|
- Update the L39 and L93–97 comment blocks where the "Full page" URL is described — the contract now matches `documentUrl`'s (cite it, do not restate it).
|
||||||
|
2. `frontend/assets/app.js` — the five chat-view call sites pass `"/"` (chat is the shell's home route, the value `documentUrl` already uses): L1700 (`openDocumentModal(s.source, s.path, chip)`), L1762 (`… link)`), L1804 (`… fig)`), L1825 (`… chip)`), L1908 (`… fig)`).
|
||||||
|
3. `frontend/assets/sources.js` L2023 (`openDocumentModal(d.source, d.path, link)`) — UNCHANGED (null → no param → the viewer's `/sources.html` default). Do not touch it; the byte-identity is the contract.
|
||||||
|
4. `frontend/assets/app.js` L447–457: correct the comment where it says the chip URL "is also what the modal's 'Full page' link points at" — after this task that is true on the chat path (the `back` param rides along); name the fix, keep the existing contract language.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level pins in the frontend test suite gain: `fullPageUrl("s", "p")` byte-identical to pre-phase (no `back`); `fullPageUrl("s", "p", "/")` === `"/document.html?source=s&path=p&back=%2F"`; `openDocumentModal` with no `back` arg leaves the Sources path byte-identical; the no-skeleton fallback uses the stored `back`.
|
||||||
|
- Coverage: **>90%** on the modified JS (source-level pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] chat chip → modal → "Full page" URL carries `&back=%2F`; Sources table → modal → "Full page" URL is byte-identical to pre-phase
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] no behavior change in the phase-13/26/144 paths (the direct chip link, the context-menu escape hatch, the modal back-close)
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 02 — Copy path on the standalone titlebar and the modal meta
|
||||||
|
|
||||||
|
**Phase:** `153_document_viewer_context` · **Story:** `.agents/user_stories/document-viewer.md`
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The owner copies the document path with one click on both surfaces — the standalone viewer's titlebar meta row and the modal's meta block — with the established clipboard + inline-fallback contract (a non-secure homelab origin rejects `navigator.clipboard`, so the failure path must always work).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/document.html` — the titlebar meta row (phase 10/13, L108–113 area): after the mono path cell, add the Copy-path control — an icon (`aria-hidden`) + visible "Copy path" text (the text IS the accessible name; no redundant `aria-label` needed). Follow the icon+text button house pattern (the Sources table's "Ignore paths"/"Remove" buttons are the shape reference).
|
||||||
|
2. `frontend/assets/document.js` — the control's behavior (a page-local helper, the per-page duplication idiom — history.js L301):
|
||||||
|
- the copied string is `source + "/" + path` (ASSUMPTION A1 — the exact `source/path` form the `read` tool and the chips use; the same two values the page already renders in the meta row);
|
||||||
|
- `navigator.clipboard.writeText(…)` in a try: success → a transient visible status "Path copied." near the control (auto-clears ~4 s, the toast idiom — the node is visual-only) + the page's live region reads the same text (one a11y announcer, the share pattern: app.js L219–221); failure → an inline fallback field (an input-like node, prefilled with the path, selects its full content on focus — the `.share-link-fallback` pattern) + the status line "Path ready — copy it from the field." (the owner-locked fallback wording, path-adapted). A failed copy NEVER clears the previous state; re-clicking re-attempts.
|
||||||
|
- the live region: reuse the page's existing `role="status"` surface where one exists; if the standalone page has none, add one visually-hidden node (the a11y frame is otherwise phase-10's — do not restructure it).
|
||||||
|
3. `frontend/assets/document-modal.js` — the modal's meta block: the SAME control shape + the same page-local helper (per-page duplication — the modal module is separate). The modal already has `statusLine(text)` (L100) for its loading states — the success/failure status reuses that node shape (`role="status"`), and the fallback field is placed in the meta block next to the control.
|
||||||
|
4. `frontend/assets/styles.css` — the control's styles (icon+text button, the house pattern) + the fallback field (input-like, select-on-focus is JS; the styling mirrors `.share-link-fallback`). Theme-aware (built-in variables only — no new palette entry, B3/B4: an all-default theme stays byte-identical in the served HTML… the control is in `document.html`/`index.html` markup — it is ALWAYS present, so check the B4 byte-identical contract: the control is a new static node in the built-in pages; B4's contract is about THEME no-ops (no `ui_settings` row → no `#bor-theme` tag), not about page content — the new static node is fine, but the unit test that parses `document.html`/`index.html` structure (if any) must gain the node).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: source-level pins — the copied string is `source + "/" + path` (pinned byte-for-byte for a path with spaces + slashes); the success status string "Path copied."; the fallback field's prefill + select-on-focus contract; the failure status string "Path ready — copy it from the field."; a rejected-clipboard E2E-mock (the suite's http-origin idiom) renders the fallback field.
|
||||||
|
- Coverage: **>90%** on the modified JS (source-level pins + E2E).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] both surfaces: click → clipboard success → "Path copied." (visible + live region); clipboard rejected → prefilled fallback field + "Path ready — copy it from the field."
|
||||||
|
- [ ] the fallback field's content selects on focus (the share-link-fallback idiom)
|
||||||
|
- [ ] the a11y frame is intact: labeled control, `role="status"` announcer, no color-alone state
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 03 — unit pins for the URL contract and the copy-path strings
|
||||||
|
|
||||||
|
**Phase:** `153_document_viewer_context` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The two contracts of this phase — the `back` param on the modal's full-page URL, and the copy-path status/fallback strings — are pinned at the source level so a later refactor cannot silently drop the origin or break the fallback contract.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/` — locate the existing frontend source-level pin suite (the `document-modal.js` / `document.js` / `app.js` pins — the phase-13/144 suites are the precedent) and extend it:
|
||||||
|
- `fullPageUrl`: no-`back` byte-identical to pre-phase; `back = "/"` → `&back=%2F` appended (the exact string pinned); `back = null` and absent are identical;
|
||||||
|
- `openDocumentModal`: default `back` is null; a chat open (`back = "/"`) is what the "Full page" anchor and the no-skeleton fallback use;
|
||||||
|
- copy-path: the composed copy string for a `source`/`path` pair with spaces + slashes (pinned byte-for-byte); the success string; the fallback field prefill; the failure string;
|
||||||
|
- regression guard: the phase-13 resolution pins stay green (a missing/garbage/protocol-relative `back` falls back to `/sources.html`; a same-origin relative `back` wins — do NOT modify those pins, only add beside them).
|
||||||
|
2. If `document.html`'s structure is pinned by a unit test (check the existing suite — the no-CDN test pins `index.html` per AGENTS.md rule 6; check whether a `document.html` pin exists), gain the Copy-path node.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; the suite's existing pins all stay green (no pin was deleted or loosened — additions only).
|
||||||
|
- Coverage: the pins exercise the modified branches; `app/`-side code is untouched (frontend-only phase).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
|
||||||
|
- [ ] the no-`back` URL pin is byte-identical to the pre-phase string (the byte-identity contract, the phase-13 precedent)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 04 — the dedicated Playwright suite: origin back + copy path
|
||||||
|
|
||||||
|
**Phase:** `153_document_viewer_context` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_document_viewer_context.py`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated E2E proves the two user-facing contracts: a document opened through the modal from Chat lands back on Chat (and from Sources on Sources) after "Full page", and Copy path works on both surfaces — including the rejected-clipboard fallback branch.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_document_viewer_context.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the neighboring suites: the mock-LLM server by default, the DB truncate/re-import per module):
|
||||||
|
- **test_chat_origin_full_page_backs_to_chat:** the mock-LLM answer renders a citation-bearing bubble (the mock's standard grounded turn); the chat view's source chip is clicked (the phase-26 intercept opens the modal); "Full page" is clicked (new tab — the Playwright context's second page); on the standalone page: `#doc-back`'s `href === "/"` AND its text is "Chat" (the phase-13 label-match contract, now actually fed).
|
||||||
|
- **test_sources_origin_full_page_backs_to_sources:** the RAG folder table's path link is clicked (the modal); "Full page"; the standalone `#doc-back` `href === "/sources.html"` + text "Sources" (the default — the URL carries NO `back` param; assert the param's ABSENCE too).
|
||||||
|
- **test_direct_chip_link_still_carries_back (regression guard):** a context-menu / middle-click on the chat chip (the `documentUrl` escape hatch, pre-phase behavior) still lands with `&back=%2F` — the phase-13 path is untouched.
|
||||||
|
- **test_copy_path_clipboard_success:** the standalone titlebar's Copy path is clicked with the clipboard permitted (the suite's https-origin idiom, or a granted clipboard mock): the status "Path copied." is visible + the live region reads it; the clipboard content is exactly `source/path`.
|
||||||
|
- **test_copy_path_fallback:** the clipboard is rejected (the suite's http-origin idiom — the established pattern): the fallback field appears, prefilled with the path, and focuses+selects its content on focus; the status reads "Path ready — copy it from the field."; the modal surface branch is covered by the same two tests run against the modal's control (parameterize the surface: standalone / modal).
|
||||||
|
- **test_direct_open_without_back_defaults (regression guard):** `GET /document.html?source=…&path=…` (no `back`, e.g. a shared-chat document link) still resolves to `/sources.html` + "Sources" — the phase-13 fallback.
|
||||||
|
2. The suite boots exactly like the neighboring story suites (the app under the mock LLM; a saved chat with a citation-bearing answer is seeded the same way `test_document_viewer.py` / `test_document_back_navigation.py` do — read those two files for the fixture idiom and mirror it, do not invent a new one).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the file above, run in isolation: `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||||
|
- The suite is DETERMINISTIC: mock-LLM answers only (no `E2E_REAL_LLM` dependency), fixed fixture document (the test-KB document with a known `source/path` — the copied-string assertion is a byte-for-byte pin).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` green in isolation
|
||||||
|
- [ ] every contract of the phase's Completion Criteria is asserted by the suite (the origin back on both surfaces, the label match, the `back` absence on the Sources path, the copy string, both clipboard branches)
|
||||||
|
- [ ] the suite adds no real-LLM dependency and no fixture beyond the established idiom
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Phase 154 — sync + upload stamps say WHEN (date-qualified freshness)
|
||||||
|
|
||||||
|
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "RAG 'Synced 16:06' badge is time-only — ambiguous the next day" + "Sources 'Uploaded — press Sync sources to import it.' has no timestamp — it's server-persisted and re-rendered on every mount; after that archive is imported by a sync, the line can sit there indefinitely and mislead."
|
||||||
|
**Story:** n/a (audit-derived phase)
|
||||||
|
**Context:** `sources.js::fmtSyncTime(iso)` (L814–819) — `HH:MM` only; its consumer `applySyncSuccess` (L1195–1196) settles the RAG sync button as `Synced ${time}` (or bare "Synced" when the ISO is unparseable — the app-restart case: the sync status is in-memory, A12). The label parts writer (L821 onward) carries the same text in the button `title` + the running `aria-label` — the qualified stamp rides all three (the phase-128 parts-writer idiom: write once, three surfaces). `git-sources.js::fmtUploadResult(detail, name)` (L1128–1133) — the no-count contract (phase 90): `Uploaded <name> — press Sync sources to import it.` (fresh 202 settle, the page-local `lastUploadName`) or the nameless `Uploaded — press Sync sources to import it.` (the boot re-render, `initUploadStatus` L1279–1304, where the name is unknown after a reload). `GET /api/git-sources/upload/status` (git_sources.py L704–729) ALREADY exposes `started_at` / `finished_at` (ISO) + `detail` — the server change is ZERO; this phase is frontend-only. The per-page duplication idiom is established (history.js L301's copyText, sources.js vs app.js) — the date-qualifier helper is duplicated into the two modules, not shared.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Both freshness stamps tell the owner *when*: a same-day stamp stays `HH:MM` (byte-identical to today's behavior); a non-same-day stamp carries the date (`M/D, h:MM AM/PM` — the table-date family); the Sources upload line carries its completion stamp on every render path (fresh settle AND boot re-render), so a stale "Uploaded" line is self-dating.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `150_drop_sync_result` (complete) — the settled sync button is the sole "Synced" surface (the phase-149 box is the running label; the button settles it).
|
||||||
|
- `64_sync_upload_progress` (complete) — the upload 202/poll/re-attach state machine this phase's line change rides on.
|
||||||
|
- `90_upload_no_scan` (complete) — the no-count result-line contract (phase 90, A2) this phase EXTENDS (a stamp is added; no counts return).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The date-qualifier rule (ASSUMPTION A1, owner-locked):** the stamp is computed from the ISO's LOCAL date (the page's clock — the table dates already use local time). Same local year as now AND same calendar day → `HH:MM` (byte-identical to the current `fmtSyncTime` output — a same-day badge is unchanged). Otherwise, same year → `M/D, h:MM AM/PM` (e.g. `10/2, 3:14 PM`); different year → `M/D/YYYY, h:MM AM/PM` (e.g. `1/3/2025, 9:12 PM`). An unparseable/missing ISO → the existing fallback (sync: bare "Synced"; upload: no stamp — the pre-phase strings). `h:MM` uses the locale 12-hour form WITHOUT leading zero on the hour (matching the table dates' `5:34:06 PM` family — drop the seconds), month/day without leading zero (the tables' `M/D/YYYY` shape).
|
||||||
|
- **RAG sync badge (task 01):** `fmtSyncTime` becomes the date-qualifier (keep the name — its contract widens; or add `fmtStamp` and delegate — the executor's choice, the pins fix the output). `applySyncSuccess` is unchanged in shape (`settleSyncButton(time ? \`Synced ${time}\` : "Synced")` — the qualifier lives inside the formatter). The parts writer's comment block (L821 onward) names the qualified stamp in the label parts where the `HH:MM` contract is cited (cite, don't restate).
|
||||||
|
- **Sources upload line (task 02):** `fmtUploadResult(detail, name, when)` — `when` is the precomputed stamp string (the caller computes it from `status.finished_at`, falling back to `status.started_at`; the success state always has a `finished_at` on the server, the fallback is the defensive guard for a legacy/partial payload). Shapes (the phase-90 contract extended, name present/absent): `Uploaded <name> (<stamp>) — press Sync sources to import it.` / `Uploaded <stamp> — press Sync sources to import it.`; `when` empty (running state, legacy payload) → BYTE-IDENTICAL to the current strings (the no-stamp fallback). Call sites: the poll settle (L1239–1244, the page-local `lastUploadName`) and `initUploadStatus`'s success branch (L1279–1304, nameless) — both read `status.finished_at` from the same `/upload/status` payload. The L78–96 and L1125–1133 comment blocks name the stamp in the contract (cite, don't restate).
|
||||||
|
- **A11y:** the stamps are text in existing `role="status"` / button-label surfaces — no new nodes, no color-alone state (B5).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_rag_sync_stamp.md` — the date-qualifier in `sources.js` (`fmtSyncTime`'s contract widens; same-day byte-identical).
|
||||||
|
2. `02_upload_line_stamp.md` — the `when` param on `fmtUploadResult` + both call sites reading `finished_at`/`started_at`.
|
||||||
|
3. `03_unit_pins.md` — the qualifier's boundary pins + both stamp shapes (with/without name, with/without stamp).
|
||||||
|
4. `04_e2e_status_freshness.md` — the dedicated Playwright suite.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: source-level pins — the qualifier (same-day `HH:MM` byte-identical; yesterday `M/D, h:MM AM/PM`; last-year `M/D/YYYY, h:MM AM/PM`; garbage/missing ISO → the pre-phase fallbacks — a fixed clock in the test, not wall-clock math on `new Date()`); `fmtUploadResult`'s four shapes (name×stamp, name×no-stamp byte-identical, no-name×stamp, no-name×no-stamp byte-identical); the `applySyncSuccess` settle string for a non-same-day `finished_at`.
|
||||||
|
- Integration: ZERO server change — the `/upload/status` payload is untouched (regression guard: the full-dict pin, if one exists, stays byte-identical).
|
||||||
|
- Coverage: **>90%** on the modified JS (source-level pins + E2E).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the stamp format is a visible UI change — name it where the sync/upload surfaces are described).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a same-day sync settles as `Synced HH:MM` — byte-identical to pre-phase (the pin + the E2E)
|
||||||
|
- [ ] a non-same-day `finished_at` (the unit pins; the E2E asserts the same-day shape on a real mock sync — the date branch is unit-pinned with a fixed clock, the E2E never depends on the wall clock)
|
||||||
|
- [ ] the upload line carries the stamp on BOTH render paths (the fresh 202 settle with the name; the boot re-render without it) — the E2E seeds a prior upload via the API and reloads the page
|
||||||
|
- [ ] a running/legacy upload payload renders the pre-phase strings byte-identically (no partial-state regression)
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 01 — the RAG sync badge is date-qualified
|
||||||
|
|
||||||
|
**Phase:** `154_sync_status_freshness` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The settled RAG sync button reads `Synced HH:MM` when the run finished today and `Synced M/D, h:MM AM/PM` (or `M/D/YYYY, …` across years) otherwise — a badge that no longer lies about its age.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/sources.js`:
|
||||||
|
- `fmtSyncTime(iso)` (L814–819): the contract widens to the date-qualifier (the phase design's A1 rule). Keep the function name if the delegation is clean; otherwise add `fmtStamp(iso)` and let `fmtSyncTime` delegate — the pins fix the OUTPUT, the executor's choice is internal. The parse guard (`Number.isNaN(d.getTime()) → ""`) is UNCHANGED — an unparseable ISO still yields the bare "Synced" settle.
|
||||||
|
- the qualifier compares LOCAL calendar dates (the page's clock — the tables' `Updated` columns already render local dates; the idiom is there in `history.js`'s updated-cell formatter — mirror its `toLocaleDateString`/manual shape, the `M/D(/YYYY), h:MM AM/PM` family with no leading zeros).
|
||||||
|
- `applySyncSuccess` (L1195–1196) is UNCHANGED in shape — the qualifier lives inside the formatter; the settle string for a same-day run is byte-identical to pre-phase.
|
||||||
|
- the parts-writer comment block (L821 onward) names the qualified stamp where the `HH:MM` label contract is cited (cite, don't restate — the block already lists the label parts; the badge is one of them).
|
||||||
|
2. No CSS change (the label part ellipsizes; the longer stamp is just more text in the same part).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level pins gain the qualifier's boundary set with a FIXED clock (the test's fake clock — never `new Date()` in the test body): same-day → `HH:MM` (byte-identical to the pre-phase pin); yesterday (same year) → `M/D, h:MM AM/PM`; last-year same M/D → `M/D/YYYY, h:MM AM/PM`; hour formatting (`3:14 PM` — no leading zero on the hour; `12:05 AM` edge); garbage ISO → `""` (unchanged).
|
||||||
|
- Coverage: **>90%** on the modified JS.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] same-day settle is byte-identical to pre-phase (`Synced HH:MM`)
|
||||||
|
- [ ] the boundary pins pass on a fixed clock; the wall clock never appears in a test assertion
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — the Sources upload line carries its completion stamp on every render path
|
||||||
|
|
||||||
|
**Phase:** `154_sync_status_freshness` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`Uploaded — press Sync sources to import it.` becomes self-dating — `Uploaded 21:14 — …` (or `Uploaded 10/2, 3:14 PM — …`) on BOTH render paths (the fresh 202 settle with the page-local name, the boot re-render without it) — so a line that outlives its upload can no longer masquerade as fresh.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/git-sources.js`:
|
||||||
|
- `fmtUploadResult(detail, name, when)` (L1128–1133): the third param is the PRECOMPUTED stamp string (the phase design's A1 format, computed by a page-local `fmtUploadStamp(status)` helper — the per-page duplication idiom; it reads `status.finished_at` first, falls back to `status.started_at`, and yields `""` when both are missing/absent). Shapes:
|
||||||
|
- `detail.message === "uploaded"` + `name` + `when`: `Uploaded ${name} (${when}) — press Sync sources to import it.`
|
||||||
|
- `detail.message === "uploaded"` + no `name` + `when`: `Uploaded ${when} — press Sync sources to import it.`
|
||||||
|
- `when === ""` (any name state): BYTE-IDENTICAL to the current strings (the legacy/partial-payload guard — the phase-90 contract for a payload without timestamps).
|
||||||
|
- the non-`uploaded` branch (`"The upload finished."`) is unchanged.
|
||||||
|
- the poll settle (L1239–1244): `fmtUploadResult(detail, lastUploadName, fmtUploadStamp(status))` — the poll tick already holds the full status object.
|
||||||
|
- `initUploadStatus`'s success branch (L1279–1304): the same third arg from the boot-fetched status (the name stays null — the page-local `lastUploadName` is null after a reload; the stamp comes from the SERVER payload, so the nameless line is still self-dating after a reload — that is the point).
|
||||||
|
- the L78–96 and L1125–1133 comment blocks name the stamp in the no-count contract (cite, don't restate — the blocks already document the nameless re-render; add the stamp to the described string shape).
|
||||||
|
2. NO server change: `GET /api/git-sources/upload/status` already exposes `started_at` / `finished_at` (git_sources.py L704–729) — do not touch `app/` in this task.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level pins gain the four `fmtUploadResult` shapes (name×stamp, name×no-stamp → byte-identical to the pre-phase string, no-name×stamp, no-name×no-stamp → byte-identical) + `fmtUploadStamp` (finished wins over started; started when finished is missing; `""` when both absent; a garbage ISO → `""`).
|
||||||
|
- Coverage: **>90%** on the modified JS.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the four shape pins pass; the two no-stamp shapes are byte-identical to pre-phase
|
||||||
|
- [ ] `initUploadStatus`'s success branch passes the stamp (a boot re-render of a prior upload shows the stamp even with the name unknown)
|
||||||
|
- [ ] `app/` is untouched in this task (a `git diff --name-only` scoped check)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Task 03 — unit pins for the qualifier and the four stamp shapes
|
||||||
|
|
||||||
|
**Phase:** `154_sync_status_freshness` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The date-qualifier rule (A1) and both stamp line shapes are pinned at the source level on a fixed clock — a later refactor cannot silently revert to time-only or break the byte-identity fallbacks.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/` — locate the existing `sources.js` / `git-sources.js` source-level pin suite (the phase-64/90/150/152 suites are the precedent — read `test_frontend_sync_upload.py` and its siblings for the idiom) and extend it:
|
||||||
|
- the qualifier (task 01's boundary set — fixed clock; the byte-identity of the same-day output against the PRE-PHASE pinned string — pin the old string verbatim in the test so the byte-identity is assertable, not assumed);
|
||||||
|
- `fmtUploadResult`'s four shapes (task 02 — including the two byte-identity fallbacks pinned against the pre-phase strings);
|
||||||
|
- `fmtUploadStamp` (finished/started/missing/garbage).
|
||||||
|
2. The `GET /api/git-sources/upload/status` payload pin (if the suite pins the full dict — the phase-64 precedent): UNCHANGED — this phase adds no server keys; the pin stays byte-identical (a regression guard that task 02 really touched nothing in `app/`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; every pre-existing pin stays green (additions only — no pin deleted or loosened).
|
||||||
|
- Coverage: the pins exercise the modified branches.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
|
||||||
|
- [ ] the two byte-identity fallback pins assert against the verbatim pre-phase strings
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Task 04 — the dedicated Playwright suite: date-qualified stamps
|
||||||
|
|
||||||
|
**Phase:** `154_sync_status_freshness` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_sync_status_freshness.py`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated E2E proves the two visible contracts on the live UI: a fresh mock sync settles the RAG button as `Synced HH:MM` (same-day shape), and the Sources upload line carries its stamp on BOTH render paths — the fresh settle (with the file name) and the boot re-render after a reload (nameless, still stamped from the server payload).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_sync_status_freshness.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the neighboring sync suites — `test_sync_image_progress.py` (phase 152) and the phase-90/64 suites are the precedents; read one and mirror its boot, do not invent a new fixture):
|
||||||
|
- **test_fresh_sync_settles_same_day_shape:** a mock sync run (the suite's sync fixture — a small fixture source; the mock-LLM makes the run fast) → the RAG sync button settles; assert the label matches `Synced \d{2}:\d{2}` (the same-day shape — the test runs "now", so the qualifier must yield `HH:MM`; a regex pin, NOT a wall-clock comparison — the suite never asserts on the actual time value).
|
||||||
|
- **test_upload_line_fresh_settle_carrys_name_and_stamp:** a real small fixture archive is uploaded through the UI (the phase-90 suite's upload idiom — a generated zip in `tmp_path`) → the 202 toast → the poll settles → the ready-for-sync line reads `Uploaded <name> (<stamp>) — press Sync sources to import it.` (the name is the fixture's safe name; the stamp matches the `(\d{2}:\d{2}|\d{1,2}/\d{1,2}(?:/\d{4})?, \d{1,2}:\d{2} [AP]M)` shape pin — same-day in practice, the shape pin covers both).
|
||||||
|
- **test_upload_line_boot_rerender_stamped_nameless:** continue from the previous state (or re-seed): RELOAD the Sources view/page → the boot re-render (`initUploadStatus`'s success branch) shows the NAMELESS line WITH the stamp: `Uploaded <stamp> — press Sync sources to import it.` (the name is gone after the reload — the phase-90 contract — but the stamp survives because it comes from the server payload: THIS is the regression the phase fixes).
|
||||||
|
- **test_running_upload_line_unchanged (regression guard):** a running upload (the poll's processing state) never renders the ready-for-sync line (the phase-90 contract — the line appears only on settle; assert the bare "Processing…" label during the run with the suite's slow-LLM/delay idiom if a deterministic in-flight observation is needed, otherwise the settle-only assertion).
|
||||||
|
2. The suite is DETERMINISTIC: mock-LLM only, no `E2E_REAL_LLM` dependency; time assertions are SHAPE pins (regex), never value comparisons against the wall clock.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` green in isolation
|
||||||
|
- [ ] the boot-re-render test proves the nameless line carries the stamp (the phase's actual fix)
|
||||||
|
- [ ] the suite asserts shapes, not wall-clock values; no real-LLM dependency
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Phase 155 — folder summaries in varied, natural language
|
||||||
|
|
||||||
|
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "All `lite` folder descriptions open with 'This folder contains…' — formulaic; a prompt nudge for variety would help."
|
||||||
|
**Story:** n/a (audit-derived phase; the folder-summary machinery is phase 94's — `94_ls_tree_drilldown` (complete))
|
||||||
|
**Context:** `app/rag/folder_summaries.py::FOLDER_SUMMARY_INSTRUCTION` (L102–108) — the current neutral instruction: "From the document list below, write a 1-3 sentence plain-text summary of what this folder's documents cover, in natural language. Do not use markdown. Do not invent anything that is not in the list." The `FOLDER_SUMMARY_MODE` marker (L91) and the `FOLDER_HEADER_PREFIX` (`"Folder: "`) are the E2E mock's keys — `tests/e2e/mock_llm.py` keys on the marker + the header line (a canned reply that is a pure function of the request), NOT on the instruction text — an instruction change is mock-safe as long as the marker + header prefix stay byte-identical. `generate_folder_summaries(db, llm, *, skip, only_missing, changed_docs, on_progress)` (L440) — `changed_docs=None` is the FULL regeneration (every candidate folder; the `scripts/import_docs.py` CLI's default); the sync path is change-gated (phase 127). PLAN's "persona text changes through the plan, not in code" does NOT bind this constant — it is a one-shot prompt constant (the phase-94/127 idiom), not the chat persona; the marker contract is the load-bearing piece and it is untouched. LLM calls are free (A21) — a full regeneration's cost is not a constraint.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
New and regenerated folder summaries open in varied, natural language — no more catalogue of "This folder contains…" — via an instruction nudge, plus a one-shot `scripts/regen_folder_summaries.py` that refreshes every stored summary against the new instruction (the live RAG catalogue's 15+ source-root rows currently all carry the formulaic opener).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `94_ls_tree_drilldown` (complete) — the folder-summary machinery (grouping, prompt, upsert, prune).
|
||||||
|
- `127_sync_incremental_folder_summaries` (complete) — the `changed_docs` mode contract the new script's FULL mode (`changed_docs=None`) must not disturb.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The nudge (task 01):** `FOLDER_SUMMARY_INSTRUCTION` gains the variety direction: keep the 1–3 sentence / plain-text / no-markdown / no-invention constraints UNCHANGED; add: "Open in a varied, natural way — state what the folder is for, the way a person would describe it. Do NOT begin with 'This folder contains' or 'The <folder name> folder contains'." (ASSUMPTION A1, owner-locked — the exact wording above; the executor may polish grammar but not the two forbidden openers, which are the pin.) `FOLDER_SUMMARY_MODE` and `FOLDER_HEADER_PREFIX` stay byte-identical (the mock contract — a unit pin asserts it).
|
||||||
|
- **The regen script (task 02):** `scripts/regen_folder_summaries.py` — the `scripts/import_docs.py` entrypoint idiom (argparse; the app settings/`get_settings()` fail-loud boot; the DB session construction; the `LLMClient` construction — read `scripts/import_docs.py`'s main path and mirror its exact setup, the `--limit N` debug passthrough included if import_docs has one — check and match). Body: `await generate_folder_summaries(session, llm)` — FULL mode (`changed_docs=None` the default), `only_missing=False` (a REGEN — every stored row is refreshed, not just gaps), `skip=False`. End: the importer's cron-safe single summary log line idiom (one line: counts out of the returned stats dict) + the per-`uv run` exit code 0 only on success. NO new API surface, NO schema change, NO change to the sync path's change-gated behavior (phase 127's contract is untouched — the script is a standalone maintenance entrypoint).
|
||||||
|
- **The mock (task 04's suite prerequisite):** `tests/e2e/mock_llm.py`'s CANNED folder-summary reply may evolve (it is a pure function of the request — the marker/header keys are unchanged); its canned text for a folder is updated to the new voice (e.g. "Documentation for <folder>, covering <first-doc-theme> …" — a natural opener, never the forbidden ones) so the E2E can assert the stored text's shape. The mock's OTHER markers/contracts are untouched.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_prompt_nudge.md` — the instruction update + the marker/header byte-identity pins.
|
||||||
|
2. `02_regen_script.md` — `scripts/regen_folder_summaries.py` (full mode, the import_docs idiom).
|
||||||
|
3. `03_unit_pins.md` — the instruction pin, the mock-contract guard, the script wiring.
|
||||||
|
4. `04_e2e_folder_voice.md` — the dedicated Playwright suite (the mock's new canned voice + the RAG table rendering it).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `FOLDER_SUMMARY_INSTRUCTION` pinned (the two forbidden openers present as the constraint; the 1–3 sentence/plain-text/no-markdown/no-invention words preserved); `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX` byte-identical (the mock contract); the script's wiring (a fake LLM + a fixture catalogue: `generate_folder_summaries` called with `changed_docs=None`, `only_missing=False`, `skip=False` — the stats dict surfaces in the summary line; a bad settings boot fails loud — the app's validator idiom).
|
||||||
|
- Integration: a full-mode regeneration over a fixture catalogue upserts every candidate row (the phase-94 idiom) and prunes nothing (no catalogue change — the prune rule is count-driven; a pure regen touches only `updated_at` + text).
|
||||||
|
- Coverage: **>90%** on new/modified code (`app/rag/folder_summaries.py`, the script).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py`).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new script is a user-visible command — it joins the README's import/update workflow list beside `import_docs`; the summary voice is a visible content change).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX` are byte-identical to pre-phase (the mock contract pin)
|
||||||
|
- [ ] `uv run python -m scripts.regen_folder_summaries` runs a FULL regeneration (the stats summary line logs; exit 0; every candidate row's `updated_at` advances, unchanged folders' text is the mock's new voice)
|
||||||
|
- [ ] the sync path's change-gated behavior is byte-identical (phase 127's pins stay green — no `app/api/sync.py` change)
|
||||||
|
- [ ] the E2E: a mock-LLM folder summary is stored + rendered in the RAG table WITHOUT the forbidden openers
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Task 01 — the instruction nudge (varied openers, marker untouched)
|
||||||
|
|
||||||
|
**Phase:** `155_folder_summary_voice` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`lite` stops opening every folder summary with "This folder contains…" — the instruction gains the variety direction while the E2E mock's keys (the `FOLDER_SUMMARY_MODE` marker + the `FOLDER_HEADER_PREFIX`) stay byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/folder_summaries.py` — `FOLDER_SUMMARY_INSTRUCTION` (L102–108): keep the existing four constraints (1–3 sentence; plain text; no markdown; no invention) and ADD the variety direction (the phase design's A1 wording — the two forbidden openers are the pin): "Open in a varied, natural way — state what the folder is for, the way a person would describe it. Do NOT begin with 'This folder contains' or 'The <folder name> folder contains'." (grammar may be polished; the two forbidden openers and the "varied, natural way" direction must survive).
|
||||||
|
2. `FOLDER_SUMMARY_MODE` (L91) and `FOLDER_HEADER_PREFIX` (`"Folder: "`) — UNTOUCHED (byte-identical — the mock's keys; a unit pin asserts it).
|
||||||
|
3. `app/rag/folder_summaries.py` module docstring — where the instruction's contract is described, name the nudge (cite the phase, don't restate the whole contract).
|
||||||
|
4. `tests/e2e/mock_llm.py` — the CANNED folder-summary reply (the pure function of the request): update its text to the new voice (a natural opener naming the folder — e.g. "Documentation for <folder> …" — NEVER the two forbidden openers). The marker + header keys are untouched; the reply stays a pure function of the request (the same inputs → the same output — the determinism contract).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the instruction pin (the two forbidden openers present as the constraint; the four preserved constraint words present); the marker + header prefix byte-identical pins; the mock's canned-reply shape (a folder's reply names the folder, avoids the forbidden openers, is a pure function of the request — two identical requests → identical replies).
|
||||||
|
- Coverage: **>90%** on the modified module (the pins exercise the changed constant).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the marker + header prefix pins are byte-identical to pre-phase
|
||||||
|
- [ ] the mock's canned folder reply avoids the forbidden openers and stays deterministic
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 02 — `scripts/regen_folder_summaries.py` (one-shot full regeneration)
|
||||||
|
|
||||||
|
**Phase:** `155_folder_summary_voice` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A standalone maintenance entrypoint refreshes EVERY stored folder summary against the (new) instruction in one `uv run` — the live catalogue's 15+ source-root rows all currently carry the formulaic opener, and the sync path's change-gated mode (phase 127) never re-touches unchanged folders, so a one-shot is the only way to re-voice the whole catalogue.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `scripts/regen_folder_summaries.py` — the `scripts/import_docs.py` entrypoint idiom (READ that file first and mirror its exact setup — do not invent a second pattern):
|
||||||
|
- `build_parser()` (argparse) — `--limit N` (the debug passthrough, same meaning as import_docs': cap the walk/generation; the summary line names it) if import_docs' parser has it — match import_docs' actual flags, do not guess;
|
||||||
|
- `main(argv: list[str] | None = None) -> int` + the `if __name__ == "__main__": raise SystemExit(main())` tail (match import_docs' tail);
|
||||||
|
- `get_settings()` + `configure_logging(settings.log_level)` + `configure_debugging()` (the phase-8 debug idiom — `DEBUGPY=1` honored, the AGENTS.md rule 7 contract);
|
||||||
|
- `llm = LLMClient()`; the short-lived DB session (the phase-53 convention — the script's own session commits; the generator only flushes — read import_docs' session handling and mirror it);
|
||||||
|
- the body: `stats = await generate_folder_summaries(session, llm)` — FULL mode (`changed_docs=None` is the default — do NOT pass it), `only_missing=False` (a REGEN — every stored row is refreshed, not just gaps; do NOT pass it either — the defaults), `skip=False` (do not pass);
|
||||||
|
- the cron-safe SINGLE summary line (the importer's idiom — one greppable line: `regen_folder_summaries: folders=<n> …` out of the returned stats dict) on the logger;
|
||||||
|
- exit 0 on success; non-zero + a named stderr line on failure (the fail-loud idiom — a missing admin secret / DB outage names the failure, never a silent partial).
|
||||||
|
2. NO new API surface; NO schema change; NO change to `app/api/sync.py` or the importer (phase 127's change-gated contract is untouched — the script is a consumer of the existing `changed_docs=None` full mode).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the script's wiring (a fake LLM + a fixture catalogue: `generate_folder_summaries` is called with the FULL-mode defaults — assert the call's kwargs; the stats dict surfaces in the summary line; a settings/boot failure exits non-zero with the named line — the app validator idiom).
|
||||||
|
- Integration: a full-mode regeneration over a fixture catalogue upserts EVERY candidate row (text + `updated_at` advance for all of them — a pure regen, no prune: the catalogue is unchanged, so the count-driven prune rule fires for nothing).
|
||||||
|
- Coverage: **>90%** on the script (the wiring pins exercise main()'s paths).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run python -m scripts.regen_folder_summaries` runs the full regeneration against a fixture DB: every candidate row's `updated_at` advances, the single summary line logs, exit 0
|
||||||
|
- [ ] `DEBUGPY=1 uv run python -m scripts.regen_folder_summaries` honors the listener (the debugging idiom — the unit test for `configure_debugging`'s contract, the phase-8 precedent)
|
||||||
|
- [ ] `app/api/sync.py` + `app/rag/importer.py` are untouched (`git diff --name-only` scoped check)
|
||||||
|
- [ ] `uv run pytest -q` green (the script's pins); `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Task 03 — unit pins: the instruction, the mock contract, the script wiring
|
||||||
|
|
||||||
|
**Phase:** `155_folder_summary_voice` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The load-bearing pieces are pinned: the instruction carries the two forbidden openers as its constraint (the four old constraints preserved), the mock's keys are byte-identical, the mock's canned reply is deterministic and opener-free, and the script calls the generator in full mode.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/` — the folder-summaries pin suite (the phase-94/96/127 suites are the precedent — locate `test_folder_summaries.py` or its equivalent and extend it):
|
||||||
|
- `FOLDER_SUMMARY_INSTRUCTION`: the two forbidden openers present as the constraint; the four preserved constraint words (1–3 sentence; plain text; no markdown; no invention) present;
|
||||||
|
- `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX`: byte-identical to the pre-phase values (pin the verbatim strings — the mock contract);
|
||||||
|
- the mock's canned folder reply (in `test_mock_llm.py` or the mock's own suite — the phase's idiom): names the folder, avoids the two forbidden openers, is a pure function of the request (two identical requests → identical replies; different folders → different replies keyed on the header line).
|
||||||
|
2. The script's wiring pins (task 02's list — a new test module beside the import_docs' script pins, mirroring that idiom).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; every pre-existing pin stays green (additions only).
|
||||||
|
- Coverage: the pins exercise the changed constant + the script's paths.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
|
||||||
|
- [ ] the marker/header pins assert the verbatim pre-phase strings
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Task 04 — the dedicated Playwright suite: the new voice, end to end
|
||||||
|
|
||||||
|
**Phase:** `155_folder_summary_voice` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_folder_summary_voice.py`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated E2E proves the voice change end to end: a KB change triggers the folder regeneration under the mock LLM, the STORED summary carries the mock's new-voice canned text (no forbidden opener), and the RAG catalogue renders it.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_folder_summary_voice.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-127/94 suites — read `test_incremental_sync.py` / the folder-summary E2E for the boot + the KB-change fixture, mirror it, do not invent a new one):
|
||||||
|
- **test_regen_summary_avoids_forbidden_openers:** the fixture KB change (a new document under a fixture source folder — the suites' idiom) triggers the sync's folder regeneration (the change-gated path, phase 127) → the stored `folder_summaries` row for that folder equals the mock's canned reply for that folder (a pure function — assert the EXACT canned string) AND does not start with "This folder contains" / "The <folder> folder contains" (the shape pin);
|
||||||
|
- **test_rag_table_renders_the_stored_voice:** the RAG view's folder table (the phase-97 catalogue) shows the stored text in the DESCRIPTION cell for the regenerated folder (the text is user-derived — `textContent` only, the house rule — assert the visible cell text equals the canned reply, truncated display aside: the cell's `title`/full text attribute carries the full string if the display ellipsizes — check the kb-table cell's idiom in the phase-97 suite and assert accordingly);
|
||||||
|
- **test_unchanged_folders_keep_their_rows (phase-127 regression guard):** a KB change in folder A does not touch folder B's stored row (text + `updated_at` unchanged — the phase-127 contract; assert via the API/DB the way the phase-127 suite does).
|
||||||
|
2. The suite is DETERMINISTIC: mock-LLM only (the canned reply is a pure function — no `E2E_REAL_LLM` dependency).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` green in isolation
|
||||||
|
- [ ] the stored-text assertion is an exact match against the mock's canned reply (deterministic)
|
||||||
|
- [ ] the phase-127 regression guard is asserted (unchanged folders keep their rows)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Phase 156 — tables: sticky headers + ellipsis on clipped labels
|
||||||
|
|
||||||
|
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "sticky table headers (RAG/Sources/History — the tables are long; the header scrolls away)" + "At ~1,009px window width the rightmost table columns start clipped with no ellipsis hint on the 'Hidden'/'Edit' labels (the wrapper scrolls fine — just subtle)".
|
||||||
|
**Story:** n/a (audit-derived phase; the table machinery is phase 97's `97_kb_tree_catalog` + phase 86's `86_history_page_width`, both complete)
|
||||||
|
**Context:** `styles.css:: .table-wrap` (L2528–2534: `position: relative; background; border; radius; box-shadow; overflow-x: auto`) wraps `.docs-table` (`width: 100%; min-width: 640px`) — the RAG folder table (the `kb-folders-table` class), the Sources table (`#git-sources-table-wrap`, L3390), and the History table (the phase-50/86 width contract, L5583). The thead already has a TINTED background (L530 comment: "the thead tinted") — the sticky design needs it OPAQUE (rows must not show through on scroll). The clipped cells: the Sources table's `Hidden` checkbox label + `Ignore paths`/`Remove` buttons, the RAG table's `Edit` button, the History table's `Copy`/`Unshare`/`Create link`/`Delete` buttons — today a clipped label hard-cuts (no `text-overflow: ellipsis`). **The sticky constraint (the design's load-bearing fact):** a `position: sticky` `th` sticks to its NEAREST SCROLLING ANCESTOR — with `overflow-x: auto`, the wrapper's computed `overflow-y` is `auto` too, so the wrapper IS the scroll container; but it has no height bound, so it never scrolls vertically — the PAGE does. A sticky header in an unbounded wrapper therefore scrolls away with the page. The header sticks only if the wrapper itself is the vertical scroll region (a bounded `max-height`) — the standard scrollable-table-with-sticky-header pattern. That is the UX trade this phase makes (ASSUMPTION A1): the tables become bounded internal-scroll panels; the page no longer carries their height.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
On the three catalogue tables (RAG folders, Sources, History): the header row stays pinned while the user scrolls the table's own (now bounded) vertical scroll, and a clipped cell label shows `…` instead of a hard cut — the full text remains reachable (title/aria, the established idiom).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `97_kb_tree_catalog` (complete) — the RAG folder table + the `.table-wrap` card.
|
||||||
|
- `86_history_page_width` (complete) — the History table's width contract (the sticky design must keep it — the table stays full-width inside the bounded panel).
|
||||||
|
- `63_unambiguous_document_listing` (complete) — the Sources table's column idiom (the `Hidden` label + per-row actions).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **A1 (owner-locked):** the bounded-panel design — `.table-wrap` gains `max-height` (ASSUMPTION A2: `min(70vh, 64rem)` — the phase picks this; it must keep a 15-row table scrolling while a 3-row one fits without a scrollbar: the `min()` of a viewport fraction and a fixed cap does exactly that) + `overflow: auto` (superseding `overflow-x: auto` — the horizontal scroll contract is UNCHANGED: the table's `min-width: 640px` still triggers the wrapper's horizontal scroll at narrow widths; `overflow: auto` keeps both axes). The `th` sticky: `position: sticky; top: 0; z-index: 1;` on `.table-wrap table thead th` with an OPAQUE background (the existing tint made opaque — `background: var(--surface)` or the existing tint token, whichever the L530 comment names; check the current value and make it non-transparent; do NOT add a new palette variable — B3: the 17-variable contract is untouched, and an all-default theme stays byte-identical in the served CSS — the rule changes apply to BOTH themes equally).
|
||||||
|
- **A3 (owner-locked):** the ellipsis hint — the clipped cell LABELS get `max-width` + `overflow: hidden; text-overflow: ellipsis; white-space: nowrap;` (a `.cell-clip` class or a direct selector on the action-cell text nodes — the executor's choice; the `docs-table`'s existing cell padding is untouched). Coverage: the Sources `Hidden` checkbox label + `Ignore paths` + `Remove`; the RAG `Edit`; the History action buttons (`Copy`/`Unshare`/`Create link`/`Delete`). The FULL text stays reachable: every clipped label's control already carries its accessible name (the `Remove` button's `aria-label` names the full source — the phase-89 idiom; CHECK each of the other controls and add a `title`/`aria-label` where one is missing — the audit: the RAG `Edit` button, the History action buttons, the `Ignore paths` button — add the missing labels in the same task).
|
||||||
|
- **Scope guard:** the CHAT view's `.md-table-wrap` (L539 — the markdown-rendered tables) is a DIFFERENT class — UNTOUCHED (it has no card, no height bound; the design is for the catalogue tables only). The mobile ≤640px contract (PLAN §7.1) is re-verified in the E2E (the hamburger nav + the tables' horizontal scroll at ≤640px still work — the `max-height` must not break the mobile scroll model).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_sticky_header.md` — the bounded-panel `max-height`/`overflow: auto` + the opaque sticky `th` (all three catalogue tables share the class — one rule).
|
||||||
|
2. `02_ellipsis_hints.md` — the clipped-label ellipsis + the missing title/aria audit (the five control families).
|
||||||
|
3. `03_unit_pins.md` — the CSS contract pins (the existing styles-pinning idiom — the `:root`-drift test parses `styles.css`; extend the same suite).
|
||||||
|
4. `04_e2e_table_polish.md` — the dedicated Playwright suite (narrow viewport: the sticky offset + the computed ellipsis style + the mobile contract).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the CSS pins — `.table-wrap` carries `max-height: min(70vh, 64rem)` + `overflow: auto`; `.table-wrap table thead th` carries `position: sticky; top: 0; z-index: 1` + an opaque `background` (parse the stylesheet — the styles-pinning idiom, the `:root`-drift test's precedent); the `.md-table-wrap` rule is byte-UNCHANGED (the scope guard).
|
||||||
|
- Integration: none (CSS-only phase — the server pins stay byte-identical as regression guards).
|
||||||
|
- Coverage: **>90%** on modified code (CSS pinned by the source-level suite + the E2E).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the bounded-panel scroll is a visible UX change — name it where the pages' layout is described, if anywhere).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] all three catalogue tables: after scrolling the wrapper's vertical scroll, the `thead` stays at the panel top (the E2E asserts the `th`'s offsetTop ≈ 0 while the wrapper's scrollTop > 0)
|
||||||
|
- [ ] the horizontal-scroll contract is unchanged (narrow viewport: the table's `min-width: 640px` still scrolls inside the wrapper — the E2E)
|
||||||
|
- [ ] a clipped action label shows `text-overflow: ellipsis` (the E2E asserts the COMPUTED style on a forced-narrow cell — deterministic, no pixel assertions) and its full text is reachable (title/aria present on every clipped control)
|
||||||
|
- [ ] the mobile ≤640px contract holds (the hamburger nav + the table scroll — the E2E)
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Task 01 — the bounded-panel sticky header (one rule, three tables)
|
||||||
|
|
||||||
|
**Phase:** `156_table_ux_polish` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The catalogue tables' header rows stay pinned while the table's own vertical scroll moves — the standard scrollable-table-with-sticky-header pattern, shared by all three tables through the one `.table-wrap` class.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css`:
|
||||||
|
- `.table-wrap` (L2528–2534): `overflow-x: auto` → `overflow: auto` (both axes — the horizontal contract via the table's `min-width: 640px` is unchanged) + `max-height: min(70vh, 64rem)` (the phase design's A2 value). Keep `position: relative` (the sticky's containing context) — the rest of the block (background/border/radius/shadow) is untouched.
|
||||||
|
- `#git-sources-table-wrap` (L3390) — check whether it OVERRIDES the base `.table-wrap` (a height/overflow of its own): if it does, harmonize it (the same `max-height`/`overflow: auto`; do not leave a divergent rule that would make the Sources table unbounded while the others are bounded — the one-rule contract).
|
||||||
|
- the sticky rule: `.table-wrap table thead th { position: sticky; top: 0; z-index: 1; }` + an OPAQUE background. Check the existing thead background (L530 comment: "the thead tinted" — the `.docs-table thead th` rule at L542 is the MARKDOWN table (`md-table-wrap`), a DIFFERENT class — the catalogue tables' thead background is set elsewhere; locate the catalogue thead's current background and make it opaque (the tint token as-is if it is already opaque; otherwise the surface token). NO new palette variable (B3); the rule applies equally under a themed page (B4 — a themed page gets the same sticky behavior, the `#bor-theme` override of the background token handles the color).
|
||||||
|
- the `.md-table-wrap` block (L539) is UNTOUCHED (the scope guard — a pin asserts it).
|
||||||
|
2. Comment the rule in the file's established voice (the phase-97/86/50 comment blocks that describe the scrollable `.table-wrap` — name the sticky + the bounded panel, cite the phase, don't restate the whole contract).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the CSS pins (task 03's suite) — the `max-height` value, `overflow: auto`, the sticky th rule, the opaque background, the `.md-table-wrap` byte-unchanged.
|
||||||
|
- Coverage: **>90%** (CSS pinned by the source-level suite).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the CSS pins pass; the `#git-sources-table-wrap` override is harmonized (no divergent unbounded rule)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 02 — ellipsis on clipped labels + the title/aria audit
|
||||||
|
|
||||||
|
**Phase:** `156_table_ux_polish` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A clipped cell label shows `…` instead of a hard cut, and every clipped label's full text stays reachable (title/aria) — the five control families across the three catalogue tables.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — the ellipsis rule (the phase design's A3): the clipped cell LABELS get `max-width` + `overflow: hidden; text-overflow: ellipsis; white-space: nowrap;`. The executor's choice of selector shape (a `.cell-clip` class added to the label nodes, or direct selectors) — the constraint: the `docs-table` cell padding is untouched, and a cell that FITS shows its full text (the `max-width` must be generous enough that `Edit`/`Hidden`/`Remove` at full width are never clipped — the ellipsis is for the CLIPPED case only; the `max-width` is on the label span, not the cell).
|
||||||
|
2. The label nodes (markup — `frontend/index.html` + `frontend/assets/git-sources.js` + `frontend/assets/history.js` + `frontend/assets/sources.js` — the tables' renderers build their cells with the DOM APIs, the house rule): wrap the clipped labels in the `.cell-clip` span (or the chosen selector's class) — the five families:
|
||||||
|
- the Sources table: the `Hidden` checkbox LABEL (the visible text — the checkbox's own `aria-label` already names the full source, the phase-105 idiom), the `Ignore paths` button text, the `Remove` button text (its `aria-label` names the full source — KEEP it; the visible text clips);
|
||||||
|
- the RAG folder table: the `Edit` button text;
|
||||||
|
- the History table: the `Copy`/`Unshare`/`Create link`/`Delete` button texts (the `Delete` button's `aria-label` already names the chat title, the phase-66 idiom — CHECK the other three and add the missing `aria-label`s: `Copy the share link for <title>` / `Unshare <title>` / `Create a share link for <title>` — the audit's additions).
|
||||||
|
3. The missing title/aria (the audit, one task): for every clipped control that has NO accessible name beyond the visible text, add the `aria-label`/`title` — the `Remove` (Sources) and `Delete` (History) precedents are the idiom; the additions must name the ROW's identity (the source URL / the chat title — the values the renderers already have in hand).
|
||||||
|
4. Comment the change in the file's established voice (cite the phase; the B5 "text + color, never color alone" idiom is already satisfied — the labels are text).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level pins — the renderers' cell builders carry the `.cell-clip` span (or the chosen class) on the five label families; the added `aria-label`s are present on the audited controls (assert the attributes the renderers set — the source-pinning idiom).
|
||||||
|
- Coverage: **>90%** on the modified JS/markup (the pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the five control families carry the ellipsis class (the pin) and the computed style yields `text-overflow: ellipsis` on a forced-narrow viewport (the E2E, task 04)
|
||||||
|
- [ ] every clipped control's full text is reachable: the audit's `aria-label` additions are present (the pin + the E2E)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 03 — CSS contract pins (the styles-pinning idiom)
|
||||||
|
|
||||||
|
**Phase:** `156_table_ux_polish` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The design's load-bearing CSS facts are pinned at the source level — a later "cleanup" cannot silently unstick the header, unbound the panel, or transparent the thead.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/` — the styles-pinning suite (the `:root`-drift test that PARSES `styles.css` is the idiom — the phase-91/92 theme suites extended it; locate the stylesheet-parsing test module and extend it):
|
||||||
|
- `.table-wrap`: `max-height: min(70vh, 64rem)` (the exact value pinned) + `overflow: auto`;
|
||||||
|
- `.table-wrap table thead th`: `position: sticky` + `top: 0` + `z-index: 1` + an OPAQUE `background` (the parse asserts the background is a non-`transparent` token — the exact token is the phase's choice, the OPACITY is the pin);
|
||||||
|
- `#git-sources-table-wrap`: no divergent unbounded override (if the rule exists, it carries the same `max-height`/`overflow: auto` — the pin asserts harmony);
|
||||||
|
- `.md-table-wrap`: byte-UNCHANGED (the scope guard — pin the rule's exact pre-phase text);
|
||||||
|
- the ellipsis rule: the clipped-label selector carries `text-overflow: ellipsis` + `overflow: hidden` + `white-space: nowrap`.
|
||||||
|
2. The `:root`-drift test itself is UNCHANGED (the theme contract is untouched — this phase adds no palette variable; a pin asserts the 17-variable set is byte-identical).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; every pre-existing pin stays green (additions only).
|
||||||
|
- Coverage: the pins exercise the changed CSS.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
|
||||||
|
- [ ] the `.md-table-wrap` pin asserts the verbatim pre-phase rule
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 04 — the dedicated Playwright suite: sticky + ellipsis + mobile
|
||||||
|
|
||||||
|
**Phase:** `156_table_ux_polish` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_table_ux_polish.py`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated E2E proves the visible contracts on the real DOM: the header stays pinned while the table's own scroll moves, a forced-narrow viewport shows the ellipsis computed style on a clipped label with its full text reachable, and the mobile ≤640px contract (hamburger nav + horizontal table scroll) survives the bounded panel.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_table_ux_polish.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-97/86 suites — read one of their E2E files for the boot + the table fixtures, mirror it, do not invent a new one):
|
||||||
|
- **test_sticky_header_on_vertical_scroll (×3 tables, parameterized — RAG folders / Sources / History):** a tall fixture (the suites' idiom — enough rows to exceed the panel's `max-height`); scroll the wrapper (`el.scrollTop = <a large value>`); assert `wrapper.scrollTop > 0` AND the `thead th`'s `offsetTop ≈ 0` relative to the panel (the offset assertion — the Playwright `boundingBox` delta, the standard sticky assertion; NO pixel-tolerance games: the sticky offset is exactly 0 while scrolling).
|
||||||
|
- **test_horizontal_scroll_contract_unchanged:** a narrow viewport (the E2E browser resizes — the suite's idiom) → the table's `min-width: 640px` still scrolls horizontally inside the wrapper (`wrapper.scrollWidth > wrapper.clientWidth` with a table wider than the panel) — the phase-97/86 contract.
|
||||||
|
- **test_ellipsis_on_clipped_label:** force the action column's clip (a narrow viewport that clips the rightmost cells — the audit's ~1,009px case, or a tighter fixture width): assert the clipped label's COMPUTED style `text-overflow === "ellipsis"` (deterministic — the computed style, not a pixel), AND `el.scrollWidth > el.clientWidth` (the label is actually clipped), AND its full text is reachable: the control's `aria-label`/`title` (the audit's additions) contains the row identity (the source URL / the chat title).
|
||||||
|
- **test_mobile_contract:** ≤640px viewport (the PLAN §7.1 contract — the phase-88 suite's idiom): the hamburger nav opens + the catalogue table's horizontal scroll works inside the bounded panel + the `thead` still pins (the mobile regression guard — the `max-height` must not break the mobile scroll model).
|
||||||
|
2. The suite is DETERMINISTIC: fixed fixture row counts (the suites' idiom), computed-style assertions (never pixel/offset tolerance beyond the exact-0 sticky offset), no real-LLM dependency.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` green in isolation
|
||||||
|
- [ ] the sticky assertion is the exact-0 offset while `scrollTop > 0` (all three tables)
|
||||||
|
- [ ] the ellipsis assertion is the computed style + the actual-clip check + the reachable full text
|
||||||
|
- [ ] the mobile contract test passes (the phase-88 idiom)
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Phase 157 — hero suggestion chips cap their display at 90 characters
|
||||||
|
|
||||||
|
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "hero suggestion chips — consider a char cap — long opening questions get hard-truncated mid-word… the chips stretch near full container width before clipping."
|
||||||
|
**Story:** n/a (audit-derived phase; the chip machinery is phase 80's `80_history_suggestion_chips` + phase 104's `104_chip_sizing_question_cap`, both complete)
|
||||||
|
**Context:** `app.js::renderChips(container, items, onSelect)` (L1272–1315) — the ONE shared chip component (the hero `#suggestions` + the deflection "Maybe try" chips). Phase 104's contract (owner-locked 2026-09-12, in-code): the single-line chip CLIPS long questions (CSS), `title` is the hover reveal, `aria-label` is the clipped-case accessible name (the source-chip pattern), and the click submits the FULL `text` (`submitSuggestion(text)` + `onSelect(text, btn)`). Today there is NO display-length cap — a 300-char opening question stretches its chip to near full container width before the CSS clip engages (the audit's observation: the hero's third chip ran the full reading-column width). Phase 104 also capped the COMPOSER (the input) — a different surface, untouched here. The chips' content is the 3 newest saved chats' opening questions (phase 80/103) — owner-derived, `textContent`-only (the house rule), so a display cap is a pure presentation change: the click payload stays full.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The chip's DISPLAYED text caps at 90 characters with a clean `…` (the chip stays compact — no more full-width hero chips), while the `title` hover reveal, the clipped-case `aria-label`, and the click payload all keep the FULL question (phase 104's contract, extended).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `80_history_suggestion_chips` (complete) — the hero chips' content contract (the 3 newest opening questions).
|
||||||
|
- `104_chip_sizing_question_cap` (complete) — the single-line CSS clip + the `title`/`aria-label` idiom this phase EXTENDS (the cap makes the CSS clip a secondary guard, not the primary one).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **The cap (ASSUMPTION A1, owner-locked):** `CHIP_CAP = 90` — a named constant in `app.js` (the module's top, beside the other chip constants). Applied to the DISPLAYED text ONLY in `renderChips`: `btn.textContent = text.length > CHIP_CAP ? text.slice(0, CHIP_CAP).trimEnd() + "…" : text`. `slice(0, 90).trimEnd()` avoids the trailing-space-before-`…` artifact when char 90 is mid-word (the `trimEnd` is the pin). The `title` is UNCHANGED (the full `text` — the hover reveal, phase 104). The `aria-label` logic is UNCHANGED (the clipped-case accessible name is the full `text` — with the cap, a long chip is ALWAYS "clipped" in the visual sense, but phase 104's `scrollWidth > clientWidth` check still governs the aria-label: a 91-char display still overflows its single-line box at typical widths, so the aria-label lands; a SHORT string (< 90) is byte-identical to pre-phase in every respect — the no-cap case is unchanged).
|
||||||
|
- **The click payload (the load-bearing invariant):** `submitSuggestion(text)` + `onSelect(text, btn)` take the FULL `text` (the un-capped value) — UNCHANGED. The cap is display-only: a capped chip sends the full question (the user gets their full question submitted — the phase-80 contract). A unit pin asserts the handler receives the FULL string for a >90-char item.
|
||||||
|
- **Scope:** `renderChips` is shared (hero + Maybe-try) — the cap applies to both (a single code path; the Maybe-try chips are title-derived and short — the cap is a no-op there, and the byte-identity of short strings is the guard). NO other file changes (the CSS clip stays as the secondary guard; the composer's phase-104 cap is a different surface — untouched).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_chip_cap.md` — the `CHIP_CAP` constant + the display-only cap in `renderChips` + the comment in the phase-104 voice.
|
||||||
|
2. `02_unit_pins.md` — the boundary pins (89/90/91), the full-text invariant, the short-string byte-identity.
|
||||||
|
3. `03_e2e_chip_cap.md` — the dedicated Playwright suite.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: source-level pins — the 90-char boundary (an 89-char display is unchanged; a 90-char display is unchanged (the cap is `>`); a 91-char display is `slice(0,90).trimEnd() + "…"` — the exact string pinned); the `title` is the full text in all three cases; the click/`onSelect` payload is the FULL `text` for a >90-char item (the load-bearing invariant); a <90-char item is byte-identical to pre-phase in textContent/title/aria-label (the short-string guard).
|
||||||
|
- Integration: the `/api/suggestions` contract is untouched (no server change — a regression guard only).
|
||||||
|
- Coverage: **>90%** on the modified JS (the pins).
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py`).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the cap is a visible UI change — name it where the onboarding chips are described, if anywhere).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a >90-char opening question renders a ≤91-char display (the cap + the `…`) — the E2E
|
||||||
|
- [ ] clicking the capped chip submits the FULL question (the user bubble shows the full text — the E2E's load-bearing assertion)
|
||||||
|
- [ ] the `title` hover reveal + the clipped-case `aria-label` carry the full text (the E2E)
|
||||||
|
- [ ] a <90-char question is byte-identical to pre-phase (the pin)
|
||||||
|
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
|
||||||
|
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Task 01 — the `CHIP_CAP` display cap in `renderChips`
|
||||||
|
|
||||||
|
**Phase:** `157_chat_chip_cap` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The shared chip component caps its displayed text at 90 characters with a clean `…` — a long opening question no longer stretches its chip to full container width — while the `title`, the `aria-label`, and the click payload keep the full text.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js`:
|
||||||
|
- `const CHIP_CAP = 90;` — the named constant at the module top, beside the other chip constants (the module's established placement — the `suggestions` section header, L1272, is the anchor; place it just above `renderChips`).
|
||||||
|
- `renderChips` (L1296–1315): the display line becomes the cap (the phase design's A1 shape): `btn.textContent = text.length > CHIP_CAP ? text.slice(0, CHIP_CAP).trimEnd() + "…" : text;` — the `title` line (`btn.title = text`) is UNCHANGED (the full text — the hover reveal, phase 104); the `aria-label` logic (L1314: the `scrollWidth > clientWidth` check → `aria-label = text`) is UNCHANGED; the click handler (`submitSuggestion(text)` + `onSelect(text, btn)`) is UNCHANGED (the full `text` — the load-bearing invariant).
|
||||||
|
- the phase-104 comment block (L1298–1301, "the single-line chip clips long questions — `title` is the hover reveal, `aria-label` the clipped-case accessible name") gains the cap sentence in the same voice (name the phase-157 cap as the PRIMARY guard, the CSS clip as the secondary — cite, don't restate the whole contract).
|
||||||
|
2. NO other file changes: the CSS clip rule stays (the secondary guard); the composer's phase-104 cap is a different surface (untouched); the "Maybe try" path reuses `renderChips` automatically (a single code path).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: task 02's pins.
|
||||||
|
- Coverage: **>90%** on the modified JS (the pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the constant + the cap line land exactly as designed (the source pin)
|
||||||
|
- [ ] the `title` / `aria-label` / click-handler lines are byte-unchanged (a `git diff` scoped check)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 02 — unit pins: the 90-char boundary, the full-text invariant, the short-string guard
|
||||||
|
|
||||||
|
**Phase:** `157_chat_chip_cap` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The cap's three contracts are pinned at the source level: the 90-char boundary (display), the full-text invariant (title/aria/click), and the short-string byte-identity (the no-cap case).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/` — the `app.js` source-level pin suite (the phase-80/104 suites are the precedent — locate the chip pin module and extend it):
|
||||||
|
- **the boundary:** an 89-char item → `textContent` byte-identical to the input (no cap); a 90-char item → byte-identical (the cap is `>`, not `≥`); a 91-char item → `slice(0, 90).trimEnd() + "…"` — the EXACT string pinned (pin a 91-char fixture where char 90 is a letter AND one where it is a space — the `trimEnd` case is the artifact the cap avoids);
|
||||||
|
- **the full-text invariant:** for a >90-char item: `title` === the full text; the click handler's payload (the `onSelect` spy) === the full text; `submitSuggestion`'s argument === the full text (the load-bearing assertion — a capped chip sends the FULL question);
|
||||||
|
- **the short-string guard:** a <90-char item → `textContent`/`title` byte-identical to pre-phase + the `aria-label` logic unchanged (the `scrollWidth > clientWidth` check — the DOM-simulation idiom the phase-104 suite uses);
|
||||||
|
- **the Maybe-try path (the shared-component guard):** the cap applies identically through `appendMaybeTry` (a >90-char suggestion there renders capped — a single code path; a short one is byte-identical).
|
||||||
|
2. The `/api/suggestions` payload pin (if the suite pins it — the phase-80 idiom): UNCHANGED (no server change in this phase).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; every pre-existing pin stays green (additions only).
|
||||||
|
- Coverage: the pins exercise the changed branch.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
|
||||||
|
- [ ] the full-text invariant pin asserts the full string (not a length) in `title`, `onSelect`, and `submitSuggestion`
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 03 — the dedicated Playwright suite: the capped chip, the full submit
|
||||||
|
|
||||||
|
**Phase:** `157_chat_chip_cap` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_chat_chip_cap.py`)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An isolated E2E proves the user-facing contract: a >90-char opening question renders a compact capped chip (display ≤ 91 chars ending in `…`), the `title` hover reveal + the clipped-case `aria-label` carry the full text, and clicking the capped chip submits the FULL question (the user bubble shows the full text).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_chat_chip_cap.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-80 suite — read `test_suggestion_chips.py` (or its actual name in `tests/e2e/`) for the saved-chat-seeding idiom, mirror it, do not invent a new one):
|
||||||
|
- **the seed:** a saved chat whose opening question is a >90-char string (the suite builds it — a fixed >90-char question, e.g. 120 chars, so the cap engages deterministically; the saved-chat API the phase-80 suite uses is the seeding idiom) → the hero renders the 3-chip contract (phase 80/103) with the seeded question as a chip.
|
||||||
|
- **test_capped_display:** the chip's `textContent` length ≤ 91 AND ends with `…` AND equals the pinned `slice(0, 90).trimEnd() + "…"` of the seeded question (the exact-string pin — the E2E asserts the byte-for-byte display, not just a length);
|
||||||
|
- **test_full_text_reachable:** the chip's `title` attribute === the full seeded question; the chip's `aria-label` (the clipped case — the 91-char display overflows its single-line box at the E2E viewport) === the full seeded question (the phase-104 idiom);
|
||||||
|
- **test_click_submits_full (the load-bearing assertion):** the chip is clicked → the mock-LLM turn runs → the USER bubble's text === the FULL seeded question (not the capped display — the phase-80 contract: the chip sends the full question; the cap is display-only);
|
||||||
|
- **test_short_chip_unchanged (the regression guard):** a seeded <90-char opening question renders byte-identical to pre-phase (the chip's `textContent` === the question, no `…` appended) — the short-string guard on the live DOM.
|
||||||
|
2. The suite is DETERMINISTIC: the mock-LLM answer (no `E2E_REAL_LLM` dependency); the seeded questions are fixed strings (the pins, not generated lengths).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` green in isolation
|
||||||
|
- [ ] the full-text assertion is byte-for-byte (the `title`, the `aria-label`, and the user bubble all equal the FULL seeded question)
|
||||||
|
- [ ] the short-chip guard passes (a <90-char question is byte-identical to pre-phase)
|
||||||
@@ -0,0 +1,427 @@
|
|||||||
|
# Phase 158 — Chronological thinking / tool-call timeline
|
||||||
|
|
||||||
|
**Source:** owner report (live interactive-browser session, 2026-10-04, shared page
|
||||||
|
`https://brain.reeseapps.com/shared/95dd7a2a-8cda-478b-b694-53936070b9fb` — a real
|
||||||
|
`turbo` turn with 6 tool calls): "the thinking block happens first but LLM sent a
|
||||||
|
normal response, not a thinking response, and then went back to thinking. This is
|
||||||
|
awkward. Then the tool calling happens, and the thinking block reads strangely
|
||||||
|
because the tool calls are shown below the thinking block. … show the thinking /
|
||||||
|
tool call / responses in order, rather than in single big blocks each."
|
||||||
|
**Story:** n/a (owner bug report — live site, 2026-10-04)
|
||||||
|
**Context:** `app/rag/agent.py::run_agent` (yields `thinking` / `content` /
|
||||||
|
`ToolCallPiece` / `ToolResultPiece` in TRUE chronological order per round —
|
||||||
|
thinking → tool call → thinking → … → answer; the SSE mapping in
|
||||||
|
`app/api/chat.py` preserves that order, so NO backend stream change is needed);
|
||||||
|
`frontend/assets/app.js` — `ensureThinkingBlock` (~L1053, ONE `details.thinking`
|
||||||
|
per message, idempotent; phase 109 D15 re-opens it on every later thinking frame),
|
||||||
|
`closeThinkingBlock` (~L1075), `closeToolCalls` (~L1086), `appendToolLine`
|
||||||
|
(~L1136, ONE `details.tool-calls-disclosure` per message — phase 117 D3
|
||||||
|
open-live/fold-at-rest), `armToolLineClock` (~L1479, the phase-87 clock queries
|
||||||
|
`.tool-calls .tool-call:last-child`), the turn handler in `runTurn`
|
||||||
|
(thinking ~L3148 / tool ~L3193 / delta ~L3300 / done ~L3313 branches;
|
||||||
|
`thinkingAcc` ~L3071, `toolAcc` ~L3080, `segRuns` — NEW — module scope beside
|
||||||
|
them; the done save point ~L3355, the `tool_result` branch ~L3389, the
|
||||||
|
stream-drop `finalizeFailedTurn` call ~L3417), `rememberBrainTurn` (L2589,
|
||||||
|
`meta` spread — the pagehide partial save reuses it), `finalizeFailedTurn`
|
||||||
|
(L2983); `renderStoredMessage` (~L2089: the `m.thinking` block + the `m.tools`
|
||||||
|
loop, both rendered collapsed); `frontend/assets/shared.js` — `addThinkingBlock`
|
||||||
|
(L126) + `addToolLines` (L175, the byte-parity copy, rendered closed);
|
||||||
|
`app/schemas.py` — `ChatMessage` (L1103, `extra="forbid"`, phase-83 value caps,
|
||||||
|
the wrap serializer that omits absent image keys), `ToolCall` (L1017),
|
||||||
|
`HistoryTurn` (L58 — the history wire, UNCHANGED); `tests/e2e/mock_llm.py`
|
||||||
|
(`_tool_flow` L1820, `_tool_call_stream` L3813, `_sse_stream` L3727,
|
||||||
|
`compose_thinking` L3589, the trigger-dispatch order in `chat_completions`
|
||||||
|
L3872 — the more-specific-phrase-wins convention); the E2E suites whose pins
|
||||||
|
this phase updates (task 05): `test_turn_progress_loader.py` (the phase-109 D15
|
||||||
|
re-open pin), plus the unit source pins in `test_frontend_feedback.py`,
|
||||||
|
`test_frontend_turn_loader.py`, `test_frontend_tool_states.py`,
|
||||||
|
`test_chat_persistence.py`, `test_shared_page.py`, `test_tool_call_compact.py`,
|
||||||
|
`test_thinking_scroll.py`, `test_big_read_progress.py`, `test_read_range_tool_lines.py`,
|
||||||
|
`test_read_truncation_frontend.py`, `test_search_tool_scope.py`,
|
||||||
|
`test_frontend_failed_turn.py` (each pins `ensureThinkingBlock` / `appendToolLine`
|
||||||
|
/ the clock in different depths — inspect each; ONLY the pins the redesign breaks
|
||||||
|
change, the rest stay green untouched).
|
||||||
|
|
||||||
|
## Bug basis (code-traced, 2026-10-04)
|
||||||
|
The server streams the turn in true chronological order (`run_agent` yields per
|
||||||
|
round: thinking → tool call → thinking → … → answer), but the client LUMPS:
|
||||||
|
`app.js` funnels every thinking frame of the turn into ONE `details.thinking`
|
||||||
|
block (`ensureThinkingBlock` is idempotent — the phase-109 D15 re-open re-opens
|
||||||
|
that same block for each later agent round) and every tool frame into ONE
|
||||||
|
`details.tool-calls-disclosure`. DOM order is therefore always
|
||||||
|
`[ALL thinking] → [ALL tool calls] → [answer]` regardless of the actual order —
|
||||||
|
the owner's "thought everything, then called all the tools, then answered"
|
||||||
|
reading. The "thinking → answer → back to thinking" awkwardness is the D15
|
||||||
|
re-open live: the model emits a short mid-round reply alongside a tool call
|
||||||
|
(React style), the first `delta` closes the one thinking block and the answer
|
||||||
|
starts, then the next round's thinking frame re-opens the SAME block above the
|
||||||
|
tool lines — the answer begins, then thinking resumes on top of it. The persisted
|
||||||
|
`bor.chat.v1` record only stores `thinking` (one concatenated string) + `tools`
|
||||||
|
(one array) — the per-run order is never captured, so the shared page
|
||||||
|
(`shared.js`) renders the same lumped layout, collapsed.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Render each agent turn as a chronological timeline — each CONTIGUOUS run of
|
||||||
|
thinking frames in its own `details.thinking` block, each CONTIGUOUS run of tool
|
||||||
|
calls in its own `details.tool-calls-disclosure` (`Tool call(s) (N)` per run),
|
||||||
|
segments in arrival order — live, on restore, and on the shared page; persist the
|
||||||
|
run order in a new optional `segments` array on the brain record (additive;
|
||||||
|
records without it — all pre-phase — restore byte-identically through the
|
||||||
|
untouched legacy path).
|
||||||
|
|
||||||
|
## Owner decisions (chat, 2026-10-04 — recorded per AGENTS.md rule 3)
|
||||||
|
- **D1 — Per-run timeline.** Each contiguous thinking run = its own Thinking
|
||||||
|
block; each contiguous tool-call run = its own disclosure. Consecutive frames
|
||||||
|
of the same kind extend the current run; the first frame of a new kind closes
|
||||||
|
the previous run and opens the new one.
|
||||||
|
- **D2 — Option B mid-round placement (owner-chosen, shown as a live DOM demo in
|
||||||
|
the owner's browser 2026-10-04).** ALL segments render ABOVE the answer
|
||||||
|
bubble, in arrival order — including runs that start AFTER the first `delta`
|
||||||
|
(the mid-round reply case). The bubble stays ONE element; its content
|
||||||
|
accumulates in arrival order (the mid-round text and the final answer are one
|
||||||
|
streamed text). Rejected: Option A (post-answer segments below the bubble —
|
||||||
|
the bubble grows above its own scratchpad).
|
||||||
|
- **D3 — Plain `Thinking` label on every block (owner-chosen).** The phase-17
|
||||||
|
pinned summary literal stays byte-identical across runs (zero new literals;
|
||||||
|
position distinguishes the runs). The disclosure summary stays the phase-117
|
||||||
|
`Tool call(s) (N)` — now per run.
|
||||||
|
- **D4 — Additive persistence.** The brain record gains an OPTIONAL `segments`
|
||||||
|
array; `thinking` (full concatenated text — feeds the history wire + the
|
||||||
|
per-turn log) and `tools` (all call records) are KEPT (old-client graceful
|
||||||
|
degradation; the phase-113 lesson applied up front). `segments` is omitted when
|
||||||
|
absent (the `undefined`-drops-the-key house pattern) — a no-thinking,
|
||||||
|
no-tools record stays byte-identical to pre-phase.
|
||||||
|
- **D5 — Open live, fold at rest** (extends phase 117 D3 + the phase-17
|
||||||
|
open-while-thinking contract): the active run's block is OPEN while its frames
|
||||||
|
stream; a new run's first frame closes the previous run; the first `delta`,
|
||||||
|
`done`, and the stop/abort settle fold the open runs; the restore path and
|
||||||
|
the shared page render everything COLLAPSED.
|
||||||
|
- **D6 — Server: a validated `segments` field on `ChatMessage`.**
|
||||||
|
`extra="forbid"` means the new key would 422 every saved-chat write without a
|
||||||
|
schema field (the phase-113 lesson: a missing key silently UNSAVED every
|
||||||
|
grounded brain record). The field is a discriminated union (`kind` =
|
||||||
|
`"thinking"` | `"tool"`) with phase-83-style value caps, omitted-when-`None`
|
||||||
|
in the serializer (the phase-50 round-trip contract). No migration (the record
|
||||||
|
is opaque JSONB), no SSE contract change, no history-wire change.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### The `segments` entry shape (D4)
|
||||||
|
```
|
||||||
|
segments: [
|
||||||
|
{ kind: "thinking", text: "<that run's full text>" },
|
||||||
|
{ kind: "tool", name: "…", argument?: "…", offset?: int, limit?: int,
|
||||||
|
path?: "…", truncated?: true, chars_shown?: int, chars_total?: int },
|
||||||
|
…
|
||||||
|
]
|
||||||
|
```
|
||||||
|
- One `thinking` entry per RUN (the run's accumulated text — the same text that
|
||||||
|
streams live into that run's block). One `tool` entry per TOOL CALL (the exact
|
||||||
|
`toolAcc` record shape — the phase-37 `{name, argument}` plus the phase-139
|
||||||
|
`offset/limit` and phase-146 `path` additive fields, and the phase-95
|
||||||
|
`truncated/chars_shown/chars_total` stamp when present).
|
||||||
|
- Built at save time from the run tracker: an EMPTY thinking run stores NOTHING
|
||||||
|
(a zero-char run — e.g. a `BOR_STREAM_THINKING`-suppressed turn — is absent);
|
||||||
|
a turn with no thinking and no tools stores no `segments` key at all.
|
||||||
|
- `thinking` (the full concatenation, for the history wire + the log) and `tools`
|
||||||
|
(every call record, for old clients) are UNCHANGED in the record.
|
||||||
|
|
||||||
|
### `app/schemas.py` (task 01)
|
||||||
|
- `ChatThinkingSegment`: `kind: Literal["thinking"]`, `text: str`
|
||||||
|
(`min_length=1`, `max_length=32_000` — `HistoryTurn`'s cap, phase 83).
|
||||||
|
- `ChatToolSegment`: `kind: Literal["tool"]` + the EXACT `ToolCall` field set
|
||||||
|
(name/argument/offset/limit/path/truncated/chars_shown/chars_total with their
|
||||||
|
existing caps) — one model, no duplication of the field definitions where
|
||||||
|
practical (the docstring cross-references `ToolCall`).
|
||||||
|
- `ChatSegment = Annotated[Union[ChatThinkingSegment, ChatToolSegment],
|
||||||
|
Field(discriminator="kind")]`.
|
||||||
|
- `ChatMessage.segments: list[ChatSegment] | None = Field(default=None,
|
||||||
|
max_length=50)` (the phase-83 cap style — far past any realistic turn: the
|
||||||
|
round cap is 10 → ≤ 20 entries; 50 matches the `tools` cap).
|
||||||
|
- The EXISTING wrap serializer (`_drop_image_when_absent`) gains the rule:
|
||||||
|
`segments is None → pop "segments"` (ABSENT, never `null` — the phase-50
|
||||||
|
contract; a record WITH segments keeps them). `HistoryTurn` and the
|
||||||
|
`history_to_messages` mapper are UNTOUCHED (segments never travel the wire —
|
||||||
|
the concatenated `thinking` travels exactly as today).
|
||||||
|
|
||||||
|
### `frontend/assets/app.js` — the live turn (task 02)
|
||||||
|
- Turn state: `segRuns = []` (module scope, reset in `runTurn` beside
|
||||||
|
`thinkingAcc`/`toolAcc` — an array of run objects
|
||||||
|
`{ kind, text, records }`), `currentRun = null` (turn-local).
|
||||||
|
- **Insertion anchor (D2):** every run element is inserted `before`
|
||||||
|
`body.querySelector(".bubble")` (get-or-create the bubble first, as today).
|
||||||
|
There is no other anchor — segments ALWAYS sit above the bubble, in arrival
|
||||||
|
order. `ensureThinkingBlock`'s old `.tool-calls-disclosure ?? .tool-calls ??
|
||||||
|
.bubble` anchor is REPLACED by this.
|
||||||
|
- **`thinking` frame (per frame):** `settleToolLine()` (unchanged);
|
||||||
|
`thinkingAcc += ev.text` + `sawThinking` + `clearTurnTimeout()` + `if (!wrap)
|
||||||
|
wrap = addMessage("brain","")` + `removeTyping()` (all unchanged). Then the
|
||||||
|
run logic: if `currentRun?.kind === "thinking"` → extend it (re-render
|
||||||
|
`renderMarkdown(run.text)` into THAT block's `.thinking-text` — the existing
|
||||||
|
pre-update pin-state / bottom-follow logic moves verbatim into this branch,
|
||||||
|
now per-block); else → close `currentRun` (if any, `details.open = false`),
|
||||||
|
create `details.thinking` (`open = true`, `<summary>Thinking</summary>` +
|
||||||
|
`.thinking-text`, inserted before the bubble), push `{kind:"thinking",text:"",
|
||||||
|
records:[]}` to `segRuns`, `currentRun` = it, extend.
|
||||||
|
- **`tool` frame (per frame):** the existing guards stay byte-identical
|
||||||
|
(the phase-139 `frameOffset`/`frameLimit`, the phase-146 `framePath`
|
||||||
|
well-formedness; `toolAcc.push(toolRecord)`; `clearTurnTimeout()`; the
|
||||||
|
`toolStatus` label logic; the phase-70 legacy-name mapping). Then the run
|
||||||
|
logic: if `currentRun?.kind === "tool"` → extend that disclosure; else → close
|
||||||
|
`currentRun` (if any), create `details.tool-calls-disclosure` (`open = true`,
|
||||||
|
`summary.tool-calls-summary` + the `div.tool-calls` list `role=list`
|
||||||
|
`aria-label="Tool calls"`, inserted before the bubble), push
|
||||||
|
`{kind:"tool",text:"",records:[]}` to `segRuns`, `currentRun` = it; extend =
|
||||||
|
build the line EXACTLY as today (`appendToolLine` — now with the container
|
||||||
|
passed in: `appendToolLine(container, name, argument, offset, limit, scope)`,
|
||||||
|
the line bytes + the four pinned `line.textContent` label literals + the
|
||||||
|
phase-139 range suffix + the phase-146 scope span UNCHANGED, the summary
|
||||||
|
count updated per append — `Tool call(s) (N)` per run), then
|
||||||
|
`armToolLineClock(wrap, lineEl)` (the NEW signature — task 02's clock fix).
|
||||||
|
- **`armToolLineClock(wrap, lineEl)` (the phase-87 multi-disclosure fix):**
|
||||||
|
store the `lineEl` reference (module scope, reset with the wrap); the interval
|
||||||
|
appends the `.tool-elapsed` suffix to `lineEl` instead of querying
|
||||||
|
`.tool-calls .tool-call:last-child` (that query breaks with multiple
|
||||||
|
disclosures — it would find the last line of the FIRST list). The visible
|
||||||
|
behavior is identical (the suffix on the newest line; `settleToolLine` still
|
||||||
|
removes every `.tool-elapsed` in the wrap + resets the reference).
|
||||||
|
- **`delta` handler:** `settleToolLine()` + `acc += ev.text` + the
|
||||||
|
`thinking→streaming` transition UNCHANGED; replace
|
||||||
|
`closeThinkingBlock(wrap)` + `closeToolCalls(wrap)` with `closeCurrentRun()`
|
||||||
|
(folds `currentRun` if open; idempotent; no-op without a run).
|
||||||
|
- **The other settle-pair sites — `finalizeFailedTurn`'s `if (wrap)` block
|
||||||
|
(L2990–L2991), the STOP path (L3477–L3478), and every remaining
|
||||||
|
`closeThinkingBlock(wrap); closeToolCalls(wrap);` pair (grep — the
|
||||||
|
redesign removes both functions):** `closeAllRuns(wrap)` (closes EVERY
|
||||||
|
open `details.thinking` + `details.tool-calls-disclosure` in the wrap —
|
||||||
|
defensive; by construction at most the last run is open, but a late
|
||||||
|
double-close must be a no-op). The rest of `done` is unchanged.
|
||||||
|
- **Save payloads (the FIVE sites):** `buildSegments(segRuns)` (a helper —
|
||||||
|
maps runs: thinking run → `{kind:"thinking",text}` when non-empty; tool run
|
||||||
|
→ one `{kind:"tool",…record}` per record, in order) — the `done` save point
|
||||||
|
(~L3355); the `finalizeFailedTurn` parameter + its `if (wrap)` saved record
|
||||||
|
(the two call sites — stream-drop ~L3431, network-error ~L3519 — pass
|
||||||
|
`buildSegments(segRuns)`); the STOP path's save meta (L3487–L3491,
|
||||||
|
`stopped: true`); the pagehide partial (L3642–L3650 — `segRuns` is module
|
||||||
|
scope, readable there). All include `segments` non-empty only. The
|
||||||
|
zero-frame fallback save needs nothing (zero frames → no runs → no key).
|
||||||
|
The `thinking`/`tools` keys are UNCHANGED at every site.
|
||||||
|
- The UI state machine, `#send-status` copy, the 300 s pre-token guard, the
|
||||||
|
stream-drop guard, and the `thinking-without-answer` fallback are ALL
|
||||||
|
UNCHANGED (§7.4 never-stale intact). `BOR_STREAM_THINKING=0` needs no client
|
||||||
|
change (no thinking frames arrive → no thinking runs).
|
||||||
|
|
||||||
|
### `frontend/assets/app.js` — the restore path (task 03)
|
||||||
|
- `renderStoredMessage`: after the bubble exists, IF
|
||||||
|
`Array.isArray(m.segments) && m.segments.length` → `restoreTimeline(wrap,
|
||||||
|
m.segments)` and SKIP the legacy `m.thinking` block + the `m.tools` loop
|
||||||
|
(segments take precedence — they carry the same data in order); ELSE the
|
||||||
|
legacy path is BYTE-IDENTICAL (a pre-phase record — or a `BOR_STREAM_THINKING=0`
|
||||||
|
turn saved with the old client — restores exactly as today).
|
||||||
|
- `restoreTimeline`: iterate `m.segments` in order; group CONSECUTIVE
|
||||||
|
`kind:"tool"` entries into ONE `details.tool-calls-disclosure` (created
|
||||||
|
CLOSED — D5; summary `Tool call(s) (N)`; each line through the SAME
|
||||||
|
`appendToolLine` container-signature path with the EXISTING tOffset/tLimit/tPath
|
||||||
|
guards + the existing `appendTruncatedNote` call for `t.truncated` records —
|
||||||
|
the current loop's guards move verbatim inside the grouping); a `kind:"thinking"`
|
||||||
|
entry → its OWN `details.thinking` (created CLOSED; `.thinking-text` =
|
||||||
|
`renderMarkdown(entry.text)` — the restore convention stays escape-first);
|
||||||
|
unknown `kind` values are SKIPPED (never render from storage — textContent
|
||||||
|
only, the house rule). All elements inserted before the bubble.
|
||||||
|
|
||||||
|
### `frontend/assets/shared.js` — the shared page (task 03)
|
||||||
|
- `addSegments(wrap, segments)` — the shared-page parity copy of
|
||||||
|
`restoreTimeline` (pure render, always COLLAPSED — D5): the same grouping,
|
||||||
|
the same line bytes (the four label literals + the phase-95 truncation note
|
||||||
|
+ the phase-139/146 suffixes, already parity-pinned in `addToolLines`), the
|
||||||
|
same summary count; the `addThinkingBlock`/`addToolLines` legacy path is
|
||||||
|
BYTE-IDENTICAL when the record has no `segments`. The record-dispatch site
|
||||||
|
(~L456): `if (Array.isArray(m.segments) && m.segments.length)
|
||||||
|
addSegments(wrap, m.segments); else { the existing addThinkingBlock +
|
||||||
|
addToolLines }`.
|
||||||
|
|
||||||
|
### `tests/e2e/mock_llm.py` (task 04)
|
||||||
|
- `THINK_TOOLS_TRIGGER = "think out loud and use your tools"` — checked in the
|
||||||
|
streaming branch BEFORE the `_tool_flow` dispatch (the more-specific-phrase-wins
|
||||||
|
convention — the phrase contains both `think out loud` and `use your tools`,
|
||||||
|
so it must not fall through to the plain answer path; no other existing
|
||||||
|
trigger phrase is a substring of it, and no existing E2E question carries it —
|
||||||
|
verify both before wiring).
|
||||||
|
- `_thinking_tools_flow(body)` — stateless, shaped like `_tool_flow` (gated on
|
||||||
|
the `tools` parameter + the `<tools>` section, like the other marker flows):
|
||||||
|
- request 1 (tools offered, no tool results yet): `thinking` =
|
||||||
|
`"Step 1: list the documents to find the reference server config."` →
|
||||||
|
`_tool_call_stream("ls", {}, "call_0", thinking=thinking)`;
|
||||||
|
- request 2 (a `tool`-role top-level source listing — the agent's `ls` tree
|
||||||
|
format, no file lines yet): `"Step 2: drill into the first source."` → the
|
||||||
|
`_drill_target` scoped `ls` (id `call_1`), with thinking;
|
||||||
|
- request 3 (a `tool`-role folder listing with file lines — parse the FIRST
|
||||||
|
file line's labeled `source:` / `path:` fields, the phase-63/70 format):
|
||||||
|
`"Step 3: read the file line's document."` → `read` on the JOINED
|
||||||
|
combined `source/path` (id `call_2`), with thinking;
|
||||||
|
- request 4 (a `tool`-role read result — content starting with the agent's
|
||||||
|
`"Document <source/path>:"` header): `"Step 4: compose the answer from the
|
||||||
|
read document."` → the phase-37 answer convention: `Read <source/path>.
|
||||||
|
<first 80 chars of the read document's content>` (content stream, with
|
||||||
|
thinking ahead of it).
|
||||||
|
- `_tool_call_stream(name, arguments, call_id, thinking="")` — the optional
|
||||||
|
`thinking` param: when non-empty, yield `reasoning_content` chunks FIRST
|
||||||
|
(the 12-char split — the `_sse_stream` convention), then the existing
|
||||||
|
`tool_calls` frames + `finish_reason: "tool_calls"` (unchanged). Default
|
||||||
|
`""` → every pre-phase 3-arg caller streams BYTE-IDENTICAL.
|
||||||
|
- Docstring: the new trigger entry (the phase-158 flow) — the mock stays
|
||||||
|
mock-only by design (`E2E_REAL_LLM` note unchanged).
|
||||||
|
|
||||||
|
### `tests/e2e/test_thinking_timeline.py` (task 04 — dedicated suite, run in isolation)
|
||||||
|
The conftest `app_server`/`page`/`seeded_kb` fixtures + the phase-37 admin
|
||||||
|
login + the trigger question (the KB must be seeded so request 4's read result
|
||||||
|
exists — mirror `test_agent_document_tools.py`). Six tests:
|
||||||
|
1. **`test_timeline_order_after_completed_turn`** — after `done`: the brain
|
||||||
|
wrap's `.msg-body` children before the bubble are EXACTLY
|
||||||
|
`[details.thinking, details.tool-calls-disclosure, details.thinking,
|
||||||
|
details.tool-calls-disclosure, details.thinking, details.tool-calls-disclosure,
|
||||||
|
details.thinking]` (7 elements); the four `.thinking-text`s carry DISTINCT
|
||||||
|
`Step 1`…`Step 4` markers (distinctness = the runs are separate blocks, not
|
||||||
|
one lump); each disclosure summary is `Tool call (1)`; the answer bubble
|
||||||
|
carries the `Read <path>` echo.
|
||||||
|
2. **`test_runs_fold_at_rest`** — after `done`: NO `.thinking` /
|
||||||
|
`.tool-calls-disclosure` has the `open` attribute; the summaries are
|
||||||
|
visible (D5).
|
||||||
|
3. **`test_live_runs_open_while_streaming`** — during the turn (an
|
||||||
|
`expect.poll` predicate over the mock's 0.1 s pacing window): at least one
|
||||||
|
`.thinking` or `.tool-calls-disclosure` is `open` while the answer bubble is
|
||||||
|
still empty (the D5 live-open contract — the live feedback the owner keeps).
|
||||||
|
4. **`test_restored_timeline_and_record`** — RELOAD (same context): the
|
||||||
|
restored message renders the same 7-element timeline, all closed; the
|
||||||
|
localStorage `bor.chat.v1` record for that brain turn carries a `segments`
|
||||||
|
array of EXACTLY 7 entries in the pattern
|
||||||
|
`thinking, tool, thinking, tool, thinking, tool, thinking` (4 thinking runs
|
||||||
|
+ 3 tool entries).
|
||||||
|
5. **`test_shared_page_timeline`** — the phase-51 save + share flow, a FRESH
|
||||||
|
context on `/shared/<token>`: the same 7-element timeline renders, all
|
||||||
|
collapsed (the `shared.js` parity — `addSegments`).
|
||||||
|
6. **`test_no_thinking_turn_single_disclosure`** — a plain `use your tools`
|
||||||
|
question (NO thinking trigger): the turn renders EXACTLY ONE
|
||||||
|
`.tool-calls-disclosure` (3 lines, `Tool calls (3)`) and ZERO `.thinking`
|
||||||
|
(the run-grouping contract: no thinking frames → one tool run — the
|
||||||
|
phase-117 behavior for no-thinking turns is preserved).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_schema_segments.md` — the discriminated `ChatSegment` models + the
|
||||||
|
`ChatMessage.segments` field + the serializer omission + the integration
|
||||||
|
contract (accept / caps / byte-identical round-trip / shared read).
|
||||||
|
2. `02_live_timeline.md` — the per-run live rendering (run tracker, D2 anchor,
|
||||||
|
the clock re-target, the settle sites, the save payloads) + the JS source
|
||||||
|
unit pins.
|
||||||
|
3. `03_restore_and_shared.md` — the `restoreTimeline` dispatch in
|
||||||
|
`renderStoredMessage` + the `shared.js` `addSegments` parity + the unit pins.
|
||||||
|
4. `04_mock_and_e2e.md` — the mock `THINK_TOOLS_TRIGGER` flow + the dedicated
|
||||||
|
`test_thinking_timeline.py` story suite, run in isolation.
|
||||||
|
5. `05_regression_update_and_commit.md` — the phase-109 suite pin update + the
|
||||||
|
remaining unit-pin updates, the full gate, the documentation pass, the
|
||||||
|
atomic commit, the phase move.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (new): `tests/unit/test_thinking_timeline.py` — the source-level pins:
|
||||||
|
the per-run creation (a NEW `details.thinking` when the run kind changes,
|
||||||
|
insert-before-bubble anchor — D2), the plain `Thinking` literal (D3), the
|
||||||
|
per-run disclosure + the `Tool call(s) (N)` per-run count, the
|
||||||
|
`armToolLineClock` reference (the `:last-child` query is GONE), the
|
||||||
|
`segRuns` → `buildSegments` mapping (empty-run skip, the tool-record shape),
|
||||||
|
the `delta` `closeCurrentRun` + the settle-pair sites → `closeAllRuns`, the
|
||||||
|
FIVE save-payload sites (done / finalizeFailedTurn + its 2 call sites /
|
||||||
|
stop path / pagehide), the restore dispatch (segments vs legacy — the
|
||||||
|
legacy path byte-identical), the `shared.js` `addSegments` parity, and
|
||||||
|
`innerHTML` ABSENT from the restore/shared builders (the house rule).
|
||||||
|
- Integration: `tests/integration/test_chats_api.py` (extended) — a record WITH
|
||||||
|
segments (both kinds, every additive tool field) is accepted and round-trips
|
||||||
|
through create/get; `segments` > 50 → 422; a thinking-segment text > 32 000 →
|
||||||
|
422; an unknown segment `kind` → 422 (`extra="forbid"` intact — a stray
|
||||||
|
`<b>`-ish key still 422s); a record WITHOUT `segments` round-trips with the
|
||||||
|
key ABSENT (never `null` — the phase-50 byte-identical contract, pinned);
|
||||||
|
the shared read (`/api/chats/<id>/shared` or the public read path) carries
|
||||||
|
segments (the public shape).
|
||||||
|
- Existing unit suites MUST stay green: every pin NOT broken by the redesign
|
||||||
|
(the tool-line label literals, the phase-117 deboxed CSS, the phase-87
|
||||||
|
`.tool-elapsed` behavior, the phase-139/146 suffix pins, the emoji guard,
|
||||||
|
the `test_tool_call_compact.py` pins — a no-thinking turn is one run, so the
|
||||||
|
phase-117 E2E contract is UNCHANGED).
|
||||||
|
- E2E: `tests/e2e/test_thinking_timeline.py` (new — the six tests above), green
|
||||||
|
**in isolation** (`uv run pytest tests/e2e/test_thinking_timeline.py -v
|
||||||
|
--no-cov`), mock-only.
|
||||||
|
- Regression E2E (task 05, in isolation): `test_thinking_display.py` (a
|
||||||
|
no-tools thinking turn is ONE run — the suite's contract is unchanged),
|
||||||
|
`test_tool_call_compact.py` (unchanged — no-thinking single-run),
|
||||||
|
`test_turn_progress_loader.py` (UPDATED in task 05 — the D15 pin → the
|
||||||
|
per-run contract), `test_big_read_progress.py` (the clock's visible behavior
|
||||||
|
is identical — must stay green UNCHANGED), `test_agent_document_tools.py`,
|
||||||
|
`test_share_chat.py`, `test_chat_persistence.py`, `test_smoke.py`.
|
||||||
|
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` **>90%** on
|
||||||
|
`app/` (task 01's schema code is unit/integration-covered; the rest is held
|
||||||
|
by the suite).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- The phase is "Complete" only when the **entire** project test suite passes
|
||||||
|
100% (every test in the project, not just this phase's).
|
||||||
|
|
||||||
|
## Documentation & Consistency
|
||||||
|
- The final pass reviews `README.md` and `AGENTS.md` in full against the
|
||||||
|
project as it stands after this phase — fixing every inconsistency found,
|
||||||
|
**including pre-existing ones not caused by the phase**.
|
||||||
|
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are
|
||||||
|
user-visible (the timeline is a visible UI change — name it where the chat
|
||||||
|
thinking/tool-call display is described, if anywhere).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A completed tool+thinking turn renders `Thinking → Tool call (1) →
|
||||||
|
Thinking → Tool call (1) → … → answer` — the E2E's 7-element order pin
|
||||||
|
(task 04 test 1); the runs fold at rest (test 2) and stay open while
|
||||||
|
streaming (test 3).
|
||||||
|
- [ ] A reload restores the same timeline + the record's `segments` array
|
||||||
|
(test 4); a fresh context on `/shared/<token>` renders the same timeline
|
||||||
|
(test 5).
|
||||||
|
- [ ] A no-thinking tool turn is ONE disclosure, 3 lines (test 6) — the
|
||||||
|
phase-117 contract is preserved; `test_tool_call_compact.py` +
|
||||||
|
`test_thinking_display.py` + `test_big_read_progress.py` green in
|
||||||
|
isolation UNCHANGED.
|
||||||
|
- [ ] A pre-phase saved record (no `segments`) restores byte-identically
|
||||||
|
(the integration pin + the E2E regression suites).
|
||||||
|
- [ ] `uv run pytest` green (100% of the project suite);
|
||||||
|
`uv run pytest --cov=app --cov-report=term-missing` > 90%;
|
||||||
|
`tests/e2e/test_thinking_timeline.py` green in isolation;
|
||||||
|
`uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`,
|
||||||
|
`frontend/assets/shared.js`, `tests/`, and the phase files.
|
||||||
|
- [ ] README.md and AGENTS.md are accurate and consistent with the whole
|
||||||
|
project (verified in the final pass).
|
||||||
|
- [ ] ONE owner-directed exception to "no behavior change in completed phases":
|
||||||
|
the phase-109 D15 re-open contract is REPLACED by the per-run blocks
|
||||||
|
(owner direction 2026-10-04 — recorded above); `test_turn_progress_loader.py`'
|
||||||
|
pin is updated in task 05 to the per-run contract (two thinking blocks,
|
||||||
|
each carrying its own round, open in its live window, folded at rest).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` commit (below); the phase dir moved to
|
||||||
|
`.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ app/schemas.py frontend/ tests/ README.md && git commit --no-gpg-sign -m "feat(ui): render thinking and tool runs as a chronological timeline (per-run blocks, segments persistence)"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **The SSE contract is UNCHANGED (A15 intact).** The frames already arrive in
|
||||||
|
true chronological order (`run_agent`'s yield order — verified in code); this
|
||||||
|
phase is client-grouping + record persistence only. `app/rag/` is untouched.
|
||||||
|
- **D1–D6 (owner, 2026-10-04)** — the per-run timeline, Option B placement,
|
||||||
|
the plain `Thinking` label, the additive `segments` record, open-live/
|
||||||
|
fold-at-rest, and the validated `ChatMessage.segments` field (the
|
||||||
|
`extra="forbid"` boundary preserved with phase-83-style caps).
|
||||||
|
- **A10 history wire UNCHANGED** — segments never travel to the LLM (the
|
||||||
|
concatenated `thinking` travels as `reasoning_content` exactly as today;
|
||||||
|
`HistoryTurn` + the mapper are untouched).
|
||||||
|
- **A13 UNCHANGED** — no migration (the record is opaque JSONB; the schema
|
||||||
|
field is validation at the boundary only).
|
||||||
|
- **A16 honoured** — one new story E2E suite + the task-05 regression updates;
|
||||||
|
mock-only by design. **A17 honoured** — one atomic `--no-gpg-sign` commit.
|
||||||
|
- **The phase-87 clock re-target** (the `:last-child` query → the tracked
|
||||||
|
reference) is a bug fix in service of D1; its visible behavior (the suffix on
|
||||||
|
the newest line) is byte-identical — `test_big_read_progress.py` stays green
|
||||||
|
unchanged. No other anchor changed.
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Task 01 — the validated `ChatMessage.segments` field (server)
|
||||||
|
|
||||||
|
**Phase:** `158_thinking_timeline` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Saved-chat writes accept the new optional `segments` array (a discriminated
|
||||||
|
`thinking`/`tool` union with phase-83-style caps), omit it when absent so
|
||||||
|
pre-phase records round-trip byte-identically, and reject malformed values at
|
||||||
|
the `extra="forbid"` boundary.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py`:
|
||||||
|
- `ChatThinkingSegment` (a `BaseModel`, beside `ToolCall`): `kind:
|
||||||
|
Literal["thinking"]`, `text: str = Field(min_length=1, max_length=32_000)`
|
||||||
|
(`HistoryTurn`'s cap, phase 83 — docstring cross-reference).
|
||||||
|
- `ChatToolSegment` (a `BaseModel`): `kind: Literal["tool"]` + the EXACT
|
||||||
|
`ToolCall` field set (name / argument / offset / limit / path / truncated /
|
||||||
|
chars_shown / chars_total with their existing types and caps — mirror, do
|
||||||
|
not weaken; docstring cross-references `ToolCall` as the shape source).
|
||||||
|
- `ChatSegment = Annotated[Union[ChatThinkingSegment, ChatToolSegment],
|
||||||
|
Field(discriminator="kind")]`.
|
||||||
|
- `ChatMessage.segments: list[ChatSegment] | None = Field(default=None,
|
||||||
|
max_length=50)` — with the phase-83 voice comment (the cap sizing: far
|
||||||
|
past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches
|
||||||
|
the `tools` cap).
|
||||||
|
- The EXISTING wrap serializer (`_drop_image_when_absent`) gains the
|
||||||
|
omission rule: after the two image pops, `if self.segments is None:
|
||||||
|
data.pop("segments", None)` — ABSENT, never `null` (the phase-50
|
||||||
|
round-trip contract; the phase-122 `SourceRef.image_url` omission
|
||||||
|
precedent, the phase-113 lesson applied up front: the field EXISTS so
|
||||||
|
the boundary does NOT 422 the done-time auto-save carrying it).
|
||||||
|
- The `ChatMessage` docstring gains the phase-158 paragraph (the shape,
|
||||||
|
the omission rule, the `extra="forbid"` boundary) in the established
|
||||||
|
voice; `HistoryTurn` and everything in `app/rag/` is UNTOUCHED (the
|
||||||
|
history wire stays `{who, text, thinking}`).
|
||||||
|
2. NO migration, NO router change (`app/api/chats.py` validates through
|
||||||
|
`ChatMessage` — `m.model_dump()` + `ChatMessage.model_validate(m)` already
|
||||||
|
carry the new field end to end, including the public shared read
|
||||||
|
`SharedChatOut.messages`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration (extend `tests/integration/test_chats_api.py` — the file already
|
||||||
|
pins the brain-record contract, phase 83/113/120/123/124):
|
||||||
|
- a record WITH `segments` (one `thinking` entry + two `tool` entries
|
||||||
|
carrying every additive field: `offset/limit`, `path`,
|
||||||
|
`truncated/chars_shown/chars_total`) is accepted by `POST /api/chats` and
|
||||||
|
round-trips through `GET` (the shape preserved, entry order preserved);
|
||||||
|
- `segments` with 51 entries → 422; a thinking-segment `text` of 32 001
|
||||||
|
chars → 422; a segment with `kind: "bogus"` → 422; a record with a stray
|
||||||
|
top-level key (e.g. `"foo": 1`) still → 422 (the `extra="forbid"`
|
||||||
|
boundary intact);
|
||||||
|
- a record WITHOUT `segments` (a pre-phase brain record, the full phase-113
|
||||||
|
shape) round-trips with the key ABSENT from the served JSON (never
|
||||||
|
`null` — the byte-identity pin, the phase-50 contract);
|
||||||
|
- the shared read path (the public read endpoint's `messages`) carries the
|
||||||
|
segments of a saved record with them (the public shape, the phase-51
|
||||||
|
contract).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (the schema code is
|
||||||
|
exercised by the integration tests).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the three models + the field + the serializer rule land exactly as
|
||||||
|
designed (a source check)
|
||||||
|
- [ ] the new integration tests are green; the EXISTING
|
||||||
|
`test_chats_api.py` tests stay green unchanged
|
||||||
|
- [ ] `uv run pytest tests/integration -q` green; `uv run ruff check . && uv
|
||||||
|
run pyright` clean
|
||||||
|
- [ ] no change in `app/rag/`, no migration, no router change (a `git diff
|
||||||
|
--stat` scoped check)
|
||||||
@@ -0,0 +1,195 @@
|
|||||||
|
# Task 02 — the per-run live timeline in `app.js`
|
||||||
|
|
||||||
|
**Phase:** `158_thinking_timeline` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Replace the single-block/single-disclosure turn rendering with the per-run
|
||||||
|
timeline: each contiguous thinking run streams into its own `details.thinking`
|
||||||
|
block, each contiguous tool-call run into its own
|
||||||
|
`details.tool-calls-disclosure`, all segments ABOVE the answer bubble in
|
||||||
|
arrival order (D2); the runs open live and fold at rest (D5); the phase-87
|
||||||
|
clock is re-targeted to a tracked line reference; the save payloads persist
|
||||||
|
the `segments` array (D4).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — turn state (the `runTurn` locals / module scope,
|
||||||
|
~L3065–L3080):
|
||||||
|
- `segRuns = []` — MODULE scope (beside `thinkingAcc`, which the pagehide
|
||||||
|
partial reads), reset to `[]` in `runTurn` with the other turn locals.
|
||||||
|
Entries: `{ kind: "thinking", text: "", records: [] }` /
|
||||||
|
`{ kind: "tool", text: "", records: [] }` (the tool entry's `records`
|
||||||
|
holds the `toolAcc` record objects — the SAME objects `toolAcc` holds,
|
||||||
|
so the phase-95 `tool_result` stamp (`t.truncated = true; t.chars_shown
|
||||||
|
= …; t.chars_total = …`, ~L3398–L3404) flows into both automatically —
|
||||||
|
NO new wiring).
|
||||||
|
- `currentRun = null` — turn-local (the active run's element + kind).
|
||||||
|
2. `frontend/assets/app.js` — the run builders (replace `ensureThinkingBlock`
|
||||||
|
+ the single-container logic in `appendToolLine`):
|
||||||
|
- `ensureThinkingRun(wrap)` — returns the CURRENT thinking run's
|
||||||
|
`details.thinking` if `currentRun?.kind === "thinking"`; else closes
|
||||||
|
`currentRun` (if any — `open = false`), creates a NEW
|
||||||
|
`details.thinking` (`open = true`, `<summary>Thinking</summary>` — the
|
||||||
|
phase-17 literal, D3 — built via the existing `innerHTML` template that
|
||||||
|
`ensureThinkingBlock` uses, OR createElement/textContent — either is fine,
|
||||||
|
but the RESTORE path's builder stays createElement/textContent (the house
|
||||||
|
rule pins `innerHTML` ABSENT from the restore/shared builders — see task
|
||||||
|
03)), inserts it `before body.querySelector(".bubble")` (the bubble
|
||||||
|
get-or-created first), pushes a fresh `{kind:"thinking",…}` to `segRuns`,
|
||||||
|
sets `currentRun`.
|
||||||
|
- `ensureToolRun(wrap)` — the same, for
|
||||||
|
`details.tool-calls-disclosure` (`open = true`,
|
||||||
|
`summary.tool-calls-summary` + the `div.tool-calls` list `role=list`
|
||||||
|
`aria-label="Tool calls"` — the phase-117 structure, now per run).
|
||||||
|
- `appendToolLine` — the signature becomes
|
||||||
|
`appendToolLine(container, name, argument, offset = null, limit = null,
|
||||||
|
scope = null)` (the CALLER passes the run's disclosure — no more
|
||||||
|
`body.querySelector(".tool-calls-disclosure")` single-container
|
||||||
|
derivation). EVERYTHING else in the function is BYTE-IDENTICAL: the
|
||||||
|
`.tool-calls` list get-or-create, the line bytes, the four pinned
|
||||||
|
`line.textContent` label literals, the phase-139 range suffix, the
|
||||||
|
phase-146 scope span, the `Tool call(s) (N)` count.
|
||||||
|
- `closeCurrentRun()` — folds `currentRun` (if any, `open = false`);
|
||||||
|
idempotent; no-op without a run.
|
||||||
|
- `closeAllRuns(wrap)` — closes EVERY open `details.thinking` +
|
||||||
|
`details.tool-calls-disclosure` in the wrap (defensive; the `done` +
|
||||||
|
stop/abort settle sites).
|
||||||
|
3. `frontend/assets/app.js` — the turn-handler branches:
|
||||||
|
- the `thinking` branch (~L3148): keep `settleToolLine()`,
|
||||||
|
`thinkingAcc += ev.text || ""`, `sawThinking`, `clearTurnTimeout()`,
|
||||||
|
`if (!wrap) wrap = addMessage("brain", "")`, `removeTyping()` — all
|
||||||
|
unchanged. REPLACE the `ensureThinkingBlock` + `block.open = true`
|
||||||
|
(phase-109 D15 re-open — SUPERSEDED by D1) with
|
||||||
|
`const block = ensureThinkingRun(wrap);` + the run's text extension:
|
||||||
|
the run's `text` accumulates `ev.text`; the pre-update pin-state /
|
||||||
|
bottom-follow logic (the `isThinkingNearBottom` capture +
|
||||||
|
`scrollTop = scrollHeight`, ~L3166–L3189) moves VERBATIM into this
|
||||||
|
branch, now operating on THIS block (per-block — the existing comment
|
||||||
|
block stays, updated to name the per-run placement).
|
||||||
|
- the `tool` branch (~L3193): keep the phase-139/146 guards,
|
||||||
|
`toolAcc.push(toolRecord)`, `clearTurnTimeout()`, the wrap
|
||||||
|
get-or-create, and the `toolStatus` label logic UNCHANGED. REPLACE
|
||||||
|
`appendToolLine(wrap, …)` + `armToolLineClock(wrap)` with:
|
||||||
|
`const disc = ensureToolRun(wrap); const line =
|
||||||
|
appendToolLine(disc, name, argument, frameOffset, frameLimit, framePath);`
|
||||||
|
(the function returns the appended `line` element — a one-line change,
|
||||||
|
the function already has it in scope) +
|
||||||
|
`armToolLineClock(wrap, line)`. The run record is the SAME `toolRecord`
|
||||||
|
object pushed to `toolAcc` (push it into the run's `records` too — or
|
||||||
|
derive the run's `records` from `segRuns` at save time via the call
|
||||||
|
order — the executor picks ONE, but the phase-95 stamp must reach the
|
||||||
|
`segments` payload: the simplest is the run's `records.push(toolRecord)`
|
||||||
|
alongside the `toolAcc.push(toolRecord)`).
|
||||||
|
- the `delta` branch (~L3300): `settleToolLine()` + `acc += …` + the
|
||||||
|
`thinking→streaming` transition UNCHANGED; replace
|
||||||
|
`closeThinkingBlock(wrap)` + `closeToolCalls(wrap)` with
|
||||||
|
`closeCurrentRun()`.
|
||||||
|
- the THREE remaining settle-pair sites — `finalizeFailedTurn`'s
|
||||||
|
`if (wrap)` block (L2990–L2991), the STOP path (L3477–L3478), and any
|
||||||
|
other `closeThinkingBlock(wrap); closeToolCalls(wrap);` pair (grep for
|
||||||
|
the pair — the redesign removes both functions): replace the pair with
|
||||||
|
`closeAllRuns(wrap);`.
|
||||||
|
- `removeTyping()` / the typing-indicator / the `#send-status` copy / the
|
||||||
|
UI state machine / the 300 s guard / the stream-drop guard / the
|
||||||
|
`thinking-without-answer` fallback: ALL UNCHANGED (§7.4 intact).
|
||||||
|
4. `frontend/assets/app.js` — the clock (the phase-87 re-target, ~L1479):
|
||||||
|
- `armToolLineClock(wrap, lineEl)` — the NEW second param: store `lineEl`
|
||||||
|
in a module-scope `toolLineEl` (reset to `null` with `toolLineWrap` in
|
||||||
|
`stopToolLineClock` and on re-arm); the interval body uses `toolLineEl`
|
||||||
|
(null-guarded — the wrap-reset no-op stays) instead of
|
||||||
|
`toolLineWrap?.querySelector?.(".tool-calls .tool-call:last-child")`.
|
||||||
|
The suffix element + the `(Ns)` text + the `TOOL_LINE_ELAPSED_AFTER_MS`
|
||||||
|
gate are UNCHANGED.
|
||||||
|
- `settleToolLine()` — the `querySelectorAll(".tool-elapsed")` removal stays
|
||||||
|
(it clears every line in the wrap — correct with multiple disclosures);
|
||||||
|
add `toolLineEl = null` alongside.
|
||||||
|
- the comment block updated to name the re-target (the multi-disclosure
|
||||||
|
fix, phase 158 D1).
|
||||||
|
5. `frontend/assets/app.js` — the save payloads (D4) — the FIVE sites:
|
||||||
|
- `buildSegments(segRuns)` — a helper (beside the turn state): maps the
|
||||||
|
runs IN ORDER — a thinking run → `{ kind: "thinking", text: run.text }`
|
||||||
|
ONLY when `run.text` is non-empty (an empty run stores nothing); a tool
|
||||||
|
run → one `{ kind: "tool", …record }` per record in order (spread the
|
||||||
|
record object — the additive `offset/limit/path` + the phase-95 stamp
|
||||||
|
fields ride through; NO re-validation on the client — the server's
|
||||||
|
`ChatMessage` boundary validates).
|
||||||
|
- the `done` save point (~L3355, the `rememberBrainTurn` meta):
|
||||||
|
`segments: buildSegments(segRuns).length ? buildSegments(segRuns) :
|
||||||
|
undefined` (the `undefined`-drops-the-key house pattern, D4) — the
|
||||||
|
`thinking: thinkingAcc || undefined` + `tools: toolAcc.length ? toolAcc :
|
||||||
|
undefined` lines are UNCHANGED.
|
||||||
|
- `finalizeFailedTurn` (L2983): the parameter object gains `segments` (the
|
||||||
|
TWO call sites — the stream-drop site ~L3431 and the network-error
|
||||||
|
site ~L3519 — pass `buildSegments(segRuns)`); the function's `if
|
||||||
|
(wrap)` saved record includes `segments` ONLY when the passed value is
|
||||||
|
a non-empty array (`segments: segments?.length ? segments :
|
||||||
|
undefined`); the existing `acc`/`thinking`/`tools` fields are
|
||||||
|
UNCHANGED.
|
||||||
|
- the STOP path's save point (L3487–L3491, the `stopped: true` meta):
|
||||||
|
gains the same `segments` field (non-empty only) — a stopped partial
|
||||||
|
restores its timeline.
|
||||||
|
- the pagehide partial (L3642–L3650, the module-scope read — `segRuns`
|
||||||
|
is module scope, so it is readable there): gains the same `segments`
|
||||||
|
field (non-empty only). The zero-frame fallback save (L3456–L3462)
|
||||||
|
needs NOTHING (zero frames → no runs → no key).
|
||||||
|
- the `thinking`/`tools` keys are UNCHANGED at every site.
|
||||||
|
6. `tests/unit/test_thinking_timeline.py` (NEW — the source-level house
|
||||||
|
pattern, the `js[fn : js.find("\n}\n", fn)]` body-extraction the existing
|
||||||
|
pins use):
|
||||||
|
- `ensureThinkingRun` body: creates `details` + `className = "thinking"` +
|
||||||
|
`open = true` + the plain `Thinking` summary literal (D3 — the phase-17
|
||||||
|
literal, NOT a numbered variant) + inserts before `.bubble` (D2 — the
|
||||||
|
anchor is the bubble, NOT the old `.tool-calls-disclosure ?? …` chain);
|
||||||
|
`ensureToolRun` body: `className = "tool-calls-disclosure"` + `open =
|
||||||
|
true` + the summary/list structure.
|
||||||
|
- `appendToolLine` signature: the first param is the container (no
|
||||||
|
`body.querySelector(".tool-calls-disclosure")` in the body — the
|
||||||
|
single-container derivation is GONE); the four label literals STILL
|
||||||
|
present (mirror the guard); returns the line.
|
||||||
|
- `armToolLineClock`: the `:last-child` query is GONE; a `toolLineEl`
|
||||||
|
reference is used; `settleToolLine` resets it.
|
||||||
|
- `closeCurrentRun` / `closeAllRuns` defined; the `delta` branch calls
|
||||||
|
`closeCurrentRun` (pin: appears in the delta-branch region); `done` +
|
||||||
|
the stop/abort settle call `closeAllRuns`.
|
||||||
|
- `buildSegments` present; the `done` save + the `finalizeFailedTurn`
|
||||||
|
call site + the pagehide partial all reference it (≥ 3 call sites).
|
||||||
|
- `innerHTML` NOT in `ensureThinkingRun`… — pin per the builder choice
|
||||||
|
made in Work item 2 (if the live builder reuses the phase-17 innerHTML
|
||||||
|
template, pin that; the RESTORE/SHARED builders (task 03) are pinned
|
||||||
|
innerHTML-free there).
|
||||||
|
- `segRuns` is module scope + reset in `runTurn`.
|
||||||
|
- Builder style: MIRROR the function it replaces (the live thinking
|
||||||
|
builder keeps the phase-17 static-structure convention — a static
|
||||||
|
template with NO data — the data flows through `renderMarkdown` /
|
||||||
|
`textContent` exactly as today; the live tool builder keeps the
|
||||||
|
phase-117 createElement/textContent style — `innerHTML` ABSENT from
|
||||||
|
`appendToolLine`, as pinned today). Pin the STRUCTURE + the no-data-
|
||||||
|
through-innerHTML rule, not the mechanism.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_thinking_timeline.py` (this task's pins) + UPDATE the
|
||||||
|
existing source pins THIS task's change breaks (inspect, in order:
|
||||||
|
`test_frontend_feedback.py`, `test_frontend_turn_loader.py` — the phase-109
|
||||||
|
D15 re-open pin becomes the per-run pin (a NEW block, not a re-open),
|
||||||
|
`test_frontend_tool_states.py` — the `appendToolLine` single-container pin,
|
||||||
|
`test_big_read_progress.py` — the clock-query pin, `test_frontend_failed_turn.py`
|
||||||
|
— the `finalizeFailedTurn` payload pin, `test_thinking_scroll.py`,
|
||||||
|
`test_chat_persistence.py` — ONLY the pins this task touches; the
|
||||||
|
restore-path pins are task 03's, the shared-page pins are task 03's). Every
|
||||||
|
pin this task does NOT break (the tool-line label literals, the phase-117
|
||||||
|
deboxed CSS, the phase-139/146 suffix pins, the emoji guard) stays green
|
||||||
|
UNCHANGED.
|
||||||
|
- Coverage: **>90%** on this task's modified code (the pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] a live turn renders per-run blocks in arrival order (the code paths
|
||||||
|
land exactly as designed — the unit pins)
|
||||||
|
- [ ] the clock targets the tracked line reference (the `:last-child` query
|
||||||
|
is gone — the unit pin)
|
||||||
|
- [ ] the `done` / failed / pagehide save payloads carry `segments`
|
||||||
|
(non-empty only) — the unit pins
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green (every pin this task breaks is
|
||||||
|
updated; every pin it doesn't touch stays green); `uv run ruff check .
|
||||||
|
&& uv run pyright` clean
|
||||||
|
- [ ] `git diff --stat` limited to `frontend/assets/app.js`,
|
||||||
|
`tests/unit/test_thinking_timeline.py`, and the unit pin files updated
|
||||||
|
in this task
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# Task 03 — the restore path + the shared-page timeline
|
||||||
|
|
||||||
|
**Phase:** `158_thinking_timeline` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A restored brain record renders its `segments` as the collapsed timeline (the
|
||||||
|
grouping mirrors the live run structure); a record WITHOUT `segments` — every
|
||||||
|
pre-phase record — restores byte-identically through the untouched legacy
|
||||||
|
path; the shared page renders the same timeline from the same record shape.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `renderStoredMessage` (~L2089):
|
||||||
|
- AFTER the bubble exists: `if (Array.isArray(m.segments) &&
|
||||||
|
m.segments.length) { restoreTimeline(wrap, m.segments); } else { THE
|
||||||
|
EXISTING `m.thinking` BLOCK + THE EXISTING `m.tools` LOOP,
|
||||||
|
BYTE-IDENTICAL (including the `closeToolCalls(wrap)` call — RENAME it
|
||||||
|
to `closeAllRuns(wrap)` — task 02 removed `closeToolCalls` — the
|
||||||
|
restore's folded-at-rest result is the same) }`.
|
||||||
|
- `restoreTimeline(wrap, segments)` (a new function, the house style —
|
||||||
|
createElement/textContent, `renderMarkdown` for the thinking text,
|
||||||
|
`innerHTML` ABSENT for the structure): iterate `segments` in order:
|
||||||
|
- a `kind: "thinking"` entry (guard: `typeof entry.text === "string"`)
|
||||||
|
→ its OWN `details.thinking` (created CLOSED — D5;
|
||||||
|
`<summary>Thinking</summary>` — the phase-17 literal, D3;
|
||||||
|
`.thinking-text` = `renderMarkdown(entry.text)` — the existing
|
||||||
|
restore convention, escape-first), inserted before the bubble;
|
||||||
|
- CONSECUTIVE `kind: "tool"` entries → ONE
|
||||||
|
`details.tool-calls-disclosure` (created CLOSED — D5;
|
||||||
|
`summary.tool-calls-summary` + the `div.tool-calls` list
|
||||||
|
`role=list` `aria-label="Tool calls"` — the phase-117 structure),
|
||||||
|
each entry's line through `appendToolLine` (task 02's container
|
||||||
|
signature) with the EXISTING tOffset/tLimit/tPath guards
|
||||||
|
(`Number.isInteger(x) && x >= 1` / the string guard — byte-identical,
|
||||||
|
the phase-139/146 restore precedent) + the EXISTING
|
||||||
|
`appendTruncatedNote(wrap, arg, …)` call for `entry.truncated`
|
||||||
|
records (the phase-95 marker re-render — moved verbatim inside the
|
||||||
|
grouping); the summary count `Tool call(s) (N)` set once per
|
||||||
|
disclosure after its entries;
|
||||||
|
- an entry with an unknown `kind` (or a malformed shape) → SKIPPED
|
||||||
|
(never render from storage — textContent only, the house rule;
|
||||||
|
a guard comment naming the skip).
|
||||||
|
2. `frontend/assets/shared.js` — `addSegments(wrap, segments)` (new, the
|
||||||
|
shared-page parity copy of `restoreTimeline` — pure render, always
|
||||||
|
COLLAPSED — D5): the same grouping, the same line bytes (the four label
|
||||||
|
literals + the phase-95 truncation note + the phase-139/146 suffixes —
|
||||||
|
the parity `addToolLines` already carries; mirror it), the same summary
|
||||||
|
count, the same unknown-`kind` skip. The record-dispatch site (~L456):
|
||||||
|
`if (Array.isArray(m.segments) && m.segments.length) {
|
||||||
|
addSegments(wrap, m.segments); } else { THE EXISTING addThinkingBlock +
|
||||||
|
addToolLines, BYTE-IDENTICAL }`. The `addThinkingBlock` / `addToolLines`
|
||||||
|
functions themselves are UNCHANGED (the legacy path is intact).
|
||||||
|
3. `tests/unit/test_thinking_timeline.py` — ADD the pins:
|
||||||
|
- the `renderStoredMessage` dispatch (segments take precedence; the
|
||||||
|
legacy path intact when `segments` is absent — pin the
|
||||||
|
`Array.isArray(m.segments) && m.segments.length` guard);
|
||||||
|
- `restoreTimeline` body: `document.createElement("details")` +
|
||||||
|
`className = "thinking"` (NO `open = true` — collapsed, D5) + the plain
|
||||||
|
`Thinking` literal + the consecutive-tool grouping (one disclosure per
|
||||||
|
run — pin the grouping loop) + the `appendTruncatedNote` call + the
|
||||||
|
unknown-`kind` skip; `innerHTML` ABSENT (structure built via
|
||||||
|
createElement/textContent).
|
||||||
|
- the `shared.js` `addSegments` body: the same structure pins (collapsed,
|
||||||
|
the parity) + `innerHTML` ABSENT; the dispatch guard in `shared.js`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_thinking_timeline.py` (this task's pins) + UPDATE the
|
||||||
|
existing source pins THIS task's change breaks (inspect:
|
||||||
|
`test_chat_persistence.py` — the restore-path pins, `test_shared_page.py` —
|
||||||
|
the `addToolLines`/dispatch pins; every pin this task does NOT break stays
|
||||||
|
green UNCHANGED).
|
||||||
|
- Coverage: **>90%** on this task's modified code (the pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the `restoreTimeline` + `addSegments` builders land exactly as designed
|
||||||
|
(the unit pins)
|
||||||
|
- [ ] the legacy restore / shared paths are byte-unchanged (a `git diff`
|
||||||
|
scoped check — only the dispatch guard is added)
|
||||||
|
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run
|
||||||
|
pyright` clean
|
||||||
|
- [ ] `git diff --stat` limited to `frontend/assets/app.js`,
|
||||||
|
`frontend/assets/shared.js`, `tests/unit/test_thinking_timeline.py`,
|
||||||
|
and the unit pin files updated in this task
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# Task 04 — the mock interleaved flow + the dedicated E2E suite
|
||||||
|
|
||||||
|
**Phase:** `158_thinking_timeline` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A deterministic mock trigger that interleaves a DISTINCT reasoning run before
|
||||||
|
every tool call and before the final answer (the phase-158 flow), and the
|
||||||
|
dedicated `test_thinking_timeline.py` Playwright suite that pins the
|
||||||
|
timeline contract (order / fold-at-rest / live-open / restore + record /
|
||||||
|
shared-page parity / the no-thinking single-run grouping).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/mock_llm.py`:
|
||||||
|
- `THINK_TOOLS_TRIGGER = "think out loud and use your tools"` (beside
|
||||||
|
`THINKING_TRIGGER`, L992) + the docstring entry for the flow (the
|
||||||
|
four-request stateless shape, the `Step N` markers, the phase-37
|
||||||
|
answer convention). BEFORE wiring, VERIFY (and note in the docstring):
|
||||||
|
no existing trigger phrase is a substring of it, and NO existing E2E
|
||||||
|
question contains it (grep `tests/e2e/*.py` for the phrase — the
|
||||||
|
convention that keeps the other suites byte-unaffected).
|
||||||
|
- `_thinking_tools_flow(body) -> tuple[str, ...] | None` (shaped like
|
||||||
|
`_tool_flow`, L1820 — the same stateless discrimination, the same
|
||||||
|
`<tools>` + `tools`-parameter gating, reusing `_drill_target` /
|
||||||
|
`_first_file_line` / the `read`-result header check):
|
||||||
|
- request 1 (tools offered, no tool results yet):
|
||||||
|
`thinking = "Step 1: list the documents to find the reference server
|
||||||
|
config."` → `_tool_call_stream("ls", {}, "call_0", thinking)`;
|
||||||
|
- request 2 (a `tool`-role top-level source listing — no file lines
|
||||||
|
yet): `thinking = "Step 2: drill into the first source."` → the
|
||||||
|
`_drill_target` scoped `ls` (id `call_1`), with thinking;
|
||||||
|
- request 3 (a `tool`-role folder listing WITH file lines — the
|
||||||
|
phase-63/70 labeled `source:` / `path:` format): `thinking = "Step 3:
|
||||||
|
read the file line's document."` → `read` on the JOINED combined
|
||||||
|
`source/path` (id `call_2`), with thinking;
|
||||||
|
- request 4 (a `tool`-role read result — content starting with the
|
||||||
|
agent's `"Document <source/path>:"` header): `thinking = "Step 4:
|
||||||
|
compose the answer from the read document."` → the phase-37 answer
|
||||||
|
convention: `Read <source/path>. <first 80 chars of the read
|
||||||
|
document's content>` (the `_sse_stream` content path — the
|
||||||
|
`thinking=` param of `_sse_stream` already exists — pass it; the
|
||||||
|
final answer's thinking streams ahead of the content, exactly like
|
||||||
|
the `think out loud` answer path).
|
||||||
|
- `_tool_call_stream(name, arguments, call_id, thinking="")` (L3813):
|
||||||
|
the optional fourth param — when non-empty, yield
|
||||||
|
`reasoning_content` chunks FIRST (the 12-char split — the
|
||||||
|
`_sse_stream` convention), then the EXISTING `tool_calls` frames +
|
||||||
|
`finish_reason: "tool_calls"` UNCHANGED. Default `""` → every
|
||||||
|
pre-phase 3-arg caller (the ~10 call sites) streams BYTE-IDENTICAL
|
||||||
|
(verify with the existing tool-flow E2E suites in the regression
|
||||||
|
pass). The docstring updated (the param, the phase-158 flow).
|
||||||
|
- the `chat_completions` streaming dispatch: check
|
||||||
|
`_thinking_tools_flow(body)` BEFORE the `_tool_flow(body)` dispatch
|
||||||
|
(L4453 region) — the more-specific-phrase-wins convention (the phrase
|
||||||
|
contains `use your tools`, so the plain `_tool_flow` must NOT see
|
||||||
|
it); on a hit, route each flow stage through the same
|
||||||
|
`StreamingResponse` wrapper the `_tool_flow` stages use.
|
||||||
|
2. `tests/e2e/test_thinking_timeline.py` (NEW — the six tests, the phase
|
||||||
|
design's contract):
|
||||||
|
- fixtures: the conftest `app_server` / `page` / `seeded_kb` (mirror
|
||||||
|
`test_agent_document_tools.py` — the admin login, the seeded KB so
|
||||||
|
request 4's read result exists);
|
||||||
|
- 1. `test_timeline_order_after_completed_turn` — the trigger
|
||||||
|
question; after the `done` state: the brain wrap's `.msg-body`
|
||||||
|
children before the bubble are EXACTLY `[details.thinking,
|
||||||
|
details.tool-calls-disclosure, details.thinking,
|
||||||
|
details.tool-calls-disclosure, details.thinking,
|
||||||
|
details.tool-calls-disclosure, details.thinking]` (assert the
|
||||||
|
tag+class sequence — 7 elements); the four `.thinking-text`s carry
|
||||||
|
DISTINCT `Step 1`…`Step 4` markers (a substring assert each —
|
||||||
|
distinctness proves the runs are separate blocks, not one lump); each
|
||||||
|
disclosure summary is `Tool call (1)`; the answer bubble carries the
|
||||||
|
`Read ` echo.
|
||||||
|
- 2. `test_runs_fold_at_rest` — after `done`: NO `.thinking` /
|
||||||
|
`.tool-calls-disclosure` has the `open` attribute (a `count` assert
|
||||||
|
on `[open]`); the summaries are visible.
|
||||||
|
- 3. `test_live_runs_open_while_streaming` — an `expect.poll` predicate
|
||||||
|
over the mock's pacing window (the tool frames' 0.1 s + the thinking
|
||||||
|
stream): a `.thinking` or `.tool-calls-disclosure` is `open` while
|
||||||
|
the answer bubble is still EMPTY (the D5 live-open contract).
|
||||||
|
- 4. `test_restored_timeline_and_record` — RELOAD (same context): the
|
||||||
|
restored message renders the same 7-element sequence, all closed; the
|
||||||
|
localStorage record (read via `page.evaluate` on the `bor.chat.v1`
|
||||||
|
storage key — the phase-14 key) for that brain turn carries a
|
||||||
|
`segments` array of EXACTLY 7 entries in the pattern
|
||||||
|
`thinking, tool, thinking, tool, thinking, tool, thinking` (assert the
|
||||||
|
`kind` sequence + the `Step N` text on the thinking entries + the
|
||||||
|
`name` on the tool entries); the `thinking` (full) + `tools` (3
|
||||||
|
entries) keys are STILL present (D4 — the legacy fallback fields).
|
||||||
|
- 5. `test_shared_page_timeline` — the phase-51 save + share flow (the
|
||||||
|
`test_share_chat.py` pattern), a FRESH context on `/shared/<token>`:
|
||||||
|
the same 7-element sequence renders, all collapsed, the `Step N`
|
||||||
|
markers visible when expanded (expand ONE — the click on a summary —
|
||||||
|
then assert), the answer bubble carries the `Read ` echo.
|
||||||
|
- 6. `test_no_thinking_turn_single_disclosure` — a plain `use your
|
||||||
|
tools` question (NO thinking trigger): EXACTLY ONE
|
||||||
|
`.tool-calls-disclosure` (3 `.tool-call` lines, summary `Tool calls
|
||||||
|
(3)`) and ZERO `.thinking` (the run-grouping contract — no thinking
|
||||||
|
frames → one tool run — the phase-117 behavior preserved).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The mock change is exercised by the suite (the trigger flow) + the
|
||||||
|
existing tool-flow suites stay byte-unaffected (the `thinking=""` default
|
||||||
|
— verified in the regression pass, task 05).
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov`
|
||||||
|
green **in isolation** (prereq: `podman compose up -d db`), mock-only.
|
||||||
|
- Coverage: the mock is test code (no `app/` change — the coverage floor is
|
||||||
|
held by the untouched suite; task 01's schema code is covered there).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the mock trigger + flow + the `_tool_call_stream` param land exactly
|
||||||
|
as designed; the pre-phase callers are byte-unchanged (a `git diff`
|
||||||
|
scoped check)
|
||||||
|
- [ ] all six tests pass in isolation
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` green
|
||||||
|
in isolation; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# Task 05 — the phase-109 suite pin update, the full gate, docs pass, commit
|
||||||
|
|
||||||
|
**Phase:** `158_thinking_timeline` · **Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Update `test_turn_progress_loader.py` from the phase-109 D15 re-open pin to
|
||||||
|
the per-run contract (the ONE owner-directed behavior change — recorded in
|
||||||
|
the phase design), run the full completion gate (unit + integration,
|
||||||
|
coverage >90%, the phase E2E + the regression E2E suites in isolation,
|
||||||
|
lint + types), run the documentation & consistency pass, and cut the single
|
||||||
|
atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_turn_progress_loader.py` — the D15 pin → the per-run
|
||||||
|
contract (the mock's turn-progress flow is `delta → tool → thinking →
|
||||||
|
delta → thinking → done` — the first thinking run starts AFTER the
|
||||||
|
first delta):
|
||||||
|
- the re-open pin (`test_thinking_block_reopens_after_delta_with_visible_loader`
|
||||||
|
— "the thinking block re-opens after the answer has started … the
|
||||||
|
scratchpad carries both thinking rounds") becomes the PER-RUN pin:
|
||||||
|
after the turn, the brain wrap holds TWO `.thinking` blocks (the
|
||||||
|
`count` assert), EACH carrying its OWN round's text (the flow's two
|
||||||
|
distinct reasoning texts — assert a marker on each, the way the mock's
|
||||||
|
two reasoning streams are already distinct); both are FOLDED at rest
|
||||||
|
(no `[open]`); the visible-during-streaming half of the pin (the
|
||||||
|
loader visible, the thinking text non-empty and open while its run
|
||||||
|
streams) is kept — a run is open in its live window (D5 replaces D15's
|
||||||
|
re-open: a new block instead of a re-opened one — the user sees the
|
||||||
|
new thinking either way; the D15 symptom — "the frozen chat" — stays
|
||||||
|
fixed). The module docstring updated (the phase-158 supersession,
|
||||||
|
owner direction 2026-10-04, recorded).
|
||||||
|
- the other tests in the file (the loader visibility pins) stay green
|
||||||
|
UNCHANGED — verify.
|
||||||
|
2. The full gate (the phase's Testing & Quality section):
|
||||||
|
- `uv run pytest` — the ENTIRE project suite, 100% green;
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on
|
||||||
|
`app/`;
|
||||||
|
- `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` — the
|
||||||
|
phase E2E, green in isolation (prereq: `podman compose up -d db`);
|
||||||
|
- the regression E2E suites, each green in isolation
|
||||||
|
(`-v --no-cov`): `test_thinking_display.py`,
|
||||||
|
`test_tool_call_compact.py`, `test_turn_progress_loader.py` (the
|
||||||
|
UPDATED pins), `test_big_read_progress.py` (UNCHANGED — the clock's
|
||||||
|
visible behavior is identical), `test_agent_document_tools.py`,
|
||||||
|
`test_share_chat.py`, `test_chat_persistence.py`, `test_smoke.py`;
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- If a regression suite fails on a pin the redesign broke that tasks
|
||||||
|
02–04 missed: update THAT pin to the per-run contract (the same
|
||||||
|
supersession logic as item 1 — note it in the commit body), and
|
||||||
|
re-run the suite in isolation.
|
||||||
|
3. The documentation & consistency pass (the phase's Documentation &
|
||||||
|
Consistency section):
|
||||||
|
- `README.md` + `AGENTS.md` reviewed in full against the project as it
|
||||||
|
stands after this phase; every inconsistency fixed, **including
|
||||||
|
pre-existing ones** (the README's chat thinking/tool-call display
|
||||||
|
description, if any, gains the timeline sentence).
|
||||||
|
- the `app/schemas.py` docstring voice (task 01) + the
|
||||||
|
`frontend/assets/app.js` / `shared.js` comment blocks (tasks 02–03)
|
||||||
|
read in the established phase-citation voice (spot-check).
|
||||||
|
4. The commit + the phase move:
|
||||||
|
- `git add -A .agents/ app/schemas.py frontend/ tests/ README.md && git
|
||||||
|
commit --no-gpg-sign -m "feat(ui): render thinking and tool runs as a
|
||||||
|
chronological timeline (per-run blocks, segments persistence)"`
|
||||||
|
(the AGENTS.md rule 8 + A17; the LLM-Generated trailer per the
|
||||||
|
git-commit skill's convention for this repo's commits — check the
|
||||||
|
last few commits' bodies for the trailer form and match);
|
||||||
|
- `mv .agents/phases/todo/158_thinking_timeline
|
||||||
|
.agents/phases/complete/158_thinking_timeline`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The gate above IS this task's testing; no new code.
|
||||||
|
- Coverage: the >90% floor (the gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `test_turn_progress_loader.py` updated to the per-run contract +
|
||||||
|
green in isolation (item 1)
|
||||||
|
- [ ] the FULL gate green (item 2 — every line)
|
||||||
|
- [ ] README.md + AGENTS.md accurate and consistent with the whole project
|
||||||
|
(item 3 — the final pass)
|
||||||
|
- [ ] `git diff --stat` of the phase's total footprint limited to
|
||||||
|
`app/schemas.py`, `frontend/assets/app.js`,
|
||||||
|
`frontend/assets/shared.js`, `tests/`, the docs, and the phase files
|
||||||
|
(the phase design's scope check)
|
||||||
|
- [ ] one atomic `--no-gpg-sign` commit; the phase dir moved to
|
||||||
|
`.agents/phases/complete/`
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Phase 159 — Edit the last question and re-ask it
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
|
||||||
|
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-10-05)
|
||||||
|
**Context:** `frontend/assets/app.js` (4,007 lines) — `getActionsRow(wrap)` (L498, the `.msg-meta.msg-actions` `role=list` row every bubble action joins as a `listitem`), `appendTuneButton` (L517) + `openTuneForm` (L866, the inline-form precedent: `form.tune-form` + labelled `textarea[maxlength]` + `.tune-form-actions` Save/Cancel + a `role=alert` status line + "one form at a time" via `document.querySelectorAll(".tune-form").forEach(f => f.remove())`), `appendRetryButton` (L617) + `markLastRetryable()` (L650, the single-target discipline: remove every `.retry-btn`, re-append on exactly one bubble, gated on the record invariant), `addMessage(who, html, scroll, image)` (L969, the ONE bubble renderer — `.msg.<who> > .avatar + .msg-body > .bubble`), `renderStoredMessage(m)` (L2254; the user branch L2262 renders `renderMarkdown(m.text)` + the optional `m.image`), `restoreConversation()` (L2410; ends with `markLastRetryable()`), `MAX_QUESTION_CHARS = 4000` (L1689, mirrors `ChatRequest.message max_length=4000`) + `updateCharCount()` (L1702, the `len/4000 — character limit` copy), `conversation` + `trimToBudget` (L2128) + `saveConversation()` (L2143, localStorage `bor.chat.v1`) + `persistConversation()` (L2532, the phase-55 auto-save), `retryLastTurn(wrap)` (L2984, the redo-in-place precedent: pop the last brain record, keep the question as the last entry, `runTurn(text, { reask: true })`), `runTurn(text, { reask, image })` (L3260; `reask` skips the user append + save point 1, `history = conversation.slice(0, -1)`), `finalizeFailedTurn` (L3209, phase 120), `UI_STATE` + `setUiState` (the thinking/streaming in-flight gate every action obeys), `lastBrainWrap`.
|
||||||
|
Server side (read-only for this phase): `app/schemas.py` `ChatRequest.message` (`max_length=4000`, L77) and `ChatMessage.text` (`max_length=32_000`, L1264, `extra="forbid"`), `app/api/chats.py` (the raw `bor.chat.v1` `messages` JSONB round-trip — logic untouched), `app/api/chat.py` L705–733 (the phase-114 terminal `error` frame `detail="Question too long — trim it and re-ask."` + its `hint` — deterministic, no retry frames: the exact turn this phase gives a remedy for).
|
||||||
|
CSS: `frontend/assets/styles.css` `.msg-meta.msg-actions` (L795), `.tune-btn` (L980), `.retry-btn` (L1010, the pill family), `.tune-form` + `label` + `textarea` + `.tune-form-actions` (L1116–1141), the mobile `min-height: 44px` action rules (L5708–5709).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The last question the user sent carries an **Edit** action: click it, the bubble becomes a textarea prefilled with the question, trim it, Save — the answer is re-asked from the edited text. A "Question too long — trim it and re-ask" turn (or any mistyped / half-formed question) is fixed by editing, never by retyping.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `158_thinking_timeline` (complete) — pipeline predecessor (execution order) only.
|
||||||
|
- Code dependencies (all complete): phase 49/53 redo-in-place retry (`retryLastTurn` + `runTurn({reask:true})`), phase 120 failed-turn records (`appendFailedNote`, `finalizeFailedTurn` — the driver case leaves a failed brain record after the question), phase 14 persistence (`conversation` / `saveConversation`), phase 55 auto-save (`persistConversation`), phase 130 `.msg-actions` row, phase 114 the "question too long" error frame, phase 123 the user-bubble image (`m.image`).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Target set (LOCKED, owner 2026-10-05): exactly ONE editable bubble** — the most recent user message. `markLastEditable()` mirrors `markLastRetryable()`: remove every `.edit-btn` first, then append to the last user bubble, but only when the last **user** record exists in `conversation` (scan newest-first for `who === "user"`; the records after it are the answers it produced — that index is `editIdx`). The bubble itself is tracked in a new `lastUserWrap` module variable (the exact mirror of `lastBrainWrap`, assigned in `runTurn`'s fresh-send `addMessage("user", …)` and in `renderStoredMessage`'s user branch, cleared by `startNewChat()`). Earlier bubbles are never editable, so no mid-session branch exists and the "what happens to the turns after it" question never arises for an old message.
|
||||||
|
- **Editing happens IN the bubble (LOCKED, owner 2026-10-05)** — `openEditor(wrap, idx)` is the `openTuneForm` family: `form.edit-form` in the bubble's `.msg-body`, `<label for="edit-input-N">`, `textarea[maxlength="4000"]` prefilled from `conversation[idx].text`, Save (submit) + Cancel, a `role=alert` `.edit-error` line. The rendered `.bubble` is hidden (`.is-editing`) while the form is open; Cancel / `Escape` restores it and returns focus to the toggle button; `Ctrl/Cmd+Enter` submits; only one editor open at a time.
|
||||||
|
- **Save = replace + truncate + re-ask (LOCKED)** — the edited text replaces `conversation[editIdx].text` in place, **every record after it is dropped** (the answers to the old text are stale by construction), `saveConversation()` + `persistConversation()` persist the new shape, the dropped bubbles are removed from the DOM, `lastBrainWrap = null`, `markLastRetryable()` re-targets, then `runTurn(newText, { reask: true })`. Because the edited question ends up as the LAST record, `runTurn`'s `history = conversation.slice(0, -1)` invariant and `reask` (skip the user append + save point 1) hold **unchanged** — this phase adds no API, no schema field, no migration (**A10 stateless holds**).
|
||||||
|
- **Empty guard:** a whitespace-only save is rejected in-form (the `.edit-error` line, the form stays open) — `ChatRequest.message` / `ChatMessage.text` both require `min_length=1`; the >4000 case is unreachable because the textarea carries `maxlength`.
|
||||||
|
- **The image survives, the re-ask is text-only** (the phase-49/123 A7 rule): the record keeps its `image` key and its bubble keeps the attachment; `runTurn(newText, { reask: true })` sends no `image`, exactly like `retryLastTurn`.
|
||||||
|
- **In-flight discipline:** `markLastEditable()` no-ops while `uiState` is `thinking`/`streaming` (the same guard `retryLastTurn` opens with), and `saveEditor` re-checks it — a save can never start a second turn.
|
||||||
|
- **NOT touched:** `runTurn` (its `reask` path is consumed as-is), `retryLastTurn`, `handleSend`, `rememberBrainTurn`, `trimToBudget`/`persistConversation` (called, not changed), `app/` server code, the SSE contract (A15), the shared-chat page.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_edit_affordance_on_user_bubbles.md` — the Edit button (`appendEditButton` / `markLastEditable`) + the inline editor (`openEditor`, Save updates the record + the bubble text + localStorage only).
|
||||||
|
2. `02_resend_in_place.md` — Save also truncates the records after the edited question (DOM + `conversation`) and re-asks through `runTurn({reask:true})`; the in-flight and empty guards.
|
||||||
|
3. `03_editable_error_turn.md` — the driver case: the question stays editable when its turn failed (the "Question too long" frame, the phase-120 failed record, a stopped answer) and after a refresh.
|
||||||
|
4. `04_edit_resend_tests.md` — unit source-pins + the `test_chats_api` edited-record round-trip + the isolated `tests/e2e/test_edit_and_resend.py`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (house style = source pins on `frontend/assets/app.js`, e.g. `tests/unit/test_frontend_failed_turn.py`): the `.edit-btn` join-the-actions-row rule, `markLastEditable`'s remove-then-append single target, the inline form's labelled textarea + `maxlength=4000` + Cancel/`Escape`/`Ctrl+Enter` paths, the replace-and-truncate record edit, the `runTurn(text, { reask: true })` call, the in-flight no-op, and `retryLastTurn`/`runTurn` byte-unchanged.
|
||||||
|
- Integration: `tests/integration/test_chats_api.py` (extend) — a `PUT /api/chats/<id>` whose last user record carries the edited text and whose later records were dropped round-trips byte-identically (no schema change is required; this pins the claim).
|
||||||
|
- Coverage: **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
|
||||||
|
- E2E (mandatory, A16): `tests/e2e/test_edit_and_resend.py`, run **in isolation** (`uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The most recent user bubble carries Edit; opening it shows a textarea prefilled with the question; Cancel/`Escape` restores the original bubble text.
|
||||||
|
- [ ] Saving trimmed text updates the bubble + the `bor.chat.v1` record in place, drops every turn after the question (DOM + storage + the saved chat), and streams a fresh answer from the edited text with no duplicated question.
|
||||||
|
- [ ] A refresh after an edit shows the edited question and the new answer (no stale tail resurrected).
|
||||||
|
- [ ] The "Question too long — trim it and re-ask" turn's question is editable, so trimming + Save re-asks successfully.
|
||||||
|
- [ ] Retry, Stop, the failed-turn path, and every successful turn behave byte-identically to before; earlier user bubbles show no Edit button.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation (DB up); regressions green in isolation: `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **Owner-locked (2026-10-05, roadmap confirmation): last question only** — one editable bubble, managed by the `markLastRetryable()` single-target discipline. Mid-conversation branching is explicitly out of scope.
|
||||||
|
- **Owner-locked (2026-10-05): inline edit in the bubble** — the `.tune-form` family, not the composer (the composer keeps its own draft untouched by editing).
|
||||||
|
- **Save truncates the tail and re-asks** (the phase-49 redo-in-place rule, one level up: the question is replaced instead of kept verbatim) — no message versioning, no "edited" history trail, one new API-free flow.
|
||||||
|
- **A10 untouched** — `POST /api/chat` stays stateless; the edited question is just the client's next `message`. No migration, no `ChatMessage` field (the ≤4000 / ≤32 000 caps already cover it).
|
||||||
|
- **Edit is available to every visitor who can chat** — no admin gate, the phase-49 Retry precedent.
|
||||||
|
- **A16/A17 honoured** — one E2E file, one atomic commit.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(chat): edit the last question and re-ask it — fixes trim-and-re-ask with no way to trim"
|
||||||
|
```
|
||||||
+37
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 01 — Edit button + inline editor on the last question
|
||||||
|
|
||||||
|
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The most recent user bubble carries an **Edit** action that swaps the bubble for a textarea prefilled with the question; Save writes the new text into the bubble and the `bor.chat.v1` record, Cancel restores the original. (The re-ask itself lands in task 02 — this task ships shippable: an editable question that persists.)
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — add `EDIT_ICON` beside `RETRY_ICON` (L606) and `TUNE_ICON` (L475): an inline `aria-hidden` pencil SVG in the same stroke family (`fill="none" stroke="currentColor" stroke-width="1.7"`, 24×24 viewBox) — no icon font, no CDN (A11).
|
||||||
|
2. `frontend/assets/app.js` — `appendEditButton(wrap)`, directly after `appendRetryButton` (L617) and shaped exactly like it: `getActionsRow(wrap)` → bail if the row is missing → one-per-bubble guard on `.edit-btn` → `btn.type = "button"`, `btn.className = "edit-btn"`, `role="listitem"` when the row reports `role="list"`, `btn.innerHTML = EDIT_ICON + "<span>Edit</span>"`, click → `openEditor(wrap, btn)`. A visible "Edit" label carries the accessible name (no ellipsis clipping needed at this width).
|
||||||
|
3. `frontend/assets/app.js` — `markLastEditable()`, the sibling of `markLastRetryable()` (L650): `messagesEl.querySelectorAll(".edit-btn").forEach(b => b.remove())` FIRST (an older bubble's button is stale the moment a newer question lands), then bail when `uiState === UI_STATE.thinking || uiState === UI_STATE.streaming` (the `retryLastTurn` in-flight guard, L2985), then scan `conversation` newest-first for `who === "user"` through a small helper `lastUserRecordIndex()` (returns `-1` when the conversation holds no user record); bail on `-1`; append the button on `lastUserWrap`. Task 02 reuses `lastUserRecordIndex()` for the truncation.
|
||||||
|
- ASSUMPTION: the last-user-record index comes from that newest-first `conversation` scan (the `markLastRetryable` last-brain-index rule, one `who` over), NOT from a DOM↔record index map — the DOM and the record list are parallel today but re-deriving the index from the record keeps the editor correct after a task-02 truncation.
|
||||||
|
- The wrap is tracked in a new module variable `lastUserWrap`, the exact mirror of `lastBrainWrap`: assigned where a user bubble is created — `runTurn`'s fresh-send `addMessage("user", …)` (L3270) and `renderStoredMessage`'s user branch (L2262, which is what makes the restore path work for free) — cleared by `startNewChat()` (L2886). `markLastEditable()` appends only when `lastUserWrap` exists AND its record really is the last user record (the `editIdx` scan above), so a stale wrap can never carry the button. Task 02 reuses the same variable.
|
||||||
|
- Call sites — pair every existing `markLastRetryable()` call (there are exactly six: `restoreConversation()` L2415, `restoreSavedChatFromUrl()` L2482, `finalizeFailedTurn()` L3247, the `done` settle L3649, the zero-frame fallback L3734, the stop finalize L3769): call `markLastEditable()` immediately after each. `startNewChat()` (L2886) needs no call: resetting `messagesEl` removes the buttons with the list (the phase-49 reasoning). Task 03 verifies the error-path pairs (L3247 / L3734 / L3769) end-to-end.
|
||||||
|
4. `frontend/assets/app.js` — `openEditor(wrap, toggleBtn)`, the `openTuneForm` (L866) family, module-level `editSeq` for unique ids (the `tuneSeq` precedent):
|
||||||
|
- `document.querySelectorAll(".edit-form").forEach(f => f.remove())` — one editor at a time.
|
||||||
|
- `form.className = "edit-form"`, `form.noValidate = true`; `<label for="edit-input-N">Edit your question</label>`; a `textarea` with `id="edit-input-N"`, `rows="3"`, `maxlength="${MAX_QUESTION_CHARS}"` (L1689 — the 4000 cap stays single-sourced), `value = conversation[editIdx].text` assigned with `.value` (never an HTML string — the house no-injection rule), `placeholder` reused from the composer if one exists.
|
||||||
|
- `.edit-form-actions` with a `submit` `.edit-save` ("Save") and a `type="button"` `.edit-cancel` ("Cancel"); a `<p class="edit-error" role="alert" hidden>` for the empty-text rejection.
|
||||||
|
- While open: `wrap.querySelector(".bubble")` gets `.is-editing` (CSS-hides the rendered question so the question appears ONCE); on close the class is removed.
|
||||||
|
- Keys: `Escape` in the textarea → cancel; `Ctrl/Cmd+Enter` → submit; cancel restores the bubble, removes the form, returns focus to `toggleBtn` (the `openTuneForm` cancel rule).
|
||||||
|
- Submit (`e.preventDefault()`): re-check the in-flight guard (no-op), `const text = textarea.value.trim()`; empty → set the `.edit-error` copy ("Ask something first — the question can't be empty."), unhide, keep the form open, return; otherwise this task's scope: `conversation[editIdx].text = text`, `saveConversation()`, re-render the bubble text through `renderMarkdown(text)` (the `addMessage` renderer, so markdown rules stay in one place), close the editor, announce through the existing `#steering-announcer`-style pattern only if a generic announcer already exists — otherwise the visible re-render is the feedback (AGENTS.md rule 10 is satisfied by the bubble change; do NOT invent a new live region).
|
||||||
|
- Leave a clearly-marked `// task 02: truncate the tail + re-ask` seam where the close happens, so the next task adds the resend without restructuring.
|
||||||
|
5. `frontend/assets/styles.css` — `.edit-btn` in the `.retry-btn` family (L1010: same pill size/spacing, the global `:focus-visible` ring applies — verify it is not scoped to `.tune-btn`/`.retry-btn` only and extend the selector list if it is), `.edit-btn:hover` in the same tint; `.msg.user .bubble.is-editing { display: none; }`; the `.edit-form` card, label, textarea and actions row cloned from the `.tune-form` block (L1116–1141) and `.edit-error` cloned from `.tune-error`; add `.edit-btn { min-height: 44px; }` beside `.tune-btn`/`.retry-btn` in the mobile block (L5708–5709) and honor `prefers-reduced-motion` if the family transitions anything.
|
||||||
|
6. `frontend/index.html` — no change (the chat shell, the `.messages` live region and the composer already exist). Confirm no new `<script>`/`<link>` is introduced (AGENTS.md rule 6, the no-CDN test).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_edit_message.py` (new) — house-style source assertions: `.edit-btn` is appended through `getActionsRow` and joins `role=list` as `listitem`; `markLastEditable` removes before appending and bails on `UI_STATE.thinking`/`streaming`; `openEditor` renders a `<label for>` + `textarea[maxlength]` bound to `MAX_QUESTION_CHARS` and prefills from `conversation`; `Escape`/Cancel restore, `Ctrl/Cmd+Enter` submits; an empty trim shows `.edit-error` and keeps the form; the bubble text is re-rendered with `renderMarkdown`; `retryLastTurn` and `runTurn` are byte-unchanged (read the file and pin the two function bodies' current text).
|
||||||
|
- Coverage: **>90%** on `app/` — unchanged (frontend-only task).
|
||||||
|
- Manual check (visible browser or `page.goto` in a scratch Playwright run): send a question, Edit → Cancel leaves the original text; Edit → Save updates the bubble and survives a reload.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -n "edit-btn\|edit-form\|markLastEditable" frontend/assets/app.js frontend/assets/styles.css` shows the button, the form and the CSS family.
|
||||||
|
- [ ] Exactly one bubble (the newest question) shows Edit at rest; after a new send the button moved to the new bubble.
|
||||||
|
- [ ] Save updates the bubble text and the `bor.chat.v1` localStorage record; a reload shows the edited question.
|
||||||
|
- [ ] Cancel / `Escape` leave the question byte-identical; an empty trim is refused in-form.
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL unchanged (>90%); `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 02 — Save re-asks: replace the question, drop the stale tail, stream the new answer
|
||||||
|
|
||||||
|
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Saving an edited question **resends** it: the answers that followed the old text are removed (DOM + `conversation` + the saved chat), the edited question becomes the last record, and a fresh answer streams from it — the phase-49 redo-in-place semantics one level up (the question is replaced instead of kept verbatim).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — in `openEditor`'s submit handler (task 01's seam), after the record text is written and BEFORE the editor closes:
|
||||||
|
- `const editIdx = lastUserRecordIndex()` (the newest-first `who === "user"` scan from task 01) and `conversation.splice(editIdx + 1)` — every record the OLD question produced is dropped outright (never rewritten, the phase-114 whole-turn rule).
|
||||||
|
- Remove the matching DOM bubbles: every `.msg` after `lastUserWrap` in `messagesEl` (`let n = lastUserWrap.nextSibling; while (n) { const next = n.nextSibling; n.remove(); n = next; }`).
|
||||||
|
- `lastBrainWrap = null` (the removed answer is no longer a retry target) and clear `leavePartialIndex` / the phase-73 pagehide partial pointer if it points past `editIdx` (a partial saved from the dropped turn must not be re-persisted into the new shape — read `runTurn`'s `leavePartialIndex` usage before deciding the exact reset; `= -1` is the safe value).
|
||||||
|
- `saveConversation()` (localStorage, `trimToBudget` rides it unchanged) + `persistConversation()` (the phase-55 auto-save pushes the truncated shape to `PUT /api/chats/<id>` — fire-and-forget, never blocks the turn).
|
||||||
|
- `markLastRetryable()` — the Retry button re-targets the (now removed) tail: after the drop there is no brain record, so the call simply clears every `.retry-btn`.
|
||||||
|
2. `frontend/assets/app.js` — same handler, then `closeEditor()` and `return runTurn(text, { reask: true })`:
|
||||||
|
- `reask: true` is the whole point — `runTurn` (L3260) skips the user-bubble append and save point 1, so the question is never duplicated; `history = conversation.slice(0, -1)` (L3340) is exactly the prior turns because the edited question is the LAST record.
|
||||||
|
- Return the promise (the `retryLastTurn` L2984/`regenerateStaleChat` convention — `runTurn` never rejects; a failure lands in the phase-120 failed-turn path).
|
||||||
|
- No `image` is passed: a re-ask sends text only (the phase-49/123 locked A7 rule); the record keeps its `image` key and the bubble keeps its attachment.
|
||||||
|
3. `frontend/assets/app.js` — guards, in this order at the top of the submit handler: in-flight (`uiState === UI_STATE.thinking || UI_STATE.streaming` → no-op, a save can never start a second turn), then the empty trim (`Ask something first — the question can't be empty.` in `.edit-error`, form stays open). Re-check the in-flight state a second time immediately before `runTurn` (the same double-check `retryLastTurn` gets by being a single synchronous stretch — keep the handler synchronous from the guard to the `runTurn` call except for nothing: no `await` between the guard and the call).
|
||||||
|
4. `frontend/assets/app.js` — keep the composer out of it: editing never touches `input.value`, `autoGrow()` or `updateCharCount()` (the user's unsent draft survives an edit), and `MAX_QUESTION_CHARS` stays the textarea's `maxlength` so a >4000 trim is unreachable server-side (A10: the server's `ChatRequest.message max_length=4000` remains the real bound).
|
||||||
|
5. No server change: `app/api/chat.py` sees an ordinary `POST /api/chat` with the edited `message` and a shorter `history`; `app/api/chats.py` stores the shorter `messages` array through the existing `SavedChatUpdate` path (no new field, no migration).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_edit_message.py` (extend) — the splice is `conversation.splice(editIdx + 1)` (the whole tail, nothing before it), the DOM removal starts at `lastUserWrap.nextSibling`, `lastBrainWrap = null` precedes the re-ask, `saveConversation()` + `persistConversation()` both run before `runTurn`, and the call is exactly `runTurn(text, { reask: true })` with no `image`; the in-flight guard precedes the record edit; `retryLastTurn` stays byte-unchanged.
|
||||||
|
- Integration: `tests/integration/test_chats_api.py` (extend) — a `PUT /api/chats/<id>` carrying the edited last user record and a truncated `messages` list round-trips byte-identically (pins "no schema change is needed"; a shortened array is already legal).
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code changes — the gate must hold).
|
||||||
|
- Manual: ask a question, get an answer, Edit → change one word → Save: the old answer disappears, the new answer streams in its place, the question appears once.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Save removes every turn after the edited question (DOM + `bor.chat.v1` + the saved-chat row) and streams exactly one fresh answer from the edited text.
|
||||||
|
- [ ] The question is never duplicated in the DOM, in localStorage, or in the request `history`.
|
||||||
|
- [ ] Retry on the new answer, Stop during the new turn, and "New chat" all behave as before.
|
||||||
|
- [ ] Editing while a turn is in flight is impossible (no Edit button, and a queued Save no-ops).
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 03 — The driver case: a failed or rejected question stays editable
|
||||||
|
|
||||||
|
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The exact turn the owner hit — the terminal `error` frame `Question too long — trim it and re-ask.` — leaves its question **editable**, so trimming + Save re-asks successfully; the same holds for every phase-120 failed turn, a stopped partial, and after a page refresh.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — verify and, where missing, add the `markLastEditable()` pairing on the paths that settle WITHOUT a `done` frame, so the question bubble always ends up with its Edit button once the turn is terminal:
|
||||||
|
- the `runTurn` non-OK-HTTP path (L3359: a 422 from `ChatRequest` — e.g. `message` over 4000 — throws with the server `detail` and lands in the error catch) and the SSE `error` frame path (the phase-114 "Question too long" frame arrives as an `error` event, L3675/L3806) — both settle through the error catch → `finalizeFailedTurn` (L3209), which already calls `markLastRetryable()` at L3247: `markLastEditable()` must be the very next statement;
|
||||||
|
- the zero-frame fallback (L3734) and the phase-48 stop finalize (L3769) — same pairing;
|
||||||
|
- the 300s guard abort (`armTurnTimeout`, L3283 → `setUiState(UI_STATE.error, …)`) — confirm the catch's `aborted` branch reaches a `markLast*()` call; if it settles without one, call `markLastRetryable()` + `markLastEditable()` there too (a timed-out turn must not strand the question).
|
||||||
|
2. `frontend/assets/app.js` — the error state must not lock the editor: `markLastEditable()`'s guard bails only on `UI_STATE.thinking` / `UI_STATE.streaming`, never on `UI_STATE.error` / `UI_STATE.idle` (a failed turn IS idle-or-error at rest — read `setUiState` and pin the two states the guard checks). `saveEditor`'s in-flight guard mirrors it exactly.
|
||||||
|
3. `frontend/assets/app.js` — the failed answer record is part of the stale tail: after an edit-save the phase-120 `{ who: "brain", failed: true, error }` record is dropped by task 02's `conversation.splice(editIdx + 1)` like any other answer, and its bubble (with `.failed-note`) is removed with it. Confirm `lastBrainWrap = null` happens on that path so no stale Retry target survives, and that `rememberBrainTurn`'s internal `trimToBudget`-driven index shift (its `brainIdx -= trimDrop` bookkeeping, L2861) can never be confused by the truncation (the truncation happens before the new turn starts, so the new turn's `rememberBrainTurn` sees the post-splice array — add the one-line comment saying so).
|
||||||
|
4. `frontend/assets/app.js` — the restore paths: a reload after a failed turn renders the failed bubble (phase 120) and the question above it; `renderStoredMessage`'s user branch sets `lastUserWrap` (task 01) and the paired `markLastEditable()` after `markLastRetryable()` (L2415 / L2482) puts Edit back on that question. Verify the shared-chat page (`?chat=` admin restore) does NOT get the button for a read-only view — ASSUMPTION: the `?chat=` restore is an editable working copy exactly like the local restore (it already is Retry-able, so Edit follows); if the code shows a genuinely read-only render path, gate on it and record the finding here.
|
||||||
|
5. `frontend/assets/app.js` — the banner: `showErrorBanner(detail, opts)` (L2918) is NOT changed (the phase-120 rule — its `opts.retryable && lastBrainWrap` condition stays byte-identical). Edit is an in-bubble affordance beside Retry, never a banner control.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_edit_message.py` (extend) — `markLastEditable()` follows every `markLastRetryable()` call site (assert the six pairings by counting both calls), the guard names only `UI_STATE.thinking`/`UI_STATE.streaming` (no `UI_STATE.error`), `showErrorBanner` is byte-unchanged, and the failed-turn splice claim (a `failed: true` record after the edited question is dropped, not kept).
|
||||||
|
- Integration: none new (the phase-120 failed-record round-trip in `tests/integration/test_chats_api.py` already covers the shape this task reads).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged — no `app/` change).
|
||||||
|
- Manual: with the mock LLM's embed-failure trigger (`tests/e2e/mock_llm.py` `EMBED_FAIL_TRIGGER`, L1321) or a >4000-char question, confirm the error frame's question is editable and a trimmed Save answers successfully.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] After the "Question too long — trim it and re-ask." error: the question bubble shows Edit, the failed bubble shows Retry, trimming + Save produces a real answer.
|
||||||
|
- [ ] The same holds for a network-error turn, a mid-stream drop, a stopped answer, and a 300s timeout.
|
||||||
|
- [ ] A page refresh after any of the above restores the question with a working Edit.
|
||||||
|
- [ ] `showErrorBanner` and the phase-120 failed-record shape are byte-unchanged.
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 04 — Tests for edit-and-resend (unit + integration + isolated E2E)
|
||||||
|
|
||||||
|
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The phase's contract is pinned: one editable bubble, the inline editor, replace-and-truncate-and-re-ask, the editable failed turn, and byte-identical Retry/Stop/persistence behavior — with the phase's own Playwright file run in isolation (AGENTS.md rule 9).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_frontend_edit_message.py` (new) — house-style source pins against `frontend/assets/app.js` (+ `styles.css`), one focused assertion cluster per behavior:
|
||||||
|
- `EDIT_ICON` exists and is an `aria-hidden` inline SVG; `.edit-btn` is created in `appendEditButton` through `getActionsRow` with the one-per-bubble guard and the `role=list` → `listitem` rule;
|
||||||
|
- `markLastEditable()` removes `.edit-btn` before appending, bails on `UI_STATE.thinking`/`UI_STATE.streaming` (and NOT on `UI_STATE.error`), uses `lastUserRecordIndex()` + `lastUserWrap`, and is called after all six `markLastRetryable()` call sites;
|
||||||
|
- `openEditor`: `<label for>` + `textarea` + `maxlength` bound to `MAX_QUESTION_CHARS`, prefilled from `conversation`, `.edit-form` uniqueness, `Escape`/Cancel restore, `Ctrl/Cmd+Enter` submit, the empty-trim `.edit-error` branch;
|
||||||
|
- save path order: guard → `conversation[editIdx].text = text` → `conversation.splice(editIdx + 1)` → DOM removal from `lastUserWrap.nextSibling` → `lastBrainWrap = null` → `saveConversation()` → `persistConversation()` → `runTurn(text, { reask: true })` (no `image` key);
|
||||||
|
- regressions pinned as byte-unchanged: `retryLastTurn`, `runTurn`, `showErrorBanner`, `handleSend`, `rememberBrainTurn`;
|
||||||
|
- `styles.css` carries `.edit-btn`, `.edit-form`, `.edit-error`, `.is-editing` and the 44px mobile rule.
|
||||||
|
2. `tests/integration/test_chats_api.py` (extend) — two cases: (a) a `PUT /api/chats/<id>` whose `messages` array is the post-edit shape (edited last user record, later records dropped) round-trips byte-identically on `GET`; (b) the same shape on a shared chat still serves through `SharedChatOut.messages` — pins "the edited/truncated conversation needs no schema change".
|
||||||
|
3. `tests/e2e/test_edit_and_resend.py` (new) — mock-LLM suite on the `app_server`/`page`/`db_ready` fixtures, the phase-120 file as the template (TRUNCATE-then-seed `saved_chats` in an autouse fixture; `auth_helpers.login` for the authed chat; seed `tests/fixtures/docs/homelab/kubernetes.md` as `docs`/`homelab/kubernetes.md` so retrieval grounds and the mock answers with `MOCK_ANSWER_MARKER`):
|
||||||
|
1. `test_edit_last_question_reasks_in_place` — send "How is my Kubernetes cluster set up?" → grounded answer → the user bubble has Edit and the OLD brain bubble count is 1 → click Edit → the textarea is prefilled with the exact question → append " in one line" → Save: exactly ONE user bubble (the edited text), the old answer bubble is gone, the new answer streams with the mock marker, `bor.chat.v1` holds `[…user(edited), brain]` with no duplicate question, and `saved_chats.messages` matches (poll the row).
|
||||||
|
2. `test_only_the_last_question_is_editable` — two turns: exactly one `.edit-btn` exists and it is on the SECOND user bubble; the first user bubble has none.
|
||||||
|
3. `test_edit_cancel_is_a_no_op` — open the editor, press `Escape`: the bubble text, the record, and the answer are unchanged (byte-identical `localStorage` payload before/after); the button returns focus (assert `document.activeElement` is the Edit toggle).
|
||||||
|
4. `test_edit_drops_a_stale_tail_and_survives_refresh` — after an edit-save, `page.reload()`: the edited question + the new answer restore, the dropped answer never reappears, and Edit is back on the question.
|
||||||
|
5. `test_failed_turn_question_is_editable_and_trim_reasks` — the driver case: send a question carrying the mock's terminal-failure trigger (`ALWAYS_FAIL_TRIGGER`, "always fail") so the turn settles with an `error` frame and a failed bubble (the phase-114 "question too long" frame takes the identical settle path — see the docstring note: its exact detail string is real-endpoint-only), assert Edit exists on the question beside the failed bubble's Retry, then Edit → Save with the trigger stripped out and watch the grounded answer stream.
|
||||||
|
6. `test_edit_is_inert_while_the_turn_runs` — start a long answer (`LONG_ANSWER_TRIGGER`, "write a long answer"), assert no `.edit-btn` while streaming, wait for `done`, assert Edit lands.
|
||||||
|
7. negative: `test_composer_draft_survives_an_edit` — type an unsent draft in the composer, edit + save the last question, the draft text is still in `#composer`'s input.
|
||||||
|
- MOCK-ONLY: the suite sets no `E2E_REAL_LLM`; the file's docstring states the trigger names, the seed, and the per-test contract mapping (the phase-120 file's shape).
|
||||||
|
4. `tests/e2e/test_edit_resend.py` is NOT created (one file per phase, A16) — regression runs stay in the phase's final gate: `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`, `test_stop_generation.py`, `test_smoke.py`, each in isolation.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Run in this order: `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` → `uv run pytest` (full unit + integration) → `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%) → `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` (DB up) → the regression files above, isolated.
|
||||||
|
- No `app/` code changes in this phase: coverage must not move; if a `tests/`-only change drops the TOTAL below 90%, the gate failure is a phase bug (investigate, do not add filler tests).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green (unit + integration) with TOTAL coverage >90%.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] Every regression E2E file listed above green in isolation.
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] The phase dir moves to `.agents/phases/complete/` with the single `--no-gpg-sign` commit (`00_phase.md` §Commit).
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Phase 160 — The agent knows its tool-call budget (and never dumps tool calls into the answer)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 — "The LLM need to know how many tool calls it has left. During long research sessions it will sometimes end the conversation with tool calls that just get dumped into chat because it ran out of turns. Every tool call should also return 'you have x tool calls remaining' or some other helpful message and then on the last one attempted tool calls should be rejected rather than spit out to chat and the LLM should be encouraged to come up with a reply or tell the user to continue the session by allowing more tool calls (or something similar)."
|
||||||
|
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-10-05)
|
||||||
|
**Context:** `app/rag/agent.py` (2,322 lines) — `run_agent` (L1995): `max_rounds = settings.agent_max_rounds` (L2103), `tools = agent_tools(settings) if max_rounds > 0 else None` (L2104), `rounds = 0` (L2106), the round loop (L2107) whose pieces are all yielded verbatim (L2136-2142: `if isinstance(piece, ToolCallPiece): calls.append(piece)` … `yield piece`), `call = calls[0]` (L2224 — "a stream can carry several calls; run the first"), `result = _execute_tool(tool_db, call, seed_docs, holder, settings)` (L2227, a plain string), `rounds += 1` (L2229), the `agent tool=%s args=%s round=%d/%d` INFO line (L2230), `messages.append({role: assistant, tool_calls: […]})` + `messages.append({"role": "tool", "tool_call_id": …, "content": result})` (L2237-2251), the phase-95 `ToolResultPiece` yields (L2258), and the **round-cap block** (L2270-2322): the `agent round cap reached (rounds=%d) — forcing a final no-tools answer` warning, the forced `chat_stream_retried(llm, messages, tools=None, …, scaffolding=final_filter)` whose loop yields EVERY piece (`ToolCallPiece` included), and the phase-71 terminal `MalformedReplyError` when the forced final carried no visible content but stripped scaffolding.
|
||||||
|
Refusal/teaching constants (the house pattern for teaching the model in-band): `ALREADY_IN_CONTEXT` (L537), `UNKNOWN_TOOL` (L541), `READ_TRUNCATION_NOTICE` (L578), `REREAD_PARTIAL` (L628), `CORRECTION_INSTRUCTION` (L732, folded into the single system message as `system_prompt + "\n" + CORRECTION_INSTRUCTION` at L2178 / L1044 in `app/api/chat.py`). `AgentHolder` (L1482: `read_docs`, `tool_calls`, `scaffold_stripped`, `read_truncations`, `shown_images`, `read_coverage` — the API layer reads it after the stream).
|
||||||
|
`app/config.py` `agent_max_rounds: int = 10` (+ its loud validator, ~L229-240 / L694 pattern), `.env.example` L69 (`# BOR_AGENT_MAX_ROUNDS=10 … (0 = no tools)`).
|
||||||
|
`app/api/chat.py` — the grounded vs deflected dispatch (L840-900), `tools=None` for the deflected turn (L870) and the kill switch, `_pump(pieces)` (L893, the ONE piece→SSE loop: `ToolCallPiece` → `tool` frame at L909, `ToolResultPiece` → `tool_result`, `thinking`/`delta`), the per-turn log line (L1196-1219: `… thinking_chars=%d tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d`), the `ChatDoneEvent` yield (~L1265).
|
||||||
|
`app/schemas.py` — `ChatDoneEvent` (L316: `type`/`deflected`/`sources`/`related`/`suggestions`/`answer_image` + the `@model_serializer(mode="wrap")` omission rule that drops `answer_image` when `None` — "ABSENT, never null"), `ChatErrorEvent.hint` (the phase-114 additive precedent), `ChatMessage` (L1264ff, `extra="forbid"`, the optional `failed`/`error`/`stopped` markers).
|
||||||
|
`app/rag/scaffolding.py` — `SCAFFOLD_PATTERNS` (the observed-forms registry: `<|tool_call_start|>…<|tool_call_end|>`, `<|tool_calls|>`, `<|tool_call|>`) + `ScaffoldingFilter` (content only, never thinking).
|
||||||
|
Tests: `tests/unit/test_agent.py` (the always-calling fake LLM: `_run(llm, holder, _settings(agent_max_rounds=N))`, the round-cap cluster L4561-4944, the forced-final scaffolding case L5241-5255 asserting `MalformedReplyError`, `llm.requests[i][1] is None` as the "no tools offered" pin), `tests/integration/test_chat_api.py`, `tests/e2e/mock_llm.py` (`TOOLS_TRIGGER` L1114 "use your tools", `MULTI_READ_TRIGGER` L1123 "read two documents", `SCAFFOLD_TRIGGER` L1375 "emit raw tool markup", `LONG_ANSWER_TRIGGER` L1035), `tests/e2e/test_agent_unlimited_tools.py` (the multi-read flow + the TRUNCATE-then-seed KB fixture).
|
||||||
|
PLAN §4 (SSE contract), §9 (the required per-turn line), A15 (event types `thinking, delta, tool, retry, done, error, tool_result`), A10 (`POST /api/chat` stateless), A16/A17.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every tool result tells the model how many tool calls it has left; when the budget is gone a further attempted call is **rejected instead of reaching the chat** (no tool lines, no raw markup, never an empty answer); the forced final is explicitly coached to answer with what it has or tell the user it can keep going; and the user gets a one-click **Continue researching** action that runs a fresh turn with a whole new budget. `BOR_AGENT_MAX_ROUNDS` stays 10 (owner-locked).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Telemetry is model-facing (LOCKED):** the remaining count rides the tool-result string the loop already appends as `{"role": "tool", …}` — no new SSE event type, A15 untouched. Trailer constants live with the other teaching strings in `app/rag/agent.py`:
|
||||||
|
`TOOL_BUDGET_REMAINING = "\n\n[tool calls remaining: {n}]"` for `n >= 2`, `… 1 call remaining` wording at `n == 1`, and `TOOL_BUDGET_SPENT = "\n\n[No tool calls remain — answer now from what you have, or tell the user to say \"continue\" to keep researching.]"` at `n == 0`. Appended in `run_agent` (NOT inside `_execute_tool`, which stays a pure `(db, call, seed_docs, holder, settings) → str` seam and keeps its existing unit tests byte-identical), after `rounds += 1`, so a truncated read's `ToolResultPiece` and the `read_coverage` bookkeeping are unaffected.
|
||||||
|
- **`agent_max_rounds = 0` (the kill switch) is untouched:** no tools are offered, so no tool result and no trailer exist — the single `tools=None` request stays byte-identical (the phase-45 pin, `tests/unit/test_agent.py::test_zero_max_rounds_is_one_request_without_tools`).
|
||||||
|
- **The bug class this phase kills:** a request made WITHOUT tools can still come back with structured `tool_calls` deltas. Today those `ToolCallPiece`s become SSE `tool` frames (`app/api/chat.py::_pump`, L909) and render as tool lines in the chat — a round-capped or deflected turn "ending with tool calls dumped into chat", exactly the owner's report. New invariant: **a request with `tools=None` never produces a `tool` frame, and never leaves the answer empty.**
|
||||||
|
- **The cap path (task 02):** the forced final's loop stops yielding `ToolCallPiece`s (they are REJECTED: not run, not shown, not counted in `holder.tool_calls` — the refusal style: "counts in nothing"), counts them in a new `AgentHolder.rejected_calls`, logs one warning (`agent: %d tool call(s) attempted after the round cap — rejected, not shown`), and — when the forced final streamed no visible content — yields the deterministic `ROUND_CAP_FALLBACK_ANSWER` content piece (an honest "I've used up this turn's research budget — here's where I got to; say continue to dig further" line) instead of dying. The model is coached (next bullet) to produce the "say continue to keep going" line itself; `ROUND_CAP_FALLBACK_ANSWER` is the deterministic backstop for the case where it emits only calls or only scaffolding.
|
||||||
|
- **Coaching the forced final (task 02):** `ROUND_CAP_WRAPUP_INSTRUCTION` (a module constant beside `CORRECTION_INSTRUCTION`) is folded into the single system message for the forced final — the exact L2178 pattern (`system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION`) — spelling out: no tools remain, NEVER emit tool-call markup or a tool call, answer from the results you already have, and if the answer is incomplete say so and tell the user they can say "continue" to keep researching.
|
||||||
|
- **The same rule for the other tool-less requests (task 02):** `app/api/chat.py::_pump` drops `ToolCallPiece`s while the request had no tools — the deflected turn (A8 `DEFLECT_MODE`, L870) and the kill-switch turn — so the phase-71 incident class is closed at the transport too. The `ScaffoldingFilter` keeps stripping textual markup forms; this task only stops STRUCTURED calls from becoming UI lines.
|
||||||
|
- **User-visible signal (tasks 03/04):** one additive `done` key `budget_exhausted: bool` (serialized ABSENT when false, the `answer_image` omission precedent — every pre-phase frame stays byte-identical), driven by `AgentHolder.cap_reached`, plus the per-turn log fields `rounds=%d` and `cap_hit=%d`. The frontend answers it with a `Continue researching` button that sends a plain next turn (`runTurn("continue")`): the budget is per-turn, so a new turn has `agent_max_rounds` again — **no client-supplied budget in the request body** (a client-tunable round cap is a DoS surface and would break the server-side kill switch, A10-adjacent).
|
||||||
|
- **NOT touched:** `BOR_AGENT_MAX_ROUNDS`'s default and validator, the `tool` SSE event shape (A15), `done.sources`/`related`/`suggestions`/`answer_image`, the retry primitive (A2/phase 67), `ScaffoldingFilter`'s pattern registry (a new form gets an observed capture, not a guess), the phase-71 recovery path's `MalformedReplyError` (non-cap: a deflected/round turn whose SECOND reply is scaffolding-only still errors).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_remaining_in_tool_results.md` — the budget trailer on every executed tool result (model-facing), + the docstring/env-comment updates.
|
||||||
|
2. `02_reject_calls_at_the_cap.md` — no `tool` frames from a `tools=None` request (the forced final, the deflected turn, the kill switch), `AgentHolder.rejected_calls`, the cap-path fallback answer, and `ROUND_CAP_WRAPUP_INSTRUCTION` on the forced final.
|
||||||
|
3. `03_budget_exhausted_flag_and_log.md` — `AgentHolder.rounds` / `cap_reached`, the additive `done.budget_exhausted` key, the `rounds=` / `cap_hit=` log fields, and the one PLAN.md §9/§4 revision note.
|
||||||
|
4. `04_continue_researching_button.md` — the UI: the button, the persisted record marker, the restore path, CSS.
|
||||||
|
5. `05_budget_tests.md` — the mock's round-cap marker + unit + integration + the isolated `tests/e2e/test_tool_budget_telemetry.py`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_agent.py` (extend) — the trailer on every tool result (the fake LLM's recorded `messages`: `[n] remaining` decreasing 9, 8, … 1, then the spent line at 0; `agent_max_rounds=1` → the spent line on the only result; `agent_max_rounds=0` → NO tool message at all), the forced final's system message carries `ROUND_CAP_WRAPUP_INSTRUCTION`, a `ToolCallPiece` on the forced final is not yielded and does not reach the API layer (`holder.rejected_calls` counts it, `holder.tool_calls` does not), the cap path with a call-only / scaffolding-only forced final yields `ROUND_CAP_FALLBACK_ANSWER` instead of raising, and the existing cap cluster (`llm.requests[i][1] is None`, the `round cap reached (rounds=N)` warning) still passes.
|
||||||
|
- Integration: `tests/integration/test_chat_api.py` (extend) — a cap-hitting mock turn streams NO `tool` frame after the cap and ends with `done` carrying `budget_exhausted: true`; a normal grounded turn's `done` frame is byte-identical to today (the key is absent, not false); a deflected turn emits no `tool` frame even when the mock returns structured calls.
|
||||||
|
- Coverage: **>90%** on `app/` (AGENTS.md rule 9) — the new branches (cap-path rejection, the fallback answer, the serializer omission) each need a test.
|
||||||
|
- E2E (mandatory, A16): `tests/e2e/test_tool_budget_telemetry.py`, run **in isolation** (`uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A grounded turn's tool messages carry a decreasing `[tool calls remaining: N]` trailer and the last one says the budget is spent (visible in a live turn's server log or a captured request).
|
||||||
|
- [ ] A turn that hits the cap ends with a readable answer — zero tool lines after the cap, zero raw tool markup in the answer, no empty answer, no `MalformedReplyError` on the cap path.
|
||||||
|
- [ ] `done.budget_exhausted` is `true` exactly on cap-reached turns and ABSENT (never `false`) everywhere else.
|
||||||
|
- [ ] The exhausted answer offers **Continue researching**; clicking it sends one new turn and streams a new answer with a fresh tool budget; earlier answers' buttons are gone.
|
||||||
|
- [ ] A refresh keeps the Continue button on a flagged answer; a shared chat never shows it.
|
||||||
|
- [ ] `agent_max_rounds=0` still sends exactly one `tools=None` request and no trailer; deflected turns are unchanged; `BOR_AGENT_MAX_ROUNDS` default is still 10.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` green in isolation (DB up); regressions green in isolation: `test_agent_unlimited_tools.py`, `test_agent_document_tools.py`, `test_chat_rag.py`, `test_harness_aligned_tools.py`, `test_grep_regex_teaching.py`, `test_smoke.py`.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **Owner-locked (2026-10-05): `BOR_AGENT_MAX_ROUNDS` stays 10** — this phase teaches the model and the user about the budget; it does not raise it.
|
||||||
|
- **Owner-locked (2026-10-05): Continue lives on the answer** — a button, not a live counter in the tool lines (the "you have x remaining" half of the TODO is model-facing).
|
||||||
|
- **A15 holds — no new SSE event type:** the count rides the tool result text; the user-facing signal is one additive `done` key (the `related`/`answer_image` precedent, PLAN §4's additive rule).
|
||||||
|
- **The boundary call still runs:** the call the model emitted while budget remained is executed and its result carries `[tool calls remaining: 0]`; only calls attempted AFTER the budget is spent are rejected (never executed, never shown) — no in-flight research is thrown away and the OpenAI wire stays valid.
|
||||||
|
- **Recorded revision of phase 71's cap-path terminal (AGENTS.md rule 3, owner-confirmed 2026-10-05):** a round-capped forced final that carries no visible content no longer raises `MalformedReplyError` — it yields `ROUND_CAP_FALLBACK_ANSWER`, because an honest "budget spent" line beats an error frame for the exact symptom this TODO reports. The non-cap recovery path (a round or deflected reply that is pure scaffolding twice in a row) keeps `MalformedReplyError` unchanged. The PLAN.md revision note in task 03 records it (the phase-45 pattern: a recorded revision, never a silent deviation).
|
||||||
|
- **A10 holds** — Continue is an ordinary stateless `POST /api/chat` with the client's history; no session state, no client-controlled budget, no migration.
|
||||||
|
- **A16/A17 honoured** — one E2E file, one atomic commit.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(agent): teach the tool-call budget and reject calls past the cap instead of dumping them into chat"
|
||||||
|
```
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 01 — Every tool result tells the model how many calls remain
|
||||||
|
|
||||||
|
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "The LLM need to know how many tool calls it has left… Every tool call should also return 'you have x tool calls remaining' or some other helpful message."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The model sees its remaining round budget in every tool result it receives — so it plans its last reads and wraps up on its own instead of hitting the wall mid-research.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — add the trailer constants beside the other in-band teaching strings (the `READ_TRUNCATION_NOTICE` L578 / `REREAD_PARTIAL` L628 family, before `CORRECTION_INSTRUCTION` L732):
|
||||||
|
- `TOOL_BUDGET_REMAINING: str = "\n\n[tool calls remaining: {n}]"` — the `.format(n=…)` call site is the only place that builds it;
|
||||||
|
- `TOOL_BUDGET_LAST: str = "\n\n[1 tool call remaining — make it count, then answer]"`;
|
||||||
|
- `TOOL_BUDGET_SPENT: str = "\n\n[No tool calls remain — answer now from what you have, or tell the user to say \"continue\" to keep researching.]"`;
|
||||||
|
- one small pure helper `def _budget_note(remaining: int) -> str` (n ≥ 2 → the formatted remaining line, 1 → `TOOL_BUDGET_LAST`, ≤ 0 → `TOOL_BUDGET_SPENT`) so the wording lives in exactly one place and the unit tests hit a pure function.
|
||||||
|
2. `app/rag/agent.py::run_agent` — after `result = _execute_tool(...)` (L2227) and `rounds += 1` (L2229), append the note before the message goes on the wire: `messages.append({"role": "tool", "tool_call_id": call.id, "content": result + _budget_note(max_rounds - rounds)})` (L2251). Keep `_execute_tool` itself unchanged — its return value is the raw tool result and its existing unit tests stay byte-identical.
|
||||||
|
- The note rides EVERY executed call, including a **rejected** call's refusal text (`UNKNOWN_TOOL`, `ALREADY_IN_CONTEXT`, …) — a refusal consumes a round, so the budget line is exactly as true there.
|
||||||
|
- A truncated read is unaffected: the phase-95 `ToolResultPiece` yields (L2258) read `holder.read_truncations`, not the result string, so the SSE `tool_result` frame and the UI marker stay byte-identical.
|
||||||
|
- `agent_max_rounds == 0` never reaches this code (no tools offered, `tools=None` at L2104) — the kill-switch request stays byte-identical.
|
||||||
|
3. `app/rag/agent.py` — extend the module docstring's loop contract (the L31-60 block, and the phase-45 round-cap paragraph at L214-235): one sentence naming the trailer, its three forms, and the rule "the count is what the model has LEFT after the call it is reading" (so the first result of a 10-round budget reads 9).
|
||||||
|
4. `.env.example` — extend the `BOR_AGENT_MAX_ROUNDS` comment (L69) to mention the telemetry: `# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools); every tool result carries the model's remaining count (phase 160)`. No new setting, no default change (LOCKED).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_agent.py`, extend the round-cap cluster at L4561): a 3-round always-calling fake LLM produces three tool messages whose contents end with `[tool calls remaining: 2]`, `[1 tool call remaining…`, `[No tool calls remain…` respectively (assert on the recorded `llm.requests[i][0]` message list — the fake already records the exact messages sent); `agent_max_rounds=1` → the single tool message ends with `TOOL_BUDGET_SPENT`; `agent_max_rounds=0` → the request carries no `role: "tool"` message at all (the existing kill-switch pin, extended with one assert); a rejected call (unknown tool) still ends with its refusal + the note; `_budget_note` boundary table (`2`, `1`, `0`, `-1`).
|
||||||
|
- The existing cap/retry/scaffolding tests must keep passing unchanged (the trailer is appended content, not a new message).
|
||||||
|
- Coverage: **>90%** on `app/` — `_budget_note` + the `run_agent` append line fully covered.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A live grounded turn's second model request contains a `role: "tool"` message ending in `[tool calls remaining: N]` with `N == agent_max_rounds - rounds_consumed`.
|
||||||
|
- [ ] The final tool result of a capped turn carries the "no tool calls remain … say \"continue\"" line.
|
||||||
|
- [ ] No SSE frame, `done` shape, `tool_calls=` log field, `read_coverage` span or `tool_result` frame changed (`uv run pytest tests/integration/test_chat_api.py -v --no-cov` green).
|
||||||
|
- [ ] `agent_max_rounds=0` still issues exactly one `tools=None` request with no trailer anywhere.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 02 — Past the cap: reject the call, coach the wrap-up, never dump into chat
|
||||||
|
|
||||||
|
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…on the last one attempted tool calls should be rejected rather than spit out to chat and the LLM should be encouraged to come up with a reply or tell the user to continue the session by allowing more tool calls (or something similar)."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A tool call attempted after the budget is spent is rejected — it never runs, never becomes a `tool` frame, never renders in the chat — the forced final is explicitly coached to answer or hand off, and a cap-path turn always ends with readable text.
|
||||||
|
|
||||||
|
## Background (the actual defect — read before coding)
|
||||||
|
The round-cap path already forces one `tools=None` request (L2270-2322), but that request's pieces are yielded **verbatim**, and `app/api/chat.py::_pump` (L893) turns every `ToolCallPiece` into an SSE `tool` frame (L909). A model that answers a tool-less request with structured `tool_calls` anyway — the documented behavior of the hosted `lite` family, same family as the phase-71 scaffolding incident — therefore lands its tool calls in the user's chat as tool lines with no answer behind them. That is the reported symptom. The same holds for the deflected turn (A8, `tools=None` at L870) and the `agent_max_rounds=0` kill switch.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — new module constants next to `CORRECTION_INSTRUCTION` (L732):
|
||||||
|
- `ROUND_CAP_WRAPUP_INSTRUCTION: str` — the forced final's coaching, in the CORRECTION_INSTRUCTION voice, covering exactly four points: no tool calls remain; do NOT emit a tool call, tool-call JSON, or any `<|tool_call…|>`-style markup (a tool call now has no effect at all); answer from the results already in the conversation, stating plainly what is still unchecked; if the answer is incomplete, end by telling the user they can say **"continue"** to let you keep researching.
|
||||||
|
- `ROUND_CAP_FALLBACK_ANSWER: str` — the deterministic backstop line (one short sentence in the brand voice: the research budget for this turn ran out before an answer came together, what was gathered still stands above, say **"continue"** to keep digging). Keep it free of markdown decoration (the UI renders it through the normal markdown path).
|
||||||
|
2. `app/rag/agent.py` — `AgentHolder` (L1482): add `rejected_calls: int = 0` (+ a docstring paragraph in the `tool_calls` style: calls the model attempted after the round cap — never executed, never shown, counted here only; the refusal style, "counts in nothing").
|
||||||
|
3. `app/rag/agent.py::run_agent`, the forced-final block (L2270-2322):
|
||||||
|
- build the coached message list the way the phase-71 recovery does it (L2172-2180): `messages_capped = [{"role": "system", "content": system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION}, *messages[1:]]` and pass THAT to `chat_stream_retried(..., tools=None, …)` (the retry/restart semantics of locked A2 are unchanged);
|
||||||
|
- inside the consumption loop, intercept before yielding: `if isinstance(piece, ToolCallPiece): holder.rejected_calls += len(…); continue` — one counter increment per attempted call, nothing yielded (no `tool` frame, no visible markup);
|
||||||
|
- keep the existing `final_content` tally for `StreamPiece(kind="content")` and the existing `holder.scaffold_stripped` / per-span warning block;
|
||||||
|
- after the stream closes, when `final_content == 0`: log one warning (`agent: round cap hit with no visible reply (rejected_calls=%d, scaffold_stripped=%d) — answering with the budget fallback`) and `yield StreamPiece(kind="content", text=ROUND_CAP_FALLBACK_ANSWER)`; **delete the `MalformedReplyError` raise on this path only** (the recorded revision, `00_phase.md` §Locked decisions — an honest "budget spent" line beats an error frame for this symptom), while every non-cap `MalformedReplyError` (the round recovery, the deflected recovery, `app/api/chat.py` L1060ff) stays exactly as it is;
|
||||||
|
- keep the `agent round cap reached (rounds=%d) — forcing a final no-tools answer` warning (its existing unit pin at `tests/unit/test_agent.py` L4944 must not change), and add the attempted-calls warning beside it: `agent: %d tool call(s) attempted after the round cap — rejected, not shown`.
|
||||||
|
- Retries still obey locked A2: the coached request goes through `chat_stream_retried` with the same `settings.llm_retries` / `llm_retry_delay` (the phase-67 pin at `tests/unit/test_agent.py` L4800 keeps passing — update its expectation only if it asserts the exact system text).
|
||||||
|
4. `app/api/chat.py::_pump` (L893) — the transport half of the same invariant: add a keyword `tools_offered: bool = True` to the piece loop and drop `ToolCallPiece`s (counting them into a local so one debug line can name them) when `tools_offered is False`. Pass `tools_offered=False` for the deflected/kill-switch request (the `chat_stream(..., tools=None)` call at L860-875 and its recovery at L1040-1060) and `True` for the grounded first pass — `run_agent` already suppresses the cap-path calls internally, so the grounded loop keeps its current behavior. A15 is untouched: the event TYPE list in PLAN §4 does not change, a frame is simply no longer emitted for a call that cannot run.
|
||||||
|
5. `app/rag/agent.py` — the `run_agent` docstring's round-cap paragraph (L214-235) gains the rejection rule + the fallback-answer rule in two sentences, and `app/api/chat.py`'s module docstring gains one line for the no-`tool`-frame-without-tools invariant.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_agent.py`, extend the phase-71 cluster at L5241 and the cap cluster at L4561): a fake LLM whose forced final (`llm.requests[i][1] is None`) streams `ToolCallPiece`s only → the run yields NO `ToolCallPiece`, yields the `ROUND_CAP_FALLBACK_ANSWER` content piece, `holder.rejected_calls == N`, `holder.tool_calls` unchanged, no exception; the same shape with tool calls + real content → the content streams and no fallback is appended; a scaffolding-only forced final → the fallback replaces the old `MalformedReplyError` (update that existing test and name the revision in its docstring); the forced final's recorded system message contains `ROUND_CAP_WRAPUP_INSTRUCTION` while `messages[1:]` (user + tool history) is unchanged; the `round cap reached (rounds=N)` and the new `attempted after the round cap` warnings both appear in `caplog`.
|
||||||
|
- Integration (`tests/integration/test_chat_api.py`, extend — the scripted `tool_script` fake at L84/L125 drives this without touching `tests/e2e/mock_llm.py`): a grounded turn with `agent_max_rounds=1` whose forced final returns a `ToolCallPiece` streams `tool` ×1 (the executed round) and then NOTHING of the sort — no second `tool` frame, no `error` frame — and settles with `done` whose delta text is the answer or the fallback; a deflected turn whose mock returns a `ToolCallPiece` emits no `tool` frame at all.
|
||||||
|
- Coverage: **>90%** on `app/` — every new branch (intercept, fallback, `tools_offered=False`, `final_content == 0`) has a named test.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A cap-hitting turn produces ZERO `tool` frames after the cap and ZERO raw tool markup in the answer.
|
||||||
|
- [ ] That turn always ends with readable content (`done`, not `error`) — the fallback line appears when the model gave nothing but calls/scaffolding.
|
||||||
|
- [ ] `holder.rejected_calls` counts the attempts; `tool_calls=N` in the per-turn log still counts executed calls only.
|
||||||
|
- [ ] Non-cap `MalformedReplyError` behavior is byte-identical (deflected twice-scaffolding integration test still green).
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 03 — Surface it: `done.budget_exhausted` + the `rounds=` / `cap_hit=` log fields
|
||||||
|
|
||||||
|
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…tell the user to continue the session by allowing more tool calls…" (the server-side half: the turn must REPORT that it ran out, so the UI can offer it and the log can prove it).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A cap-reached turn reports itself: one additive `done` key (absent on every other turn) plus two per-turn log fields — so the frontend can act on it (task 04) and the owner can grep how often the cap actually bites.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — `AgentHolder` (L1482): add `rounds: int = 0` and `cap_reached: bool = False`, documented in the existing field-doc style — `rounds` is the number of rounds CONSUMED (an executed call, a refusal, or a rejected attempt's round; today it moves only with `rounds += 1`), `cap_reached` is set exactly where the forced no-tools final runs (L2270's block).
|
||||||
|
- Mirror the loop-local `rounds` into `holder.rounds` at the same statement (`holder.rounds = rounds` beside `rounds += 1`) AND once before the forced final, so a turn that ends at the cap reports the full count; do not re-derive it in the API layer.
|
||||||
|
2. `app/schemas.py` — `ChatDoneEvent` (L316): add `budget_exhausted: bool = False` with a docstring paragraph in the `answer_image` style (phase 160, TODO L4 — true ONLY on a turn that hit `BOR_AGENT_MAX_ROUNDS`; the UI's Continue affordance keys on it) and EXTEND the existing `@model_serializer(mode="wrap")` rule so the key is popped when `False`: frames of every non-cap turn serialize byte-identically to today ("ABSENT, never null" — the `answer_image` / `SourceRef.image_url` / phase-123 `image` precedent, and the byte-identical-off rule in PLAN §13).
|
||||||
|
3. `app/api/chat.py` — the `ChatDoneEvent(...)` construction (~L1265): pass `budget_exhausted=holder.cap_reached` (the grounded turn only; the deflected path builds its own done frame and never sets it — a deflected turn runs no agent, so the default `False`/absent is correct by construction).
|
||||||
|
4. `app/api/chat.py` — the per-turn log line (L1196-1219): insert `rounds=%d cap_hit=%d` immediately after `tool_calls=%d` (PLAN §9's field order is `… tool_calls=N …`, the agent fields stay grouped), with `holder.rounds` and `int(holder.cap_reached)`. Keep every existing field, its order and its format byte-identical (several tests grep this line).
|
||||||
|
5. `.agents/PLAN.md` — the ONLY plan edit of this phase (the phase-45 pattern, owner permission = the 2026-10-05 roadmap confirmation for this phase; AGENTS.md rule 3 — a recorded revision, never a silent deviation):
|
||||||
|
- §4's SSE line: `done` `{deflected, sources[], suggestions[], budget_exhausted?}` + one sentence: a `tools=None` request never emits a `tool` frame, and a round-capped turn whose final reply carries no visible content answers with the budget fallback instead of erroring (revising the phase-71 cap-path note);
|
||||||
|
- §9's required line: add `rounds=N cap_hit=0|1` after `tool_calls=N`;
|
||||||
|
- §12 is left alone (the roadmap table lists shipped phases by directory).
|
||||||
|
6. No migration, no new table, no new setting — `agent_max_rounds` keeps its default of 10 (LOCKED).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_schemas*.py` (extend the `ChatDoneEvent` serialization pins — the phase-124 answer-image tests are the template): `budget_exhausted=False` → the key is ABSENT (assert `"budget_exhausted" not in payload`), `True` → present and `true`; a full done frame with sources + related + suggestions + no flag serializes byte-identically to the pre-phase expectation (the existing golden payload test must pass untouched). `tests/unit/test_agent.py` — `holder.rounds` counts refusals as rounds today (an unknown-tool call bumps `rounds` but not `tool_calls`), and `cap_reached` is `False` on a turn that answered inside the cap.
|
||||||
|
- Integration (`tests/integration/test_chat_api.py`, extend): a cap-hitting scripted turn ends with `data: …"budget_exhausted": true…` and the captured `caplog` line contains `rounds=N cap_hit=1`; an ordinary grounded turn's `done` frame has no `budget_exhausted` key and its log line reads `cap_hit=0`; the deflected turn's line reads `rounds=0 cap_hit=0`.
|
||||||
|
- Coverage: **>90%** on `app/` (the serializer branch + both holder fields covered).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `done.budget_exhausted` is `true` exactly on cap-reached turns and absent (never `false`) on every other turn — including all deflected turns.
|
||||||
|
- [ ] The per-turn line reads `… tool_calls=10 rounds=10 cap_hit=1 …` on a capped turn; every pre-existing field is unchanged in value and order.
|
||||||
|
- [ ] `.agents/PLAN.md` §4/§9 carry the additive revision note and nothing else changed in the plan.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 04 — "Continue researching" on an answer that ran out of tool calls
|
||||||
|
|
||||||
|
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…tell the user to continue the session by allowing more tool calls (or something similar)."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
When a turn ends because its tool budget was spent, the answer bubble offers **Continue researching**; one click sends the next turn (fresh history + a whole new round budget) so a long research session keeps going without typing.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `CONTINUE_ICON` beside `TUNE_ICON` (L475) / `RETRY_ICON` (L606): an inline `aria-hidden` SVG in the same stroke family (a forward/arrow-on form, 24×24, `fill="none" stroke="currentColor" stroke-width="1.7"`) — no icon font, no CDN (A11).
|
||||||
|
2. `frontend/assets/app.js` — `const CONTINUE_QUESTION = "continue";` and `appendContinueButton(wrap)` directly after `appendRetryButton` (L617), same shape: `getActionsRow(wrap)` (L498), one-per-bubble guard on `.continue-btn`, `role="listitem"` when the row reports `role="list"`, `btn.innerHTML = CONTINUE_ICON + "<span>Continue researching</span>"`, click → `continueTurn()`.
|
||||||
|
3. `frontend/assets/app.js` — `continueTurn()`, the `retryLastTurn` (L2984) discipline:
|
||||||
|
- no-op while `uiState === UI_STATE.thinking || uiState === UI_STATE.streaming` (one turn at a time);
|
||||||
|
- clear the button from every bubble first (`messagesEl.querySelectorAll(".continue-btn").forEach(b => b.remove())`) so it can never be clicked twice into two turns;
|
||||||
|
- `return runTurn(CONTINUE_QUESTION)` — a plain send (`reask` defaults to `false`): the user bubble appears, `conversation.push({who:"user", text:"continue"})` + `saveConversation()` + `persistConversation()` are the ordinary save points, and the request carries the full `history`, so the model resumes with everything it already read and gets a fresh `agent_max_rounds`. A10 holds: nothing server-side knows or cares that this is a continuation.
|
||||||
|
- ASSUMPTION: the continuation question is the literal word `continue` (what the model is taught to ask the user for in `ROUND_CAP_WRAPUP_INSTRUCTION` / `TOOL_BUDGET_SPENT`, so the visible bubble, the typed equivalent and the model's own suggestion all agree).
|
||||||
|
4. `frontend/assets/app.js` — the `done` settle (L3600-3652): add `budget_exhausted: ev.budget_exhausted || undefined` to the `rememberBrainTurn(…)` meta object (the `related` / `answer_image` optional-key precedent — `undefined` drops the key, so every non-cap record stays byte-identical), and after the existing `markLastRetryable()` (L3649) add `if (ev.budget_exhausted) appendContinueButton(wrap);` (so the row order is Tune → Retry → Continue → Save-as-doc, the insertion order phase 130 relies on).
|
||||||
|
5. `frontend/assets/app.js` — the restore: `renderStoredMessage`'s brain branch (after L2281) gets `if (m.budget_exhausted) appendContinueButton(wrap);` — a reload keeps the affordance (a restored `history` is exactly what the continuation turn needs). `startNewChat()` (L2886) needs nothing (resetting `messagesEl` drops the buttons with the list).
|
||||||
|
6. `app/schemas.py` — `ChatMessage` (L1264ff, `extra="forbid"`): add the optional marker `budget_exhausted: bool | None = None` documented in the phase-120 `failed` / phase-48 `stopped` style (the persisted answer that ran out of tool calls — the client's restore hint, never a server behavior). No migration: `saved_chats.messages` is raw JSONB (phase 50), and `app/api/chats.py`'s logic is unchanged.
|
||||||
|
7. `frontend/assets/styles.css` — `.continue-btn` in the `.retry-btn` family (L1010: identical pill size/spacing/color, the shared `:focus-visible` ring — extend the selector list if the ring rule is per-class) + `min-height: 44px` beside `.retry-btn` in the mobile block (L5708-5709) + any `prefers-reduced-motion` parity the family needs (PLAN §7 feedback rules).
|
||||||
|
8. Verify the shared page (`/shared/<token>`, rendered server-side, not by `app.js`) shows no Continue control for a flagged record — the key is ignored there by construction; record the check in the task's commit note (no code change expected).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (house-style source pins; new `tests/unit/test_frontend_tool_budget.py`): `appendContinueButton` joins the actions row with the one-per-bubble guard and `listitem` role; `continueTurn` bails on `thinking`/`streaming`, removes every `.continue-btn` before sending, and calls `runTurn(CONTINUE_QUESTION)` with no `reask`; the `done` handler passes `budget_exhausted` through `rememberBrainTurn` and appends the button only when flagged; the restore branch keys on `m.budget_exhausted`; `retryLastTurn`, `markLastRetryable` and `rememberBrainTurn` bodies are byte-unchanged.
|
||||||
|
- Integration (`tests/integration/test_chats_api.py`, extend): a saved chat whose brain record carries `budget_exhausted: true` round-trips byte-identically on `GET /api/chats/<id>` (and on the shared endpoint) — pins the additive `ChatMessage` field.
|
||||||
|
- Coverage: **>90%** on `app/` (the only `app/` change is the schema field — covered by the round-trip test).
|
||||||
|
- Manual: with `BOR_AGENT_MAX_ROUNDS=1` (or the task-05 mock marker) confirm the answer offers Continue, one click runs a fresh turn, and a reload restores the button.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An answer from a cap-reached turn carries a working **Continue researching** button; clicking it sends exactly one new turn whose request carries the prior `history`, and its answer streams normally.
|
||||||
|
- [ ] A non-cap answer shows no such button, and neither does any earlier answer once a newer turn lands.
|
||||||
|
- [ ] The button is inert while a turn is in flight; no double-send is possible.
|
||||||
|
- [ ] A page refresh keeps the button on a flagged answer; the shared page never shows it.
|
||||||
|
- [ ] `bor.chat.v1` records of non-cap turns are byte-identical to pre-phase (no `budget_exhausted` key).
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 05 — Budget tests: the mock's round-cap flow, unit/integration pins, isolated E2E
|
||||||
|
|
||||||
|
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "The LLM need to know how many tool calls it has left. During long research sessions it will sometimes end the conversation with tool calls that just get dumped into chat because it ran out of turns…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The whole contract is pinned end to end: a deterministic mock flow that runs a turn to the cap and then tries to dump tool calls into chat, the unit/integration assertions for the trailer / rejection / flag, and the phase's own Playwright file run in isolation (AGENTS.md rule 9).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/mock_llm.py` — a new deterministic flow, documented in the module docstring's marker table:
|
||||||
|
- `ROUND_CAP_TRIGGER = "run out of tool calls"` — detected on the LAST user message only (the mock's existing invariant: the last `user` message is the current question, the earlier non-system messages are prior turns — see the `last_user` scan around L3186);
|
||||||
|
- while tools are offered (`"tools"` present on the request) and no tool result answers the last emitted call: stream ONLY `tool_calls` deltas — `ls` with no arguments, incrementing id `call_cap_1`, `call_cap_2`, … , `finish_reason: "tool_calls"`, no content — i.e. the model never stops asking, so the turn is guaranteed to hit `BOR_AGENT_MAX_ROUNDS`;
|
||||||
|
- on the request WITHOUT tools (the forced final) in this flow: stream a `tool_calls` delta anyway (`read` on the seed path, id `call_cap_dump`) and NO content — the exact dumped-call shape the phase kills, and the shape `SCAFFOLD_TRIGGER` (L1375) does not cover (that one emits textual markup, this one is structured);
|
||||||
|
- when the last user message is exactly `continue` and an earlier message carries the trigger: stream a stable answer `ROUND_CAP_CONTINUE_ANSWER` (a fixed sentence naming the seed document, so the E2E asserts a real continuation answer rather than a tool call).
|
||||||
|
- The new branch must not disturb any existing trigger (the phase-45 multi-read flow, `TOOLS_TRIGGER`, `SCAFFOLD_TRIGGER`, the failure markers) — the existing mock-only suites are the regression gate.
|
||||||
|
2. `tests/e2e/test_tool_budget_telemetry.py` (new, MOCK-ONLY — no `E2E_REAL_LLM` support, the phase-45/120 files are the template: `app_server` / `page` / `db_ready` fixtures, `auth_helpers.login`, the TRUNCATE-then-seed KB pattern over `tests/fixtures/docs/homelab/kubernetes.md`, the autouse `saved_chats` truncate):
|
||||||
|
1. `test_tool_results_carry_the_remaining_count` — server-side pin at the wire level: run the cap flow and read the mock's recorded requests (the mock exposes its request log; otherwise assert through the app's captured log lines) that each successive tool message ends with `[tool calls remaining: N]` counting down and the last says no tool calls remain.
|
||||||
|
2. `test_capped_turn_never_shows_tool_lines_after_the_cap` — with `BOR_AGENT_MAX_ROUNDS=3` (the `app_server` env override the existing suites already use for tuning): the chat shows at most 3 tool lines total, the dump call produces NO tool line, the answer bubble holds non-empty readable text (the fallback or the coached answer), the turn settles with `done` (never the error banner), and the SSE capture has no `error` frame.
|
||||||
|
3. `test_done_flags_budget_exhausted` — the captured SSE stream's `done` frame carries `budget_exhausted: true`; a normal grounded turn's `done` carries no such key (byte-identical control case in the same file).
|
||||||
|
4. `test_continue_button_runs_a_fresh_turn` — the capped answer bubble shows **Continue researching**; clicking it appends exactly one user bubble `continue`, the mock answers with `ROUND_CAP_CONTINUE_ANSWER`, the earlier tool lines do not duplicate, and `bor.chat.v1` holds `[…brain(flagged), user("continue"), brain]`.
|
||||||
|
5. `test_continue_is_inert_while_streaming_and_single_shot` — clicking Continue twice fast (or during the streamed turn) starts ONE turn (count the `/api/chat` requests via `page.on("request")`).
|
||||||
|
6. `test_flag_survives_a_refresh` — `page.reload()` after a capped turn: the flagged answer restores WITH the Continue button and clicking it still works.
|
||||||
|
7. negative `test_deflected_turn_gets_no_continue_button` — a deflected question (A8 LOW mode) settles deflected with no `budget_exhausted` key and no Continue button.
|
||||||
|
- The file's docstring states: the markers, the `BOR_AGENT_MAX_ROUNDS` override used, the seed, and this numbered test → contract mapping (the phase-120 file's shape).
|
||||||
|
3. `tests/unit/test_tool_budget_note.py` (new, if `tests/unit/test_agent.py` gets too long for the reviewer's taste — otherwise extend that file's round-cap cluster L4561): `_budget_note` boundaries, the trailer's presence on executed AND refused calls, `agent_max_rounds=0` producing no tool message at all.
|
||||||
|
4. `tests/unit/test_frontend_tool_budget.py` — already written in task 04; extend it with anything task 04 left unpinned rather than duplicating it here.
|
||||||
|
5. Regression gate (each file in isolation, `--no-cov`): `test_agent_unlimited_tools.py` (the multi-read flow must still stream its 3 tool lines and answer), `test_agent_document_tools.py`, `test_harness_aligned_tools.py`, `test_grep_regex_teaching.py`, `test_big_read_progress.py` + `test_read_range_tool_lines.py` (the `tool_result` frame path the trailer rides next to), `test_chat_rag.py`, `test_scaffolding_guardrails.py` (or its current name in `tests/e2e/`), `test_smoke.py`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Order: `uv run pytest tests/unit -v --no-cov` → `uv run pytest` (unit + integration) → `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%) → `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` (DB up) → the regression list above, isolated.
|
||||||
|
- Every new branch of tasks 01-04 must be attributable to a named test in the coverage report (`--cov-report=term-missing` shows no new red lines in `app/rag/agent.py`, `app/api/chat.py`, `app/schemas.py`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` green in isolation.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90% with no uncovered new lines in the touched modules.
|
||||||
|
- [ ] Every regression E2E file listed above green in isolation.
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `.agents/PLAN.md` shows exactly the task-03 revision note; the phase dir moves to `.agents/phases/complete/` with the single `--no-gpg-sign` commit (`00_phase.md` §Commit).
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user