Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 192_verbatim_inline_midturn
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Details
Build and Push Containers / build-and-push-db (push) Successful in 14s
Details

Browse Source 
Phase 192 — final verification pass complete.

- **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite.
- **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**.
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%)
- `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated)
- `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts)
- `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors**
- **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run).
- **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream).
- **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened.
- **Next pending phase:** `193_sync_remaining_count`.
This commit is contained in:
ducoterra
2026-10-19 12:19:51 -04:00
parent 4f20c256d7
commit fb17d15fbe
49 changed files with 17313 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/complete/192_verbatim_inline_midturn/00_phase.md
+58
View File
@@ -0,0 +1,58 @@
# Phase 192 — Verbatim panels land while the answer is still streaming
**Source:** `TODO.md` L5 — "The verbatim tag appear at the end and only when the llm is done answering which makes the chat read strangely."
**Story:** n/a (owner report)
**Depends on:** none functionally (builds on the completed phase-182 verbatim display, phase-95 tool_result frames, phase-158/164 saved-chat restore).
## Evidence (diagnosed from the code, read this not the chat)
1. **Panels materialize only on the `done` frame.** `app/api/chat.py` resolves the turn's verbatim blocks at done-time (L1319–1347: `resolve_answer_blocks` over `answer_text` → the done frame's additive `answer_text` list); `app.js` hydrates at done (L4506–4511: `hydrateVerbatim(bubble, ev.answer_text, …)` + `appendUnlocatedVerbatim(wrap, ev.answer_text, …)`) and at restore (L3044–3046, the record's `m.answer_text`).
2. **The `show_text` result already exists mid-turn — it just isn't sent.** The agent's show_text branch (app/rag/agent.py L2238–2340) appends the resolved `VerbatimBlock` to `holder.shown_texts` (call order, the `show_text_max_blocks` budget) the instant the call succeeds, and the result text rides the round's `tool` message — but the block itself reaches the client ONLY in the done frame.
3. **So the reading order inverts.** The prose streams first; every verbatim panel (fenced or not) pops in at the END, after the model has already finished talking about the bytes: "…here is the doc: [model finishes] [panel appears]". The unfenced blocks are the worst case — `appendUnlocatedVerbatim` appends them at the answer's tail (the `answer_image` position), detached from the sentence that introduced them.
4. **The fix is additive.** The block is already in hand at the success point (the `shown_texts` append, L2322 area); the round loop already has the snapshot-then-yield pattern for phase-95 truncations (`trunc_before`, L2640 → the `ToolResultPiece` yield at L2690). One new piece + one new SSE event + one client branch; the `done` frame, the record key, the restore and the shared page keep their exact contract.
## Design (shared by all tasks — the executor reads this, not the chat)
- **A1 — the mid-turn frame (task 01).** A new additive SSE event `verbatim_block`, emitted as soon as a `show_text` call succeeds:
- `app/rag/agent.py`: a new `@dataclass VerbatimBlockPiece` (the `ToolResultPiece` pattern, beside it; the `block: VerbatimBlock` field — the class is already imported at L343). In the round loop: `shown_before = len(holder.shown_texts)` BEFORE `_execute_tool` (the `trunc_before` pattern, L2640); after the round's `ToolResultPiece` yield (L2690), `for block in holder.shown_texts[shown_before:]: yield VerbatimBlockPiece(block=block)` — the frame lands AFTER the round's `tool_result` frame and BEFORE the next model round. SUCCESS-ONLY by construction: every refusal path returns before the `shown_texts.append` (L2322), and the already-shown dedupe is a refusal — a refused call yields no piece. The generator union types (L928/L992) gain the piece.
- `app/schemas.py`: `ChatVerbatimBlockEvent` (the `ChatToolResultEvent` pattern, L321) — `type: Literal["verbatim_block"]` + the EXACT `VerbatimBlockRef` field set (L185: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `start_line` / `end_line` / `total_lines` / `text` ≤ 600 000 / `truncated` / `locator` ≤ 2100) — the client's `isVerbatimBlock` check (verbatim.js L70) accepts the event as-is; the framework's `extra` convention of the sibling events is mirrored.
- `app/api/chat.py`: the new dispatch branch after the `ToolResultPiece` branch (L1104): `if isinstance(piece, VerbatimBlockPiece): yield sse_event(ChatVerbatimBlockEvent(**asdict-style fields).model_dump()); continue`. The done-frame `answer_text` resolution (L1319–1347) and the per-turn log line (`verbatim_blocks=N`, done-frame count) are UNTOUCHED — the mid-turn frames are not counted there (the pin stays green).
- KILL-SWITCH: `BOR_ANSWER_TEXT=0` → the tool is not offered (the phase-182 D8 posture) → no success → NO piece, NO frame: the frame sequence is byte-identical to pre-phase (a wire pin, task 01 + the E2E arm).
- **A2 — the client lands the panel mid-stream (task 02).**
- `frontend/assets/verbatim.js`: a NEW `hydrateVerbatimNow(root, block, opts)` — walks the PENDING placeholders (`.verbatim-pending[data-bor-locator]`); for each, `matchVerbatimBlock([block], key)` (the SAME matching semantics as the done path — the exact locator, or the identity fallback when the list holds exactly one block of that identity); REPLACES ONLY the matched nodes with `buildVerbatimPanel(block, opts)`; the UNMATCHED pending nodes STAY PENDING (settling them is the done/failure path's job — a mid-turn pass must never burn a placeholder for a not-yet-shown fence); returns the number matched (0 → the caller appends). Never throws (the house contract — a missing `root` → 0).
- `frontend/assets/app.js`: the new `ev.type === "verbatim_block"` branch beside the `tool_result` branch (L4612): `const n = hydrateVerbatimNow(bubble, ev, VERBATIM_LINK_OPTS); if (n === 0) appendUnlocatedVerbatim(wrap, [ev], VERBATIM_LINK_OPTS);` — a fenced block lands IN PLACE at its fence (the model said "here it is" → the panel is there); an unfenced block lands at the `answer_image` position (the `appendUnlocatedVerbatim` contract — below the current text, above the sources) AT THE MOMENT the call succeeds, not at the end of the answer. `appendUnlocatedVerbatim`'s state-free idempotence (same canonical locator already placed → skip, L463–472) makes the done pass's re-run a no-op for the same block. No client-side kill-switch gate: the server's gate is the only one (a `False` turn sends no frame — the branch never fires).
- **A3 — the done/restore dedupe (task 03).** The one remaining duplicate: a mid-turn APPENDED panel (locator A) + the model later fences the SAME locator A → the fence placeholder hydrates at done into a second panel. `hydrateVerbatim` (the done AND restore path — L423) gains the placed-scan: before building a matched panel for a `key`, if a PLACED resolved panel (``.verbatim[data-bor-locator]`, NOT `.verbatim-unresolved` — the `appendUnlocatedVerbatim` placed-scan convention) carrying the same canonical `locator` exists in the message wrap (`root.closest(".msg")`), that placed panel is REMOVED first — the in-text panel wins (the fence is where the model wanted it; the appended panel has served its mid-turn purpose). TWO DIFFERENT ranges of the same file are LEGITIMATE (the `shown_texts` dedupe is per canonical locator, not per identity) and stay — the scan is per-locator, never per-identity. The restore path (L3044–3046) and the shared page ride the SAME functions — no separate change; the parity pin is the test.
- **A4 — the unchanged contract (pinned across tasks).** The `done` frame's `answer_text` list, the saved-record key, the restore path and the shared page are byte-identical to phase 182 for a turn that never calls `show_text`; a `BOR_ANSWER_TEXT=0` turn is byte-identical end to end (no frame, no placeholder, no panel); the `verbatim_blocks=N` per-turn log line counts the done-frame blocks as before.
## Context
- `app/rag/agent.py` — `VerbatimBlock` (imported L343), `holder.shown_texts` (L1848), the show_text branch (L2238–2340), the round loop (L2640–2697), the piece union types (L928/L992); `app/api/chat.py` — the piece dispatch (L1096–1130), the done-time resolution (L1319–1347); `app/schemas.py` — `VerbatimBlockRef` (L185), `ChatToolResultEvent` (L321), the `extra` convention; `app/rag/verbatim.py` — `VerbatimBlock` (L94), `block_for`, `canonical_locator` (D7).
- `frontend/assets/verbatim.js` — `isVerbatimBlock` (L70), `matchVerbatimBlock` (L404), `hydrateVerbatim` (L423), `appendUnlocatedVerbatim` (L447), `settleVerbatimPlaceholders` (L480); `frontend/assets/app.js` — the `tool_result` branch (L4612), the done branch's hydrate/append (L4506–4511), the restore hydrate/append (L3044–3046).
- `tests/e2e/mock_llm.py` (the mock's show_text flow — the phase-182 E2E precedent; verify the exact mock arm), the phase-182 E2E suites (verify the names in `tests/e2e/`), the JS marker-pin unit module (verify the name — the `test_save_chat_ui.py` convention), `tests/integration/` the chat SSE contract module (verify the name).
## Tasks
1. `01_verbatim_block_sse.md` — the `VerbatimBlockPiece`, the `ChatVerbatimBlockEvent`, the chat dispatch branch; the agent mock pins, the SSE contract pins, the kill-switch wire pin.
2. `02_midturn_hydration_and_append.md` — `hydrateVerbatimNow` + the app.js `verbatim_block` branch (in-place fence hydration at mid-turn; unfenced blocks land at the call moment; unmatched placeholders stay pending).
3. `03_done_and_restore_dedupe.md` — the `hydrateVerbatim` placed-scan dedupe (per-locator; two ranges stay; the restore/shared parity pin; the kill-switch and the `verbatim_blocks=N` pins stay green).
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the mid-turn arm, the fence arm, the no-double arm, the kill-switch wire arm, the restore arm), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the agent mock pins (a successful `show_text` → exactly one `VerbatimBlockPiece` after the round's `tool_result` and before the next round; the five refusal paths → zero; the `answer_text=False` posture → zero); the `VerbatimBlockPiece`/`ChatVerbatimBlockEvent` shape pins; the JS marker pins (the `hydrateVerbatimNow` matched-only contract, the app.js branch).
- **Integration:** the chat SSE contract — a mock turn with `show_text` → the stream carries the `verbatim_block` frame (the exact `VerbatimBlockRef` fields) before `done`; a `BOR_ANSWER_TEXT=0` app → the captured frame sequence is byte-identical to the pre-phase sequence (the kill-switch wire pin); the done frame's `answer_text` is unchanged (the phase-182 pins stay green).
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_verbatim_inline_midturn.py`, run in isolation — the owner-visible arm: the panel is on screen WHILE the answer is still streaming (before `done`), the fence hydrates in place mid-turn, no double panel, the kill-switch wire arm, the restore parity.
- **Regression:** the phase-182 E2E suites green unedited; `tests/e2e/test_chat_history.py` (the restore contract) green.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the verbatim section notes the mid-turn landing (a `show_text` panel appears the moment the call succeeds — in place when fenced, at the answer position when not; the done/restore contract is unchanged) (task 04). `AGENTS.md`: the phase-182 clause gains the mid-turn frame clause if it names the done-frame resolution (verify, do not add if absent).
## Completion Criteria
- [ ] a successful `show_text` call yields exactly one `verbatim_block` frame (the unit + integration pins); every refusal yields none.
- [ ] a `BOR_ANSWER_TEXT=0` turn's frame sequence is byte-identical to pre-phase (the wire pin) — no frame, no placeholder, no panel.
- [ ] a pending fence placeholder for a shown block hydrates IN PLACE at mid-turn (before `done`); an unfenced block is on screen at the call moment (the E2E arms).
- [ ] a mid-turn appended panel + a same-locator fence → exactly ONE panel after `done` (the dedupe pin); two different ranges of one file → two panels (the pin).
- [ ] the restore path shows exactly one panel per locator after a reload (the E2E arm); the `done` frame's `answer_text`, the record key and the `verbatim_blocks=N` log line are unchanged (the phase-182 pins green unedited).
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed phases (a turn without `show_text` is byte-identical end to end).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.