Compare commits

..
79 Commits
Author SHA1 Message Date
ducoterra a45f744604 phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
2026-10-20 06:36:21 -04:00
ducoterra ae080ac7f3 phase: 193_sync_remaining_count
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 16s
**Phase 193 — final verification pass: all gates green.**

- **Verified (all 3 tasks already complete in `complete/193_sync_remaining_count/`):** `fmtSyncLabel`'s count part is `total > 0 && total - done > 0 ? \`${total - done} remaining\` : ""` (plain int, hidden at `N==0`, `total===0` → no part); docstrings updated; chunk/image/summaries parts + `counts_line` terminal + `app/` byte-untouched (`git diff --stat`: no `app/` files).
- **Completion criteria:** `N remaining` shape + marker pins ✅ · upload shares the builder (E2E upload arm) ✅ · chunk/image/summaries pins green ✅ · terminal `counts_line` green unedited + A4 title/`#sync-progress` pin ✅ · suite green, coverage 99% ✅ · lint/types clean ✅ · commit + `todo/→complete/` left to the harness (no git index/history commands run).
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, junit: **4792 tests, 0 failures/errors/skipped**, `TOTAL 7181 16 → 99%` (>90%)
- `uv run pytest tests/e2e/test_sync_remaining_count.py -v --no-cov` → **3 passed** (isolated, 53 s)
- Isolated `--no-cov` sweep of the 14 sync-lineage suites (label_mobile, upload_progress, chunk_progress, image_progress, progress_box, summary_visibility, archive_upload_sources, status_freshness, upload_no_scan, peak_memory_visible, session_reset, fast_path_deferred_content, git_freeze, smoke) → **14/14 exit 0**
- `uv run ruff check .` → All checks passed · `uv run pyright` → **0 errors**
- **Docs:** README.md reviewed in full (all 2408 lines) + AGENTS.md. Fixed 2 inconsistencies: the sync running-label clause claimed the upload label shows the count — now states it shares the ONE builder/shape but reports no counts since phase 90 (renders bare); phase-190's "the other ~72 rows" → "the other 83 rows of the panel's 86-row catalogue" (mechanically: 86 CATALOG rows, 84 overridable). `N remaining` already in the sync paragraph; task-01's `(n/m)` examples gone. AGENTS.md verified consistent (no clause names the running count part → unchanged per the phase's "do not add if absent"). Mechanically re-verified: all 86 `BOR_*` settings documented + defaults match, `.env.example` 88 vars covered, 0 broken README anchors, all cited file paths exist, 17 palette colors, migrations head `0030`, pages/shell routes, ports 8123/8901, E2E port allocations (new suite's 8211/8914 unique).
- **Notable:** task 02's documented deviation stands — `files_done` is the importer's 1-based *current* file, so the phase doc's `[1-4]` range is unreachable; count-bearing frames are `1 … N−1` (pinned). Pre-existing, untouched: `E2E_APP_PORT_TOOLBUDGET`/`SAVEDOC` both 8129 and `SYNCUP`/`SYCNCEL` both 8133 — harmless under the one-suite-at-a-time rule.
- **Next pending phase:** none — `193_sync_remaining_count` is the last entry in `.agents/phases/todo/`.
2026-10-19 17:30:03 -04:00
ducoterra fb17d15fbe phase: 192_verbatim_inline_midturn
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 192 — final verification pass complete.

- **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite.
- **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**.
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%)
- `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated)
- `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts)
- `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors**
- **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run).
- **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream).
- **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened.
- **Next pending phase:** `193_sync_remaining_count`.
2026-10-19 12:19:51 -04:00
ducoterra 4f20c256d7 phase: 191_source_name_labels_tags
Build and Push Containers / build-and-push-app (push) Successful in 7m24s
Build and Push Containers / build-and-push-db (push) Successful in 36s
All gates re-verified against a stable tree. Report:

**Verified (all 6 tasks in `complete/`; this was the final pass)**
- `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`.
- **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned).
- **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code.
- **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria.
- **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
2026-10-19 06:13:32 -04:00
ducoterra c6ffa050a6 phase: 190_app_name_settings_fix
Build and Push Containers / build-and-push-app (push) Successful in 2m9s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates re-verified green. Report:

- **Verified (all 4 tasks already complete; this was the final pass):** `effective_settings` resolves the 3 brand strings panel-row → tab-row → env (`applied_override_keys`, presence-not-value); `PUT /api/ui-settings` ignores string body fields; Theme-tab form is colors-only (17 pickers, no `#theme-app-name`/`#theme-placeholder`/`#theme-footer`); the ONE shared `applyBrandFromConfig` re-applies title/`.brand-text`/footer/placeholder after a panel save-reset and a tab save-reset; boot path byte-identical (single `/api/config`).
- **Completion criteria:**
  - strings order panel→tab→env, both branches — ✅ `tests/unit/test_theming.py::test_effective_app_name_prefers_settings_panel_row_over_ui_settings_row`, `..._fall_back_to_ui_settings_row_without_panel_row`, `..._per_field_panel_then_tab_then_env`, `..._ignore_panel_rows_when_the_overlay_is_off`
  - PUT ignores string fields + no 3 inputs in the tab — ✅ `tests/integration/test_ui_settings_api.py::test_put_ignores_string_fields_in_body` (+2) and the E2E `test_theme_tab_dom_is_colors_only`
  - live apply without reload + incident regression — ✅ `tests/e2e/test_app_name_settings_fix.py` tests 1–4
  - byte-identical boot, no double-apply — ✅ `tests/unit/test_brand_live.py` markers + E2E "exactly one /api/config"
  - suite green, coverage >90%, ruff+pyright clean — ✅
  - no behavior change in prior phases — ✅ (palette, phase-168 panel, env fallback suites all green)
  - atomic commit — **left to the harness** (pipeline rule overrides per-phase committing; no git write run)
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **4630 passed, TOTAL 6941 16 99%**, exit 0 · `uv run pytest tests/e2e/test_app_name_settings_fix.py -v --no-cov` → **6 passed** in isolation · regression E2E (isolated, one process each, all exit 0): `test_smoke` `test_configurable_brand` `test_admin_theme_tab` `test_theme_save_and_coverage` `test_theme_semantic_completion` `test_themed_favicon` `test_dark_tech_theme` `test_admin_settings_panel` `test_admin_settings_layout` `test_settings_mobile_value_width` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Documentation:** README — fixed "saving one repaints *every* open page" (only the open document repaints; other pages on next load) and a broken paragraph wrap in *Customizing the look*; AGENTS.md rule 16 — `test_chunks_hnsw_recall.py` now described accurately (index paths forced with `enable_seqscan`/`indexonlyscan`/`bitmapscan` off, unforced plan asserted only on a pristine relation); `tests/unit/test_theme_frontend.py` — two stale section headers ("20 form fields"/"20 inputs" → 17 color). No AGENTS.md settings-overlay/resolution-order clause exists, so per the phase none was added. Full env-var cross-check (`BOR_*` in `app/config.py` vs README) → 0 gaps; 0 missing file references.
- **Notable:** the working tree also carries this phase's flake hardening of the phase-188 `tests/integration/test_chunks_hnsw_recall.py` (verified legit and green). Live infra untouched; shared DB left clean (0 `app_settings`, 0 `ui_settings`). **Flagged, not changed:** `settings_overlay.applied_override_keys` lets a `ProgrammingError` (missing `app_settings`) propagate from public `GET /api/config`, where `load_overrides` deliberately fails soft — only reachable on an un-migrated (0027) DB, outside this phase's pinned A1 semantics, so left for the owner to rule on rather than silently deviating.
- **Next pending phase:** `191_source_name_labels_tags`.
2026-10-18 02:59:19 -04:00
ducoterra 47456a44df chore(agent): phase roadmap from TODO.md, 4 phases (190-193)
Convert the four open TODO items into executable phases (append mode,
starting at 190 after the completed 189):

  190_app_name_settings_fix  - the Settings-panel app name wins over the
      Theme tab; the three brand strings stripped from the Theme tab
      (colors-only); live brand re-apply on save, no reload
  191_source_name_labels_tags - source_labels table + admin API + Sources
      page editor; the separate bounded source_name_hit_bonus retrieval
      boost; label surfacing in the <document> blocks and KB overview
  192_verbatim_inline_midturn - an additive verbatim_block SSE frame at
      the show_text success moment; in-place mid-turn fence hydration;
      the done/restore per-locator dedupe; the done frame, record key,
      restore and shared page stay byte-identical
  193_sync_remaining_count - the running sync/upload label counts down:
      "N remaining" (N = total - done), hidden at the final file; the
      chunk/image/summaries/terminal parts untouched

TODO.md is cleared (its items now live in .agents/phases/todo/).

LLM-Generated: true
2026-10-17 19:35:55 -04:00
ducoterra 140a343b92 phase: 189_chat_save_long_turns
Build and Push Containers / build-and-push-app (push) Successful in 3m4s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 189 final verification — all green**

- **Verified (all 4 tasks already implemented):** A1 caps in `app/schemas.py` — `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; `text`/`thinking` 600 000, `error`/`image` 500, `answer_text` 50, `messages` 500, 8 M budget all UNCHANGED; rationale docstrings rewritten to `merge_cited_docs`/`agent_max_rounds`. A2 `shareCurrentChat` linked branch upserts (PUT, 404→create+re-link) before `POST /{id}/share`, aborts on failure (banner, no token); unlinked branch byte-identical. A3 `savePending` + `finally` re-run. A4 `saveFailedSticky` + one 5 s `setTimeout` retry on network/5xx only, 4xx never retries.
- `uv run pytest --cov=app --cov-report=term-missing` → **4609 passed, 1 warning, EXIT=0**, `TOTAL 99%`, `app/schemas.py 100%` (>90%).
- `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` → **4 passed, EXIT=0** (isolation).
- Regression green & **unedited** (`git status` clean): `test_chat_save_payload_limits.py` 2, `test_chat_history.py` 5, `test_share_chat.py` 4. Extra blast-radius sweep (share/save/persist/segments surface, all EXIT=0): `test_save_share_ux`, `test_stale_saved_chats`, `test_history_pagination`, `test_history_copy`, `test_bottom_chat_actions`, `test_reasoning_bubble`, `test_tool_budget_telemetry`, `test_verbatim_show_text`, `test_smoke`. No leaked `saved_chats` rows (`save-long-%` = 0).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- **Completion criteria:** caps ✓ · incident payload 22/33/51 → 201/200 anonymously + 201 tools/451 segments/101 suggestions → 422 naming the offender, row unchanged ✓ · share-saves-first + abort + unlinked byte-identical ✓ · overlap re-run + sticky note + one-shot retry ✓ · phase E2E in isolation + 3 regression suites unedited ✓ · full suite / coverage / lint / types ✓ · `git diff --stat` = `app/schemas.py`, `frontend/assets/app.js`, 3 test files, `README.md` + new E2E & phase files (no `app/api/chats.py`, no migration, no new dependency) ✓ · commit + `todo/`→`complete/` move left to the harness (rules forbid git index writes).
- **Docs:** full README review (all 2 258 lines) + machine cross-check — all 85 settings documented in the Configuration reference, all 34 `.env.example` vars documented, documented defaults match `app/config.py`. Fixed 2 inconsistencies: (1) the 117-char prose line task 04 left when reflowing the Saved-chats pager paragraph; (2) the phase-189 sentence implying the 8 M total bounds the widened arrays — now states precisely that the total counts `text`/`thinking`/verbatim-block text only, and that the arrays are bounded by fixed item COUNT (matches `_require_within_total_budget`). AGENTS.md verified consistent — it enumerates no save contract or list cap (rule 5's phase-166 in-flight-turn clause untouched), so **no AGENTS.md change** per the phase mandate.
- **Notable/deviation:** the wider caps raise the *theoretical* anonymous-POST worst case (arrays are outside the 8 M char budget); left as designed (A1 fixes every other bound), documented honestly in README — realistic payloads stay ~10 MB, so the existing `client_max_body_size ≥ 16 MB` guidance still holds. No live-infra changes were made (data-repair of the two stale shared rows stays owner action).
- **Next pending phase:** none — `189_chat_save_long_turns` is the last entry in `.agents/phases/todo/`.
2026-10-17 07:36:28 -04:00
ducoterra f20d40e4fe phase: 188_chunks_hnsw_index
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00
ducoterra c054b47df5 phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
2026-10-16 15:22:51 -04:00
ducoterra 644f7de696 phase: 186_git_log_streaming_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Final report — phase 186 `git_log_streaming_dates`

**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.

### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |

Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.

### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.

Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.

⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).

Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
2026-10-16 07:50:16 -04:00
ducoterra 632fb88e43 phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
2026-10-16 04:54:01 -04:00
ducoterra 065304b52f phase: 184_sync_bounded_session
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.

**Verified / implemented (phase 184 final pass)**
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.

**Tests / lint / coverage (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria**
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
6. Phase E2E green in isolation → ✅
7. README + AGENTS accurate/consistent → ✅
8. Atomic commit — left to the harness (I ran no git write commands).

**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).

**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.

**Next pending phase:** `185_sync_defer_content_fast_path`.
2026-10-15 23:19:27 -04:00
ducoterra 7b5ea7c2ee phase: 183_sync_passive_delete_cascade
Build and Push Containers / build-and-push-app (push) Successful in 2m7s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green. Final state verified.

**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).

**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors

**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).

**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).

**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.

**Next pending phase**: `184_sync_bounded_session`.
2026-10-15 16:00:28 -04:00
ducoterra b34b8ca672 phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
2026-10-15 08:33:12 -04:00
ducoterra 814a400e6d phase: 181_svg_document_preview
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Successful in 15s
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`![alt](/api/documents/<id>/image)` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
2026-10-14 22:54:43 -04:00
ducoterra 6880bd0866 phase: 180_markdown_inline_links_images
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All verification complete. Final state summary:

**Report — phase 180 final verification pass**
- Verified (all already implemented in working tree): `inline()` chain = code→strong→em→linked-image→image→link→headings→lists in `frontend/assets/markdown.js`; `allowLinkUrl`/`allowImageUrl` allowlists (A5), shared `mdAnchorOpen` (target/rel only for absolute http(s)); `####`→`<h5>`, `#####`/`######`→`<h6>`, `#`/`##`→`<h3>`/`###`→`<h4>` unchanged; `.md-link`/`.md-img`/h5-h6 scale in `styles.css` (palette vars only). All surfaces (chat, thinking, restore, viewer, modal) route through the one renderer (pinned in `app.js`/`document.js`).
- Completion criteria:
  - `[text](url)`→`a.md-link`, `javascript:`/`data:` inert — ✅ (node probe + unit + E2E)
  - `![alt](src)`→`img.md-img loading="lazy"` with `alt`; bad src inert — ✅
  - `####`/`#####`/`######` ladder; shallow rungs unchanged — ✅
  - Fenced link/image/`####` stay code; cell-level link renders in `.md-table` — ✅ (node probe)
  - Byte-identity fixture + phase-44/138 suites pass unedited (git-verified untouched; 6 + 3 E2E pass in isolation) — ✅
  - `uv run pytest --cov=app --cov-report=term-missing`: **4198 passed, TOTAL 99%**; `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`: **4 passed**; `uv run ruff check .`: clean; `uv run pyright`: **0 errors** — ✅
- Docs: full README/AGENTS.md review vs the whole project — env-example↔README table complete, page routes match `app/main.py`, AGENTS quick-ref artifacts exist. Fixed 1 inconsistency: README understated the image allowlist (impl per task 02 accepts first segments ≤2 chars, `^.{0,2}/`, not only `./`/`../`); phase-180 paragraphs otherwise accurate.
- Notable: image allowlist `.{0,2}/` is broader than the overview's `^\.{1,2}/` but is what task 02 explicitly specifies and the docstring documents — not a deviation from the tasks; SVG sizing deliberately deferred to phase 181.
- Next pending phase: `181_svg_document_preview`.
2026-10-14 16:03:36 -04:00
ducoterra 3d82a72b6f phase: 179_git_clone_server_freeze
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; working tree holds exactly the phase's changes for the harness to commit.

## Phase 179 — Final verification report

- **Verified (all 3 tasks already complete in `complete/`)**: repro + CONFIRMED verdict, `asyncio.to_thread` fix, dedicated E2E suite.
- `app/api/sync.py::_run_sync`: both git ops (`clone_or_pull`, `file_commit_dates`) now `await asyncio.to_thread(...)`; phase-134 boundary note + module docstring document worker-thread-runs-to-completion and the A3 DB-calls-stay-on-loop note.
- **Tests / lint / coverage** (full suite 100% green):
  - `uv run pytest tests/integration/test_sync_git_freeze.py -v` → **4 passed** (post-fix probe <1.5 s during 3 s stub)
  - `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` → **2 passed in 25.65 s in isolation** (wall guard 120 s holds)
  - `uv run pytest --cov=app --cov-report=term-missing` → **4173 passed**, `app/` coverage **99%** (`app/api/sync.py` 100%)
  - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
- **Completion criteria**: ① integration probe green + verdict (CONFIRMED, mechanism, measured 3.04 s→0.003 s) in docstring — **met**; ② off-loop git ops + boundary/A3 notes — **met**; ③ full suite green, coverage 99% >90%, ruff+pyright clean — **met**; ④ E2E green in isolation (responsive-poll + cancel-during-clone contracts) — **met**; ⑤ atomic commit + move to `complete/` — **harness-owned** (pipeline rule overrides per-phase self-commit; all changes staged-ready in the working tree).
- **Documentation**: full README (1526 l) + AGENTS.md review — phase-179 bullets present and accurate (verified against fixture constants, `.env.example` fully covered by the config reference, pages table matches `app/main.py` shell routes, quick-reference commands/artifacts exist); **consistent — no further changes needed**.
- **Notable**: task-03 pipeline sessions died on context overflow but their harness validation passed; this pass independently re-ran every gate to confirm. No live infra touched; no git index commands run.
- **Next pending phase**: `180_markdown_inline_links_images`.
2026-10-14 13:06:39 -04:00
ducoterra 2b57180998 chore(repo): remove stray image.png screenshot artifact
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
The root-level image.png was committed accidentally during phase
160 (tool budget telemetry) as a bug-report screenshot. It is not
served or referenced as an asset by the app; the remaining
"image.png" mentions in tests and comments only describe the
owner's screenshot in prose, so removing the file is safe.

LLM-Generated: true
2026-10-14 10:41:04 -04:00
ducoterra 76887639f3 docs(phases): add phase 182 (verbatim file display)
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Answers to "show me my pi.dev config" re-type file contents from context,
so the code block in the answer can differ from the indexed file by a
character and cannot be trusted for copy-paste — the owner opens the
source bubble and ignores the answer.

Phase 182 makes the server the only party that supplies bytes:

- show_text(path, offset?, limit?), the display sibling of show_image,
  slices documents.content and hands the same bytes back to the model
  with a "do not retype" instruction
- inline ```bor source/path 12-40 locator fences place a resolved block
  anywhere in the prose; the model's fence body is never painted, and an
  unresolvable locator says so instead of showing guessed text
- the bytes ride inline in the done frame's additive answer_text array
  and persist in the saved chat, so restore and the shared page match
- line numbers are always the server's resolved extent (overshoot
  clamps, past-the-end refuses to the model), which removes the
  "did the model invent those numbers?" problem entirely
- a Copy button on every answer code block, a VERBATIM badge only on
  server-resolved ones
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the
  flag off every frame, record and byte budget is exactly pre-phase

Overview plus eight tasks, each with its own test gate; the phase E2E
runs in isolation and asserts the clipboard content equals the stored
bytes. Run 180 first — both touch markdown.js.

LLM-Generated: true
2026-10-14 10:33:06 -04:00
ducoterra 50658580e9 chore(agent): phase roadmap from TODO.md, 2 phases
Convert the 3 open TODO.md items into sequential phases appended to the
existing pipeline (Protocol B, numbered after 179):

- 180_markdown_inline_links_images: the shared escape-first renderer
  gains the [text](url) link transform, the ![alt](src) image
  transform, and the ####-###### heading levels, with CSS and the
  dedicated Playwright suite.
- 181_svg_document_preview: SVGs index as vector image documents
  (text extracted from title/desc/text, no vision call), are served as
  sandboxed image/svg+xml, and preview as graphics instead of raw XML.

TODO.md is cleared; its items now live in .agents/phases/.

LLM-Generated: true
2026-10-14 09:54:01 -04:00
ducoterra dad54a2854 chore(agent): phase roadmap from TODO.md (179 git clone server freeze)
TODO L3 (a long git clone seems to freeze the server) converted to phase 179:
deterministic in-process repro + verdict, the asyncio.to_thread fix for the
sync's git ops, and the dedicated Playwright E2E suite. TODO.md cleared
(items now live in .agents/phases/).

LLM-Generated: true
2026-10-12 09:27:58 -04:00
ducoterra 149e753952 phase: 178_split_thinking_levels
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 178 — final verification pass: 100% green

**Verified (all 5 tasks complete in `complete/178_split_thinking_levels/`):**
- `config.py` rename (`llm_chat_thinking_level` default `low`) + new `llm_summary_thinking_level` (default `""`); two `CATALOG` rows; `.env.example` replaced; no stale old-key references (only intentional negative pins in tests)
- Wire split: `chat_stream()` → chat level, one-shot `chat()` → summary level (absent `extra_body` when empty); per-turn line reads chat level only

**Gates:**
- `uv run pytest -k "config and thinking" -v` → 7 passed (defaults + env overrides both knobs)
- `uv run pytest` (unit + integration) → 4170 tests, exit 0; `--cov=app` → **99%** (>90% ✓)
- `uv run ruff check .` clean; `uv run pyright` clean
- `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` → 3 passed in isolation
- `test_admin_settings_panel.py` + `test_admin_settings_layout.py` → 17 passed in isolation

**Completion criteria:** all 6 test/doc criteria ✅ (admin GET lists both keys, old key gone — integration + E2E pins); commit + phase-dir move = harness's job.

**Documentation:** README.md — both new env vars correctly documented in the LLM (aipi) table with defaults/wire-mapping/off-rule, defaults match `config.py`, all `.env.example` vars covered: verified consistent, no changes needed. AGENTS.md — all referenced files/commands verified to exist, no phase-178 impact (no UI code change, A4): verified consistent, no changes needed.

**Decisions:** none required — no live changes made (E2E booted its own app subprocess against the existing DB).

**Next pending phase:** none — 178 is the last phase in `todo/`.
2026-10-11 23:38:18 -04:00
ducoterra c6223c93dc chore(agent): phase roadmap from TODO.md (178 split thinking levels)
LLM-Generated: true
2026-10-11 20:56:16 -04:00
ducoterra 36eddce672 phase: 177_themed_favicon
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Everything is verified. Final report:

## Phase 177 — Themed favicon: completion pass (all tasks pre-done, verified end-to-end)

**Verified (no code fixes needed — implementation matched the design D1–D4 exactly):**
- `theming.favicon_svg()` pure renderer + golden byte-identical pin; `GET/HEAD /favicon.svg` route (no-cache, `image/svg+xml`, built-in 200 fallback on DB blip, routes-first before static mount); 5 templates → bare `href="/favicon.svg"`, static file deleted (Containerfile cp removed); Theme-tab `data:` live preview with 4 restore points; JS↔server mirror pins.

**Tests / lint / coverage (all green):**
- `uv run pytest --cov=app --cov-report=term-missing` → 4161 passed, `app/` 99% (>90% gate)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` (isolated) → 6 passed
- `test_theme_save_and_coverage.py` (92) + `test_theme_semantic_completion.py` (93), isolated → 3 + 8 passed

**Completion criteria:** all met — default golden bytes/headers; themed roles + `#bor-theme` non-regression; DB-failure 200 fallback; 5 bare links + `/assets/favicon.svg` 404 + browser rendering; live-preview data: repaint + Save/Reset/re-show restore; gates green; E2E suites green in isolation. Commit: harness commits atomically (overrides AGENTS rule 8 — no git commands run).

**Documentation:** README.md verified consistent in full (the phase's favicon paragraph is accurate; all commands, env vars, routes, ports, scripts cross-checked) — no changes needed. AGENTS.md verified consistent — no changes needed.

**Notable:** `image-1.png` (owner's TODO screenshot, referenced by phase 176's overview) sits untracked at the repo root — pre-existing, explicitly out of scope per the phase-176 report; left untouched.

**Next pending phase:** none — 177 was the last; TODO.md is cleared.
2026-10-11 15:33:39 -04:00
ducoterra dd119d534a phase: 176_mobile_settings_value_width
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All documentation claims verify against the code. Everything is green — final verification complete.

## Phase 176 — Final Verification Report

- **Verified (all 3 tasks in `complete/`):** the mobile Value-column fix is in the working tree — `frontend/index.html` L1602 carries `class="settings-value-th"`; `frontend/assets/styles.css` has exactly two new rules inside the `@media (max-width: 640px)` block: `.settings-table { min-width: 720px }` and `.settings-value-cell, .settings-value-th { min-width: calc(13rem + 2rem) }`; `settings.js` + `app/` byte-unchanged (diff = index.html + styles.css + unit test only).
- **Documented deviation (flagged in task-03 report, not silent):** spec's `min-width: 13rem` on the border-box cell left the control at 176px (< the 192px A3 pin); executor corrected the cell to `calc(13rem + 2rem)` so the control (100% of content) lands at exactly 13rem ≈ 208px — the A2 lock intent preserved, unit pin + CSS comment updated to match.
- **Tests/lint/coverage (exact commands, all green):** `uv run pytest --cov=app` → **4138 passed**, `app/` coverage **99%** (>90% ✓); `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings.
- **Completion criteria:** 375px controls ≥ 192px — ✅ (E2E `test_mobile_value_inputs_are_usable_width`); card scrolls / page doesn't overflow at 375px — ✅ (`test_mobile_table_still_scrolls_and_page_does_not_overflow`); 1440px media query inert — ✅ (`test_desktop_value_width_is_unchanged`); tab order + zero external requests — ✅; phase E2E isolated — ✅ 4/4; phase 171 isolated — ✅ 5/5; phase 168 isolated — ✅ 12/12; unit pins — ✅ 17/17; commit/move — the harness commits; phase dir already in `complete/`.
- **Documentation:** README.md and AGENTS.md reviewed in full — every claim spot-checked against code (E2E ports 8123/8901, defaults `BOR_COMPRESS_MIN_BYTES=860` / `BOR_LOG_LARGE_RESPONSE_BYTES=65536`, audit budget 262144, `.cell-clip`, `.bubble:empty`, `min(70vh, 64rem)`, aria-pressed toggle, CDN integration test, `.gitignore` entries, all referenced scripts/routes) — verified consistent, no changes needed.
- **Notable:** `image-1.png` (owner's TODO screenshot, referenced by the phase overview) sits untracked in the tree — left alone; not part of the phase's commit scope.
- **Next pending phase:** `177_themed_favicon`.
2026-10-11 13:23:04 -04:00
ducoterra 2a5df7cb31 chore(agent): phase roadmap from TODO.md (177 themed favicon)
LLM-Generated: true
2026-10-11 11:44:07 -04:00
ducoterra bd271a6f52 chore(agent): phase roadmap from TODO.md (176 mobile settings value width)
The single TODO item (mobile settings Value inputs impossibly small) becomes phase 176_mobile_settings_value_width: 3 tasks (CSS fix in the <=640px block, unit pins, dedicated E2E suite) with owner-locked A1-A3. Protocol B append - PLAN.md and AGENTS.md untouched.

LLM-Generated: true
2026-10-11 10:28:54 -04:00
ducoterra 375169bbfc phase: 175_history_pagination
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All criteria verified. This pass found no defects — the phase was fully implemented by prior task runs, and every gate is green.

**Phase 175 final verification pass — report**

- Verified tasks 01–03 implementation in-tree: `list_chats` `?limit=` (25, 1–100) / `?offset=` + `func.count` `total`; `SavedChatList.total` required; `#history-pager` below `#history-table-wrap` (hidden default, aria-labels, aria-hidden SVGs); `history.js` `PAGE_SIZE=25`/offset state, parameterized fetch, `updatePager`, refresh-keeps-current-page; CSS pill union + ≤640px block. No code changes were needed.
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99%** (>90% gate).
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Phase E2E in isolation: `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` → **9 passed** (page 1-of-2 / 2-of-2 walk, replace-not-append, refresh-on-page-2, delete-to-empty + single-page hide, anonymous no-fetch, 375px mobile, a11y/no-CDN).
- Regression set, each in isolation: test_chat_history 5✓, test_history_page_width 5✓, test_share_chat 4✓, test_stale_saved_chats 3✓, test_history_copy 1✓; integration `test_chats_api.py` green (incl. updated `test_list_empty` two-key pin and all five new pagination pins).
- Completion criteria: all nine boxes satisfied (API slice/422/honest-total pins ✓; E2E walk + status lines ✓; refresh/delete/hide ✓; anonymous gate ✓; mobile ✓; gates ✓; E2E + regression ✓; docs ✓; commit/phase-move is the harness's).
- Documentation: README.md reviewed in full — the saved-chats section already carries the `?limit=`/`?offset=`/`total` contract from this phase; all 31 `.env.example` vars documented; AGENTS.md verified accurate. **Consistent — no changes needed.**
- Deviations: none. No defects found in previously completed work.
- Next pending phase: none — 175 is the only entry in `todo/`.
2026-10-11 04:12:53 -04:00
ducoterra 938d6cf09d phase: 174_git_sources_card_rhythm
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All checks pass. Both docs verified consistent. Final state — no live-infrastructure changes were needed (pure CSS fix; verification ran against the already-running local dev DB).

**Phase 174 final verification pass — report**

- **Verified (no new work needed; both tasks already complete in `complete/`):** the single `#git-sources-content { display:flex; flex-direction:column; gap:1.25rem }` rule in `frontend/assets/styles.css` (with the phase-174 root-cause comment), `.git-sources-shell` untouched, the 4-test unit pin module, and the 4-test Playwright suite.
- **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → **4116 passed, 99% `app/` coverage** (gate >90% ✓).
- **Lint/types:** `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings.
- **Dedicated E2E in isolation:** `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` → **4 passed** (1440px + 375px bounding-box rhythm pins 20±2px, hidden-children + revealed-joins-rhythm, anonymous gate, a11y/no-CDN).
- **Regression set, each in isolation:** `test_git_sources_admin.py` **6 passed**, `test_archive_upload_sources.py` **5 passed**, `test_git_source_dates.py` **4 passed**.
- **Completion criteria:** cards stack with uniform ≈20px gap at both viewports ✓; hidden children stay hidden / gate+content flip intact ✓; ≤640px forms column-stack + table scrolls inside its card (no page-level h-scroll) ✓; full suite + coverage + ruff + pyright ✓; dedicated + regression E2E suites ✓.
- **Documentation:** README.md reviewed in full (every `.env.example` var documented; all documented routes/pages/endpoints exist; every config default cross-checked against `app/config.py`; payload-limit table, ports 8123/8901, Containerfile/compose/entrypoint claims all match) — **verified consistent, no changes needed**. AGENTS.md verified (all referenced files/sections exist: PLAN §2/§7.1/§7.4/§9, `response_budget.py`, `debugging.py` conditional import, `.agents/` tracked + gitignore rules, `commit.gpgsign=false`, phase-pinned CSS classes) — **verified consistent, no changes needed**.
- **Notable:** commit + phase-dir move are handled by the harness per pipeline rules (no git history/index touched by this pass); no live-infrastructure changes were required.
- **Next pending phase:** `175_history_pagination` (in `todo/`).
2026-10-11 01:49:49 -04:00
ducoterra da441ea799 chore(agent): phase roadmap from TODO.md, phase 175 (history pagination)
LLM-Generated: true
2026-10-11 00:19:04 -04:00
ducoterra a86caec821 phase: 173_docs_push_creds_split
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All gates green. Final report:

**Phase 173 — final verification pass (all tasks already complete in `complete/`)**

- Verified the full implementation is in the working tree: `docs_user`/`docs_token` settings + catalogue rows (task 01), `docs_repo` PUT 422 guard + `docs_token` write-only pin (task 02), `run_git(env)` + `push_document(user, token)` credential env (task 03), `settings_migrate.py` wired into `create_app()` (task 04), E2E suite (task 05).
- **Completion criteria:**
  1. GET `/api/admin/settings` lists `docs_user` (plain) + `docs_token` (secret, `value=null`), token never in any body — ✅ `test_docs_token_is_write_only_masked_end_to_end` + E2E sentinel-absent-from-DOM
  2. PUT `docs_repo` with `://user:pass@` → 422 naming the fields; clean URL passes — ✅ integration + E2E UI-side
  3. Token set → `GIT_CURL_USERNAME`/`GIT_CURL_PASSWORD`/`GIT_TERMINAL_PROMPT=0` on every git step; empty → `env=None` — ✅ mocked-git unit tests
  4. Legacy env split once at startup, idempotent, one INFO line, `.env` untouched — ✅ `test_settings_migrate.py` (4 shapes) + `test_docs_push_migrate.py`
  5. Full suite green, coverage >90%, lint clean — ✅
  6. E2E isolated: `test_docs_push_creds.py` 4/4; regression set (panel + layout + response_to_docs) 21/21 — ✅
  7. Commit + phase move — harness's job (per instructions, not mine).
  8. README/AGENTS consistent — ✅ (below)
- Tests/lint: `uv run pytest` exit 0 (4112 passed) · `uv run pytest --cov=app --cov-report=term-missing` TOTAL 99% · `uv run ruff check .` clean · `uv run pyright` 0 errors
- Docs: README.md (incl. Docs-push config section, guard/migration paragraph, `.env.example` block) and AGENTS.md verified consistent in full; fixed two stale docstrings in `app/config.py` + `app/core/settings_catalog.py` (`is_set` → the API's actual `has_value` field).
- No live-infrastructure changes (no live step in this phase). No next pending phase — `todo/` held only 173.
2026-10-10 16:47:39 -04:00
ducoterra f27106a638 phase: 172_thinking_level
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete. Final report:

**Phase 172 final verification pass — all tasks (01–05) were already complete; verified the whole phase, no code fixes needed.**

- **Verified implementation:** `Settings.llm_thinking_level: str = "low"` + `CATALOG` row (`LLM (aipi)` / "Thinking level", plain: not secret/restart/overridable-exempt) + `.env.example` line under `BOR_LLM_CHAT_MODEL` + README table row; `_thinking_extra_body()` → `reasoning_effort` via `extra_body` on `chat()` + `chat_stream()` when trimmed-non-empty, key absent (byte-identical) when empty, embeddings untouched; per-turn line gains `thinking=<value>` (`-` when empty) before `thinking_chars`.
- **Tests/lint/coverage (all green):** `uv run pytest` → 4084 passed · `uv run pytest --cov=app --cov-report=term-missing` → 4084 passed, `app/` **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings.
- **Completion criteria:** (1) `pytest -k "config and thinking"` 4 passed + `.env.example` grep ✓ (2) live `GET /api/admin/settings` curl shows the row (str, `low`, `default`) + integration PUT/DELETE round-trip ✓ (3) 9 mocked-client unit tests assert `extra_body` present/absent ✓ (4) log-line tests assert `low`/`high`/`-`/whitespace ✓ (5) suite + coverage + lint ✓ (6) `test_thinking_level.py` 3/3 in isolation; `test_admin_settings_panel.py` 12/12; `test_admin_settings_layout.py` 5/5 ✓ (7) atomic commit + phase move = the harness's job (task files already in `complete/`; I ran no git index/history commands).
- **Documentation:** README.md reviewed in full — verified consistent (env table complete vs `.env.example`, catalog restart/non-overridable claims re-verified at runtime); AGENTS.md reviewed in full — verified consistent (no rule affected, A5 = no UI change). No changes needed. **Flag (not fixable here):** PLAN.md §9's per-turn-line spec predates the phase-172 `thinking=` field; PLAN.md is off-limits per protocol.
- **Next pending phase:** none — `todo/` contains only phase 172 (the harness moves it on this pass's success).
2026-10-10 13:19:10 -04:00
ducoterra b21b05eaee chore(agent): phase roadmap from TODO.md, phase 172 (thinking level)
LLM-Generated: true
2026-10-10 10:28:46 -04:00
ducoterra e9ae6e2ba3 phase: 171_admin_settings_layout
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Expected state: only `00_phase.md` remains in `todo/` (the harness moves it on success; I must not touch it). All verification is complete.

---

**Phase 171 final verification pass — report**

- **Verified (all 4 tasks complete, CSS-only diff confirmed byte-exact to contract):** `#settings-content` flex column + `gap: 1.25rem` (tasks 01); `.settings-group-row th` band `--brand-soft` + 700 + `--line` top hairline (02); `#settings-save:disabled` `cursor: default` (03); new phase-dedicated Playwright suite (04). Mobile media queries and `settings.js`/`app/` byte-untouched, as pinned.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **4069 passed, app/ 99%** (>90%); `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**.
- **E2E (in isolation):** `test_admin_settings_layout.py` → **5 passed**; `test_admin_settings_panel.py` (phase 168) → **12 passed**; `test_wide_desktop_column.py` → **3 passed**.
- **Completion criteria:** search/table no-intersection + gapped action row at 1440/375 px ✅ · distinct group bands (bg ≠ card, weight 700, ≥1px top hairline) ✅ · disabled Save `cursor: default` with §7.4 dim + aria-busy lifecycle intact ✅ · full suite + coverage + lint ✅ · all three E2E suites isolated ✅ · atomic commit + dir move = **harness's job** (task files already in `complete/`; per protocol I left every change uncommitted in the working tree).
- **Documentation:** README.md (all 1487 lines) and AGENTS.md (all 80 lines) fully reviewed — every config default cross-checked vs `app/config.py`, all quick-reference commands verified against existing files, AGENTS.md rules/contracts verified vs code — **verified consistent, no changes needed** (phase 171 is CSS-only: no setup/command/env/API impact).
- **Notable:** one phase-168 test failed on the first isolated run solely because an unrelated `undercooked` dev server held port 8124 (the suite's `DISABLED_PORT`); re-ran with the suite's built-in `E2E_APP_PORT=8300` override — no code change, foreign process untouched. Post-run DB state clean (`app_settings` 0, `ui_settings` 0, e2e tokens 0).
- **Next pending phase:** none — phase 171 was the last `todo/` phase.
2026-10-10 04:45:34 -04:00
ducoterra c4f098d0c0 phase: 170_edit_bubble_sizing
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete. Final report:

**Phase 170 — final verification pass (all 3 tasks were in `complete/`):**

**Verified implemented:**
- Editor renders inside the bubble: `bubble.replaceChildren(form)`, `display:none` rule deleted, `form.style.minWidth` seeded with pre-edit width, `restoreBubble()` (markdown + image re-prepend) before `closeEditor()` on Cancel/Escape, `closeEditor()` reordered before submit re-render
- Label: `editSaveBtn.textContent = "Send"` (app.js L1241)

**Gates (all 100% green):**
- `uv run pytest` → 4066 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 99% on `app/` (gate >90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors, 0 warnings)
- `uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov` (in isolation) → 6/6 passed
- Regression set in isolation: `test_edit_and_resend.py` 7✓, `test_retry_answer.py` 4✓, `test_failed_turn_retry.py` 4✓, `test_chat_persistence.py` 4✓

**Completion criteria:** in-bubble editor + width pin ✅ · "Send" label + cancel restore + trim re-ask + "Question too long" driver case ✅ · suite/coverage/lint ✅ · E2E + regression isolation ✅ · phase dir moved to `complete/` ✅ (atomic commit is the harness's to make — changes left in the working tree per protocol)

**Documentation:** README.md reviewed in full — edit section already aligned to phase 170 (in-bubble editor + "Send"), pages table matches actual routes, all 29 `.env.example` vars documented, setup commands accurate; AGENTS.md reviewed — all referenced files/commands exist. Both verified consistent — no changes needed.

**Notable decisions:** none — verification pass only, no code changes required.

**Next pending phase:** `171_admin_settings_layout`
2026-10-10 03:25:40 -04:00
ducoterra 1200437bf8 phase: 168_admin_settings_panel
Build and Push Containers / build-and-push-app (push) Successful in 4m4s
Build and Push Containers / build-and-push-db (push) Successful in 15s
**Phase 168 — Admin Settings panel — final verification: PASS**

1. **Implemented/verified:** `/settings.html` (admin-only) with the 75-field catalogue (`app/core/settings_catalog.py`), DB-over-env overrides in Postgres (`app_settings`, migration 0028 at head, `settings_overlay.py` resolver, `app/api/admin_settings.py` — 4 admin-only routes), live-apply on save with no restart + restart-required badges, Reset/Clear all, write-only secrets, `.env`-never-written, `BOR_SETTINGS_DB=0` kill switch.
2. **Phase E2E: 12/12** — run with `E2E_APP_PORT=8200` (ports 8123/8124 are held by an unrelated "undercooked" project; no process killed, no code change — the documented env override).
3. **Regression set: all 7 suites green in isolation** (theme tab, API tokens, configurable brand, global tuning, admin auth, cache busting, wide desktop column — 38 tests).
4. **Full suite: unit+integration exit 0 (4066 passed), app/ coverage 99%** (gate >90%); **ruff clean, pyright 0 errors / 0 warnings.**
5. **Log contract (rule 10) verified live:** per-turn + per-response byte-budget lines present; settings write lines carry key + action, **never the value**.
6. **Byte-budget audit: all endpoints under the 262,144 B budget**; settings catalogue 23,513 B (gzip 3,326).
7. **Defect fixed (one):** criterion 5 had no integration pin — added `test_bad_row_inserted_directly_never_bricks_boot` to `tests/integration/test_admin_settings_api.py` (malformed row inserted directly → fresh `create_app()` boots, row skipped with logged reason, rest of table applies, `/api/health` 200) — 12/12 green.
8. **Docs pass:** `README.md` and `AGENTS.md` reviewed in full — both consistent with the as-built feature (Settings page table, Settings section, Configuration-reference note, `.env.example` block); **no changes required**.
9. **Cleanup:** audit app killed (by PID); `app_settings` left empty (0 rows); all changes left in the working tree (no commit, per protocol).
10. **Completion criteria:** all 7 checkboxes verified green (criterion 7's commit is the harness's).
11. **Next pending phase:** `170_edit_bubble_sizing`.
2026-10-10 01:05:14 -04:00
ducoterra abe7388dda docs(phases): convert TODO.md L3-L4 into phases 170-171 (edit bubble, settings layout)
170_edit_bubble_sizing — the edit form renders inside the user bubble
(same box, no shrink) and its submit label reads "Send", not "Save"
(TODO.md L3).
171_admin_settings_layout — settings page CSS fixes: #settings-content
vertical rhythm (search/table overlap + smushed action row), distinct
group bands, a properly disabled Save cursor (TODO.md L4).

Numbering note: 169 (embedding generation migration) was created,
executed and abandoned before this conversion; its number is not
reused. TODO.md is cleared — the items now live in .agents/phases/.

LLM-Generated: true
2026-10-09 23:42:15 -04:00
ducoterra 542e568d25 docs(agent): record the reverted embed embedding benchmark
The embed alias was reverted to the previous backing model and re-benchmarked
over 3 runs. All gates pass and the numbers reproduce the original baseline
exactly: 768 dims, cosine margin 0.283 (quality 56) versus the 0.217 measured
on the swapped-in model, with throughput back to 109.7-123.3 vec/s.

The margin returning to the identical 0.283 confirms the locked semantic-pair
corpus is a reliable fingerprint for this endpoint, and a raw-vector probe
shows the two models are genuinely different spaces despite both being
unit-normalized 768-dim (max component diff 0.035 on the same input). Any
content embedded while the swapped-in model was live is therefore in an
incompatible space and needs re-embedding.

LLM-Generated: true
2026-10-09 12:41:38 -04:00
ducoterra a88a5a3713 docs(agent): re-run the embed embedding benchmark (speed variance)
Second 3-run pass against the re-pointed embed endpoint the same day.
Dimension stays 768/768 and the cosine margin reproduces exactly at 0.217
across all passes, confirming both the model's determinism and that the
margin drop from the 0.283 baseline is a real effect of the model swap.
Throughput measured 59.1/61.4/57.4 vec/s against 35.5/35.6 earlier, a 1.7x
swing on an unchanged model, which shows the speed check measures one batched
request's end-to-end wall time (TLS setup, network, server queueing) rather
than stable throughput - note it gates on NaN/Inf only, never on rate.

LLM-Generated: true
2026-10-09 12:38:27 -04:00
ducoterra 05f7d0645e docs(phases): add phase 169 (embedding generation stamp + in-place re-embed) and the migrate-embed-model skill
An embedding-model swap that keeps the dimension is invisible on this
app: nothing records which model made a vector, EmbeddingDimensionError
cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the
documents.file_hash fast path returns with zero embeds - so questions
embed in one geometry and chunks score in another with no error anywhere.
The owner's upgrade (embedding gemma -> embedding gemma 2) is exactly
that case.

Phase 169, seven tasks: chunks.embedding_model (reversible migration
0029, nullable, never backfilled - NULL means unknown by design), one
write helper so every vector write stamps its generation, a
generation_report() that turns the column into counts plus a stale
verdict logged after every sync, scripts/reembed_embeddings.py with
--compare (cosine agreement against the stored vectors, so 'are they
compatible?' is measured instead of guessed) / --stamp-only / --apply
(per-batch commits, resumable, never a NULL embedding, never a
sources_meta bump), an admin-only GET /api/admin/embeddings, a read-only
card in the phase-168 Settings view, and the dedicated Playwright file
plus the README runbook.

Owner-locked during authoring: detection only - retrieval never filters
by generation (a WHERE embedding_model predicate would silently deflect
every turn mid-backfill), comparability bands COMPATIBLE >= 0.995 for
the float4 round trip, a re-embed is not a KB change so saved chats stay
fresh, and no new Settings field so the phase-168 catalog pin is
untouched.

.agents/skills/migrate-embed-model is the operator twin: probe and
benchmark the new model, measure comparability as its own process
without disturbing the running app, back up and migrate production
(idle check, pg_dump off-box, alembic, flip then re-embed with the same
env), then re-tune relevance_threshold and the two floors against the
retrieval battery - after the corpus is on one generation, and with the
phase-168 warning that an app_settings row beats .env - and smoke-test
the result.

LLM-Generated: true
2026-10-09 12:35:21 -04:00
ducoterra 8fac4e64d4 docs(agent): record the re-pointed embed embedding benchmark
The aipi `embed` endpoint was re-pointed to a new backing model and was
re-benchmarked over 3 runs. All three gates pass: output stays 768-dim
(matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the
similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf
vectors, and throughput is 35.6 vec/s on the 50-vector probe.

Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin
tightens and throughput drops ~2.6x, both consistent with a model swap
behind the same alias. Vectors are unit-normalized and deterministic.
Note: same dimension on a different model means any KB indexed under the
old backing model holds stale vectors and needs a re-embed; the local dev
DB is empty, so the phase 119/162 retrieval battery was not runnable here.

LLM-Generated: true
2026-10-09 12:23:55 -04:00
ducoterra a12fdcdf0a Move phase 167 audit to complete
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-10-09 10:27:51 -04:00
ducoterra a2b2f283fd docs(phases): convert TODO.md L3 into phase 168 (admin settings panel)
One owner item, one phase: an admin-only Settings view that writes any
BOR_* setting to the database, resolves DB-over-env inside
get_settings(), and applies on Save without a restart. Six tasks: the
app_settings table + reversible migration 0028, the overlay resolver and
field catalog, the live-apply reset of the two in-process singletons
(the shared LLMClient and the chat semaphore), the admin-only
/api/admin/settings surface, the shell's eighth view, and the dedicated
Playwright suite.

Owner-locked during authoring: DB-only persistence (.env is never
written), the overlay lives inside get_settings() behind the
BOR_SETTINGS_DB kill switch, database_url and embedding_dim are not
DB-overridable, boot-wired settings stay editable with a "restart
required" badge, secrets are masked and write-only, and every write is
validated by the real Settings validators while an unusable stored row
is skipped at read so boot can never brick.

Protocol B append: PLAN.md and AGENTS.md are untouched, so PLAN's
roadmap table does not list phase 168 and its "get_settings() is
lru_cache'd" note in 13 is extended (cached env base + DB overlay +
invalidation). TODO.md's L3 item is now cleared - it lives in this
phase directory.

LLM-Generated: true
2026-10-09 10:10:13 -04:00
ducoterra 2ac37455f0 phase: 167_api_payload_diet
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All gates green. Final report:

**Verified (phase 167 final pass — all 6 tasks already in `complete/`)**
- Live re-measure of the headline criterion on the **1,000-doc live-replica corpus** (restored `data/bor_live_kb_replica.dump.sql` into a scratch DB `bor_audit`, alembic 0011→0027, dropped after; the shared dev DB was never touched): `GET /api/docs/tree` default depth=1 = **8,474 identity → 1,128 B with `Content-Encoding: gzip`** (< 40 KB bar; ~2 MB defect fixed), repeat load **304 / 0 bytes**; depth=8 265,147→23,515; node drill 20,844→1,065; `/api/docs` 246,366→45,104; `/api/documents/content` 892,124→60,776 (`LOOK`, accepted-with-why). Reproduces `AUDIT.md` to within 1 byte.
- Transport contracts proven live: `POST /api/chat` **no** `Content-Encoding` + frames arriving live + budget line `… enc=identity duration_ms=7380 stream=1`; HTML still `no-cache` with the `?v=` rewrite (and gzipped); A20 headers on 200 **and** 304; ETag identical across identity/gzip; `depth=0|9|abc`→422; anonymous tree/node→403; unknown folder/source→404 `unknown node`.

**Gates (all 100% green)**
- `uv run pytest` → **3980 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (`compression.py`, `http_cache.py`, `response_budget.py` 100%; `api/docs.py` 99%, one defensive dedupe line).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Phase + regression E2E **in isolation** (`--junitxml`, exit codes): `test_kb_tree_lazy_load` 11, `test_kb_tree` 8, `test_kb_tree_nav` 4, `test_navbar_refresh` 7, `test_document_dates` 6, `test_git_source_dates` 4, `test_sync_summary_visibility` 3, `test_table_ux_polish` 6, `test_sources_midstream_bug` 6, `test_ls_tree_drilldown` 3, `test_smoke` 3 — **0 failures/errors** (the earlier `-q -q` run had simply suppressed pytest's summary line).

**Documentation** — README.md + AGENTS.md reviewed in full (1,436 / 80 lines):
- Fixed (pre-existing, phase-167-exposed): README *Caching & deploys* claimed "One Starlette middleware (`app/core/caching.py`) applies the rule" while the section now describes three transport layers → rewritten to name caching/compression/http_cache, the real outermost→innermost order, and why `bytes=` is uncompressed.
- AGENTS.md rule 5 (UI Structure Check) gained the phase-167 clause: a lazily-drilled level keeps the current level rendered, its in-flight `role="status"` / `role="alert"`+Retry row lives in the same bounded scroll panel, and stat cards read server `totals` (never a walk of a depth-limited tree).
- Cross-checked and consistent: every `BOR_*` in `.env.example` is in the README table, every `Settings` field is documented, all 15 README-cited API routes exist in the app, no stale "one fetch / drill client-side" claim remains anywhere.

**Notable decisions / deviations**
- `compression.py` adds a documented `_RebufferSmallResponses` seam beyond the D1 sketch (the caching layer's `BaseHTTPMiddleware` re-framing otherwise defeats Starlette's `minimum_size` exemption and inflates tiny responses) — integration-pinned, math still Starlette's.
- A node-route **404 = vanished folder → honest reset to top level** (per `00_phase.md` D4/D6), not task-06's "alert + Retry"; 500 shows alert+Retry with the level preserved — both pinned, nothing weakened.
- Retrieval battery not measurable (shared E2E DB holds no corpus), but `git diff --name-only -- app/rag` is empty, so retrieval is unchanged by construction. `AUDIT.md` left byte-identical (it lives under `todo/`, off-limits to this pass); no git index/history commands run.

**Next pending phase:** none — `167_api_payload_diet` is the only entry in `.agents/phases/todo/`; the harness moves it to `complete/` and commits.
2026-10-08 21:23:49 -04:00
ducoterra ab9391d019 phase: 166_hide_empty_answer_bubble
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
**Phase 166 — final verification pass: green.** All 3 task files already in `complete/`, so this was verification + defect/doc fixing.

- **Verified:** `.msg .bubble:empty { display: none; }` exists once in the Messages block with provenance; `frontend/` diff is styles.css only (**no app.js change**); `tests/unit/test_no_empty_answer_bubble.py` 26 pins pass; phase-158/161 anchor pins (thinking_timeline + tool_states + edit_message) **75 passed**.
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_no_empty_box.py --no-cov` → **5 passed**. Mutation check (rule removed) → 2 failures, so the suite genuinely pins the defect.
- **Regression set, isolated:** thinking_timeline 6 · turn_progress_loader 3 · chat_history 5 · share_chat 4 · save_share_ux 5 · failed_turn_retry 4 · chat_answer_clusters 4 · chat_error_honesty 4 → all pass.
- **Full gates:** `uv run pytest --cov=app --cov-report=term-missing` → **3880 passed, app/ 99%** (gate >90%); `uv run ruff check .` → clean; `uv run pyright` → **0 errors**.
- **Whole E2E tree swept file-by-file (150 files, own pytest process each).** 4 suites were broken by *earlier* phases; fixed and re-run green:
  - `test_kb_tree`, `test_sync_summary_visibility`, `test_folder_summary_voice` — `td:nth-child(4) span` became a strict-mode violation when phase 156 wrapped the Edit label in `.cell-clip` → scoped to `span.kb-desc-text`.
  - `test_search_tool_scope` — pinned the pre-`690fd98` per-run `<summary>`/`open` shape → re-pinned to the ONE `details.reasoning` container + `.tool-calls-summary`.
  - `test_header_consistency` — `.doc-meta` pixel bound (`<34px`) was broken by phase 153's 44px touch-target floor at ≤640px → replaced with a shape-first single-line check that still fails a real wrap.
- **Completion criteria:** mid-turn box hidden ✅ · bubble appears at first delta, phase-130 rows below unchanged ✅ · restored/shared empty answer paints no box ✅ · no app.js change + anchors + full gates ✅ · commit/`complete/` move = harness's job (I ran no git index/history commands).
- **Docs:** README — added the phase-166 "no empty answer box mid-turn" note; fixed a **stale** Thinking lead ("collapsible Thinking blocks", pre-dated the 2026-10-06 one-disclosure refinement); QA section now states E2E isolation is mandatory (shared Postgres + fixed ports 8123/8901) with the per-file sweep loop. AGENTS.md — rule 5 UI checklist now names the phase-166 suppression rule.
- **Notable:** running all of `tests/e2e/` in one pytest session is not supported (fixture/port interleaving causes ~126 spurious failures) — documented in README. Restored the tracked `.agents/screenshots/doc_modal_scrollbar_1280.png` that `test_document_viewer.py` rewrites on every run.
- **Next pending phase:** `167_api_payload_diet`.
2026-10-08 04:27:03 -04:00
ducoterra b0c7aed613 phase: 165_prose_word_wrap
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification complete. Final report:

**Phase 165 — Prose word wrap (final verification pass)**
- Verified: `PROSE_FORMATS={txt,text}` + `isProseDoc` (single-site, image arm wins); `renderDocument` three-way branch (md → `.doc-md`, prose → `.doc-prose` text node, else `pre.doc-raw` byte-identical); `.doc-prose` CSS = 72rem column, `pre-wrap` + `overflow-wrap: anywhere`, tokens only, zero literals; `[data-view="pages"]` hide list includes `.doc-prose` as one grouped declaration; phase-116 scrollbar rules intact and modal-scoped; no `innerHTML` in the prose path.
- Tests/lint (all 100% green):
  - `uv run pytest` → **3854 passed** (unit + integration)
  - `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%)
  - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
  - `uv run pytest tests/e2e/test_prose_word_wrap.py -v --no-cov` → **5 passed** (isolation)
  - Isolation regressions: document_viewer 8, image_documents 4, pdf_pages_in_sources 4, pdf_strip_additive_selfheal 5, edit_summaries 4, summary_in_viewer 4, chat_answer_clusters 4, garbage_pdf 3, pdf_support 6, sticky_navbar 3, smoke 3 — all passed
- Completion criteria: image description wraps, full text, both surfaces ✔; `.txt` wraps same way ✔; `py`/`json`/code keep `pre.doc-raw` + horizontal overflow + themed modal scrollbar (E2E negative control + pins) ✔; no innerHTML, AA `--ink`/`--surface` pair ✔; full gates green ✔; commit + phase-dir move are harness-owned (not executed per pipeline rules).
- Documentation: README already updated by task 03 (viewer section + image-docs section); full README/AGENTS.md review found no further inconsistencies — **verified consistent, no changes needed**.
- Notable deviations: `.doc-prose` omits `font-family: inherit` from 00_phase D2's snippet — task 01's authoritative "exact property set" excludes it and `div` inherits fonts anyway (behavior identical). PDF deferral (A4) intact: `format == "pdf"` stays in `.doc-raw`, owner call recorded.
- Next pending phase: `166_hide_empty_answer_bubble`.
2026-10-07 22:57:40 -04:00
ducoterra 8d70aa946c docs(phases): close two gaps in phase 167 - the date-pin tree consumers and drifted line anchors
Audit of the written phase found the depth=1 default was specified without
naming two of its nine consumers, both of which walk the tree without a
depth parameter and assert phase-106 D9 dates:

- tests/integration/test_docs_api_dates.py (reaches a -> a/b -> deeper)
- tests/e2e/test_git_source_dates.py (_find_file recurses by design)

Task 02 gains item 9 (both migrate by requesting ?depth=8 explicitly with
every assertion intact, plus the truncation-proof pin that a truncated
folder's updated_at still equals the deep max), task 06's isolation list
names the E2E suite, and the overview's D2 now carries the full nine-file
consumer list. Also corrected the Context anchors: build_kb_tree is L780
(not 781) and _subtree_max is L760 (not 746).

LLM-Generated: true
2026-10-07 19:49:06 -04:00
ducoterra dcb1062b83 docs(phases): convert TODO L3-L5 into phases 165-167 (prose wrap, no empty answer box, API payload diet)
Three owner-reported defects become executable phases; TODO.md is cleared
once the roadmap is written (Protocol B append, PLAN.md/AGENTS.md untouched).

- 165_prose_word_wrap (TODO L3): image descriptions and .txt bodies render as
  a wrapping .doc-prose block instead of <pre class="doc-raw"> (page + modal via
  the one renderDocument core). Code and code-ish formats keep horizontal scroll
  - phase 116 locked A1 survives; extracted PDF text stays raw and the deferral
  is recorded as an owner call, not an oversight.
- 166_hide_empty_answer_bubble (TODO L4): .msg .bubble:empty { display: none }.
  CSS-only, so the phase-158/161 "reasoning container inserts before the bubble"
  anchor and its unit pins stay untouched; the box returns at the first delta,
  and the shared/restore empty-answer corners take the same path.
- 167_api_payload_diet (TODO L5): selective pure-ASGI gzip (SSE /api/chat skipped
  by exact path), a depth-limited KB tree with server-side totals + truncated
  nodes whose recursive updated_at comes from a new aggregate map, a
  GET /api/docs/tree/node drill route, content-hash weak ETag/304 on the catalogue
  reads (NOT sources_meta - bumping it would mark saved chats stale), the lazy
  client drill, and a /api/* byte-budget log line plus scripts/api_payload_audit.py
  whose measured numbers land in AUDIT.md.

Phase 97's "one fetch, drill client-side" contract is deliberately superseded
(locked A3); phase 116 A1, phase 158/161 anchors and A19/A20 middleware order are
explicitly preserved.

LLM-Generated: true
2026-10-07 19:42:34 -04:00
ducoterra 34d60efbc1 phase: 164_chat_error_honesty
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 29s
All gates green. Report:

- **Verified (phase 164, all 4 tasks already in `complete/`)**: `HistoryTurn`/`ChatMessage`/`ChatThinkingSegment` caps → 600 000, `history`/saved-messages lists → 500, new `SavedChatCreate/Update` total-8 000 000-char validator; exported `errorDetailText` at the two previously unguarded chat paths (pre-stream `!res.ok` + SSE `error` frame); `FAILED_REQUEST_TEXT` + `bubbleText` threading + phase-114 banner hint; dedicated E2E `tests/e2e/test_chat_error_honesty.py`.
- **Final-pass additions**: `tests/integration/test_chat_api.py::test_history_rejects_one_char_over_the_text_cap` (endpoint-level >600k → 422 with the pydantic list-of-objects `msg` — the criterion's upper half was only unit-pinned); README env rows for `BOR_APP_VERSION` + `BOR_SESSION_COOKIE`; restored `test_document_viewer.py`'s tracked screenshot artifact dirtied by the E2E run (via read-only `git show`, no index/history touched).
- **Tests/lint (exact commands)**: `uv run pytest --cov=app --cov-report=term-missing` → **3837 passed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_chat_error_honesty.py -v --no-cov` → **4 passed in isolation**; per-file loop over **all 148** E2E files (`uv run pytest tests/e2e/<f> -q --no-cov`) → **148 × exit 0**, incl. `test_failed_turn_retry`, `test_chat_save_payload_limits`, `test_edit_and_resend`, `test_llm_history`, `test_smoke`; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- **Completion criteria**: pre-stream 422 shows the distinct line + verbatim `msg`, `[object Object]` unrepresentable + no `input` in DOM — **met** (unit totality/source pins + E2E asserts 1–2); 525k history accepted (200 settles) / >600k → 422 / saved chat 201 + 8M bound — **met**; transport + SSE-frame settles keep `FAILED_TURN_TEXT`, stop/abort untouched, §7.4 re-enables — **met**; budget trim, `message` 4000, the 7 guarded detail-readers and all SSE shapes byte-identical (diff touches only caps + the two chat paths) — **met**; README/AGENTS full review + proxy deploy note — **met**; commit/`complete/` move = pipeline gate.
- **Docs**: README reviewed in full (env tables cross-checked programmatically: 74 `BOR_*` rows vs `Settings` defaults — no mismatches; every `.env.example` var documented; referenced scripts/pages/endpoints/`query_log` columns exist) — found & fixed the 2 undocumented real knobs above; the phase's payload-limits section, failed-turn copy and `client_max_body_size ≥ 16 MB` deploy note verified accurate. `AGENTS.md` verified consistent (rules, gitignore list, quick-reference commands, referenced paths all correct) — no changes needed.
- **Deviations**: `ChatThinkingSegment.text` raised 32 000 → 600 000 too (not itemized in A1, but its own docstring mirrors `HistoryTurn.thinking`; without it a long-thinking turn 422s its own auto-save — same mirror invariant). No live-infra changes made (the proxy limit lives in the separate deployment repo; README note only).
- **Next pending phase**: none — `164_chat_error_honesty` was the only entry in `todo/`.
2026-10-07 18:00:12 -04:00
ducoterra 6ba46e735a phase: 163_pdf_strip_additive_selfheal
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 21s
All gates verified. Final report:

**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**

- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
- **Next pending phase:** `164_chat_error_honesty`
2026-10-07 13:57:30 -04:00
ducoterra 74eb2e265c docs(phases): add phase 164 (honest pre-stream failure line, shared error-detail serializer, history caps for full-length answers)
Captures the 2026-10-07 live '[object Object]' finding: a >32k-char
answer rides back as history, the 422's list-of-objects detail hits the
unguarded body.detail assignment, and the zero-frame settle lies about
a dropped connection. Caps raise as a chain (HistoryTurn + ChatMessage
mirror per phase-83 SEC-05) with a bounded anonymous total; the budget
trim stays locked and untouched.

LLM-Generated: true
2026-10-07 09:57:40 -04:00
ducoterra 855fa27a30 docs(phases): add phase 163 (additive PDF strip view, write-first atomic copies, fast-path self-heal)
Captures the 2026-10-07 live findings on Letumdei_ Heirs to the Imperium:
a single sparse map page flipped a text PDF into the strip-only viewer,
and the strip's page copies were lost from disk while page_count stayed
stamped (no self-heal path). Owner decisions recorded: additive
text+Pages toggle for the pictorial class, write-before-delete atomic
persist, deterministic zero-LLM fast-path re-gate.

LLM-Generated: true
2026-10-07 09:46:37 -04:00
ducoterra eebc824636 phase: 162_search_recall_small_sources
Build and Push Containers / build-and-push-app (push) Successful in 1m50s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
2026-10-07 01:42:01 -04:00
ducoterra 690fd98036 fix(rag): stop the per-run blocks from folding — the container is the one disclosure
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The owner refinement (2026-10-06): the per-run thinking and tool-call
blocks inside the Reasoning bubble were individually collapsible
details elements — a fold inside the fold. A tool frame closed the
previous thinking block, so the visible thinking vanished mid-turn;
and the pinned-tail follow died on the append of every run after the
first (the pin was captured AFTER the new block's height growth
contaminated the 32px reading).

- app.js: the in-container runs are now plain divs —
  createThinkingRunEl (div.thinking + static .thinking-summary label +
  .thinking-text) and createToolRunEl (div.tool-calls-disclosure +
  static .tool-calls-summary label + the .tool-calls list); they never
  open or fold (the container is the ONE open/fold control, open
  through the turn, folded at settle — unchanged); the pin capture
  moves BEFORE ensureThinkingRun/ensureToolRun append (plus the
  first-frame edge: no container yet -> always pinned);
  closeCurrentRun just ends the run; closeAllRuns stays the defensive
  legacy guard; restoreTimeline/addSegments (shared.js) use the div
  builders; the legacy no-segments restore path keeps the old
  details-based blocks byte-identical
- styles.css: the in-container runs are distinct by label + rail
  (brand-ink Thinking label on a brand-soft rail; accent-ink
  Tool call(s) (N) label on a neutral rail); the interactive
  treatment (cursor + chevron) is scoped to the legacy <summary>; the
  inner windows stay disabled (the container's .reasoning-body is the
  ONE live scroll surface)
- unit + E2E: the D5-refinement contract is pinned — one
  details.reasoning disclosure, per-run blocks as plain divs, zero
  nested <details> inside the container, the container as the only
  open/fold control, the .reasoning-body window auto-follows the
  tail while streaming

unit+integration 3738 passed (coverage 99%), ruff + pyright clean, all
five affected E2E suites pass in isolation (reasoning_bubble 6,
thinking_timeline 6, thinking_display 5, thinking_scroll 8,
tool_call_compact 4), verified live: the follow holds through a long
thinking stream, the tool-frame transition, and the second thinking
run; restore renders the div structure without client-side errors

LLM-Generated: true
2026-10-06 14:44:41 -04:00
ducoterra 051f62deb7 phase: 161_reasoning_bubble
Build and Push Containers / build-and-push-app (push) Successful in 1m55s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All gates verified. Final check — AGENTS.md rules against the project (commands, gates, protocols) — the quick-reference commands all exist and the rules (PLAN §2/§7/§9 references, phase protocol, No CDN, Debugpy, git gates) accurately describe the project; the reasoning bubble is a pure UI change with no new commands or env vars, so no AGENTS.md change is needed.

**Verification complete — full results:**

- **Verified** (implemented by prior task passes, all in working tree): `createReasoningWrap`/`ensureReasoningWrap` live container (D1), `settleReasoning` fold at the 3 settle sites + not at delta (D2), 320px `.reasoning-body` window + pinned-tail follow (D3), zero persistence change (D4), collapsed restore + shared-page parity with D6 step counts, `styles.css` family treatment + reduced-motion still, mock `REASONING_MIDROUND_TRIGGER` flow, 6-test story suite, phase-158 pin updates
- **Full suite:** `uv run pytest` → **3738 passed, 0 failed** · `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99% (>90%)** · `uv run ruff check .` → all checks passed · `uv run pyright` → **0 errors, 0 warnings**
- **E2E in isolation** (DB up; port-8123 collision with an unrelated `undercooked` service → `E2E_APP_PORT=8125`/`E2E_MOCK_PORT=8902`, no live change made): `test_reasoning_bubble.py` **6/6** · `test_thinking_timeline.py` ✓ · `test_thinking_display.py` ✓ · `test_tool_call_compact.py` ✓ · `test_big_read_progress.py` ✓ · `test_turn_progress_loader.py` ✓ · `test_share_chat.py` ✓ · `test_chat_persistence.py` ✓ · `test_smoke.py` ✓ (all exit 0, zero failures)
- **Completion criteria:** all 9 checked against the code + test outcomes — container open/live/nested ✓, stays open past mid-round delta ✓, folds at done ✓, restore + shared collapsed with "3 steps" ✓, plain turn = no container ✓, regression suites green ✓, suite/coverage/lint green ✓, diff scope limited to `frontend/assets/{app.js,shared.js,styles.css}` + `tests/` + `README.md` + phase files (no `app/` — D4) ✓
- **Documentation:** README.md "Thinking" section names the Reasoning bubble accurately (320px window, open-until-settle incl. mid-round deltas, "N steps", byte-identical no-container, collapsed on restore/shared) — verified consistent, no changes needed; AGENTS.md — verified consistent, no changes needed; all `.env.example` vars documented
- **Decisions:** commit + phase-dir move left to the harness (per pass rules — no git index/history commands executed); `undercooked` dev server on port 8123 untouched, E2E used the env port overrides
- **Next pending phase:** none — `161_reasoning_bubble` is the last in `todo/`
2026-10-06 06:36:09 -04:00
ducoterra 8172956c15 phase: 160_tool_budget_telemetry
Build and Push Containers / build-and-push-app (push) Successful in 2m27s
Build and Push Containers / build-and-push-db (push) Failing after 2m8s
All green. Final report:

**Phase 160 completion pass — verified (final verification; all 5 tasks already in `complete/`)**

- **Verified, no code defects found:** budget trailers (`TOOL_BUDGET_REMAINING/LAST/SPENT`), cap-path rejection + `ROUND_CAP_FALLBACK_ANSWER` + `ROUND_CAP_WRAPUP_INSTRUCTION`, `done.budget_exhausted` (ABSENT-when-false), `rounds=/cap_hit=` log fields, `agent_max_rounds=10` default, frontend Continue button + restore + shared-page exclusion, mock markers.
- **Tests/lint:** `uv run pytest` 153/153 green (EXIT 0); `uv run pytest --cov=app --cov-report=term-missing` TOTAL **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors/0 warnings.
- **Criteria:** ① decreasing trailer pinned by unit tests (9→…→spent) ✅ ② cap path ends readable, no tool lines after cap (E2E) ✅ ③ `budget_exhausted` true-only-on-cap, absent otherwise (integration) ✅ ④ Continue button → one fresh `continue` turn, fresh budget, record intact (E2E) ✅ ⑤ flag survives refresh; shared chat never renders it (`shared.js` has no `budget_exhausted` path) ✅ ⑥ `max_rounds=0` byte-identical single `tools=None` request; default still 10 ✅ ⑦ full suite/coverage/lint green ✅ ⑧ `test_tool_budget_telemetry.py` **7/7** in isolation; all 6 regression E2E suites green in isolation ✅ ⑨ commit + phase move = harness (I make no git history/index changes).
- **Docs:** README.md — fixed: the "Continue researching" button and cap behavior were undocumented (added a Chat-section paragraph + an Agent-tools-section paragraph, incl. the phase-71 recorded-revision note); env-var table, commands, and architecture all verified consistent otherwise. AGENTS.md — verified consistent, no changes needed.
- **Decisions:** no code changes required — phase was fully implemented; only README docs added. No live-infrastructure touched.
- **Next pending phase:** `161_reasoning_bubble`.
2026-10-06 01:42:44 -04:00
ducoterra cb8767512a phase: 159_edit_and_resend_message
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All completion criteria verified. Final report:

**Phase 159 — Edit the last question and re-ask it: VERIFIED COMPLETE** (all 4 tasks already in `complete/`; this pass verified end-to-end)

**Verified (implementation landed + tests green):**
- `app.js`: `EDIT_ICON`, `lastUserWrap`, `appendEditButton`/`markLastEditable` (single-target discipline, in-flight guard), `openEditor` (labelled textarea, `maxlength=4000`, Escape/Ctrl+⌘+Enter, empty guard), save path: replace → splice tail → DOM removal → `lastBrainWrap=null` → `saveConversation`+`persistConversation` → `runTurn(text,{reask:true})` — matches README exactly
- `styles.css`: `.edit-btn`/`.edit-form`/`.edit-error`/`.is-editing` + mobile 44px; unit pins 40, E2E 7, integration `test_chats_api.py` extended

**Test/lint results (all green):**
- `uv run pytest` — 3,661 tests, exit 0
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **99%** (>90%)
- `uv run ruff check . && uv run pyright` — all checks passed; 0 errors
- `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` (DB up) — **7/7 passed**
- Regressions in isolation: `test_retry_answer.py` 4/4, `test_failed_turn_retry.py` 4/4, `test_chat_persistence.py` 4/4

**Completion criteria:** all 7 ✓ — Edit on last bubble + prefill + Escape/Cancel restore (E2E #3); replace+truncate+re-ask with no duplicated question (E2E #1); refresh survival (E2E #4); failed/“question too long” turn editable + trim re-asks (E2E #5); earlier bubbles un-editable + byte-identical Retry/Stop/persistence (E2E #2 + regressions + unit pins); suite/coverage/lint ✓; E2E + regressions isolated ✓. Commit + phase-dir move: harness's job (no git ops by me, per protocol).

**Documentation:** README.md — full review; phase 159 section cross-checked against `app.js` source (cap, keys, truncate, image survival) — consistent, no changes needed; all 30 `.env.example` vars documented. AGENTS.md — full review; all rules/quick-ref commands valid — verified consistent, no changes needed.

**Notable:** none — no defects found in prior work; no live-infrastructure changes required.

**Next pending phase:** `160_tool_budget_telemetry`
2026-10-05 19:07:33 -04:00
ducoterra 4d7d0f6297 chore(agent): phase roadmap from TODO.md, 1 phase (161 reasoning bubble)
LLM-Generated: true
2026-10-05 13:59:47 -04:00
ducoterra aa297ea9b1 docs(phases): convert TODO L3/L4 into phases 159 (edit & resend) and 160 (tool-call budget)
Build and Push Containers / build-and-push-app (push) Successful in 15s
Build and Push Containers / build-and-push-db (push) Successful in 12s
TODO.md's two open items become executable phase directories under .agents/phases/todo/:

- 159_edit_and_resend_message (5 tasks): inline edit affordance on the last
  user bubble, in-place resend that truncates the replaced turn, an editable
  error turn after an oversize question, and the Playwright suite
  tests/e2e/test_edit_and_resend_message.py.
- 160_tool_budget_telemetry (5 tasks): remaining tool-call count in every
  tool result, tool calls rejected (never shown) once the budget is spent,
  the additive done.budget_exhausted flag + rounds=/cap_hit= log fields,
  the Continue researching affordance, and
  tests/e2e/test_tool_budget_telemetry.py.

No application code changed; the TODO lines are recoverable from git history.

LLM-Generated: true
2026-10-05 11:16:37 -04:00
ducoterra 31bd903a93 phase: 158_thinking_timeline
Build and Push Containers / build-and-push-app (push) Successful in 8m38s
Build and Push Containers / build-and-push-db (push) Successful in 32s
All verification complete. Every gate is green and the docs check out. Final report:

## Phase 158 final verification pass — COMPLETE

**Verified (all tasks pre-completed; this pass re-ran the full gate):**
- Unit + integration: `uv run pytest --cov=app --cov-report=term-missing` → **3618 passed, 99% coverage on `app/`** (>90% gate)
- Phase E2E in isolation: `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` → **6/6 passed** (order pin, fold-at-rest, live-open, restore+`segments` array, shared page, no-thinking single disclosure)
- Regression E2E, each in isolation: `test_thinking_display` 5✓, `test_tool_call_compact` 4✓, `test_turn_progress_loader` 3✓ (per-run pin updated), `test_big_read_progress` 4✓, `test_agent_document_tools` 4✓, `test_share_chat` 4✓, `test_chat_persistence` 4✓, `test_smoke` 3✓
- Lint/types: `uv run ruff check . && uv run pyright` → **clean (0 errors)**

**Completion criteria:** all 9 met — timeline E2E pins ✓, restore/shared `segments` ✓, no-thinking single-disclosure + the three UNCHANGED regression suites green ✓, pre-phase byte-identical round-trip (integration pin: key absent, never null; >50 / >32 000 / empty text / unknown kind all 422) ✓, full suite + coverage + E2E + lint ✓, `git diff --stat` footprint limited to `app/schemas.py`, `frontend/assets/{app,shared}.js`, `tests/`, README, phase files (a pre-existing owner `TODO.md` edit is out of scope, not staged by the phase commit) ✓, D15→per-run supersession recorded in the updated pin ✓. (Commit + phase move are the harness's job; task dirs already in `complete/`.)

**Documentation:** README.md — Thinking section correctly describes the phase-158 timeline; config reference matches `.env.example` exactly; page table matches registered routes; no stale single-block wording. AGENTS.md — rules and quick-reference commands all accurate. **Verified consistent — no changes needed.**

**Decisions/deviations:** none. `test_tool_call_compact.py` carries a one-line docstring-only update (referenced the removed `closeToolCalls`); its test logic is byte-identical to the phase-117 contract and green.

**Next pending phase:** none — `todo/` contains only 158.
2026-10-05 02:52:55 -04:00
ducoterra 3f752fb77f phase: 157_chat_chip_cap
Build and Push Containers / build-and-push-app (push) Successful in 2m19s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification is green. Final report:

**Phase 157 — final verification pass (all 3 tasks already complete in `complete/`)**

- Verified the implemented work matches the design: `CHIP_CAP = 90` constant + display-only cap in `renderChips` (`slice(0,90).trimEnd() + "…"`), `title`/`aria-label`/click payload byte-unchanged; unit pins (89/90/91 letter & space, full-text invariant, short-string guard, Maybe-try single-path guard); phase-104 E2E updated to the cap contract
- Full suite `uv run pytest --cov=app --cov-report=term-missing`: **3578 passed**, app/ coverage **99%** (>90% ✓)
- `uv run ruff check .` clean; `uv run pyright`: 0 errors
- Dedicated E2E `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov`: **4/4 passed** in isolation (DB up)
- Cross-phase regression: phase-104 E2E 6/6, phase 05/80/103 chip suite 8/8 (seed chips <90 chars — cap is a no-op, assertions hold)
- Criteria: ≤91-char capped display ✓ (byte-for-byte pin) · click submits FULL 120-char question (user bubble) ✓ · `title` + clipped-case `aria-label` carry full text ✓ (375px viewport) · <90-char byte-identical ✓ (unit + live DOM) · commit: deferred to the harness (per harness protocol, overrides AGENTS.md rule 8 for this pass)
- Documentation: README.md reviewed in full — every `.env.example` var documented, all referenced scripts/pages/commands verified against the tree, the phase's chip-cap note is in the Chat section; AGENTS.md verified (commands, test files, layout-rule citations — phase 157 adds no new layout principle) → **verified consistent, no changes needed**
- No live-infrastructure changes were required or made
- Next pending phase: **158_thinking_timeline**
2026-10-04 22:18:31 -04:00
ducoterra 1513981c1c phase: 156_table_ux_polish
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 11s
Everything is verified. All gates green on the current working tree. Final report:

**Phase 156 — final verification pass (all 4 task files present in `complete/` — no outstanding work)**

- **Verified (not re-implemented):** bounded-panel `.table-wrap` (`max-height: min(70vh, 64rem)`, `overflow: auto`, opaque sticky `th` via `var(--brand-soft)`), `.cell-clip` ellipsis rule + the five label families in the three renderers, the added aria-labels, unit pins (incl. `.md-table-wrap` byte-unchanged scope guard, `#git-sources-table-wrap` harmony — no divergent rule), dedicated E2E suite (sticky ×3 tables, horizontal contract, computed-style ellipsis + reachable full text, mobile hamburger + scroll + pin)
- **Tests/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **3570 passed, 99% on `app/`** (gate >90%); `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` (DB up) → **6/6 passed in isolation**; `uv run ruff check . && uv run pyright` → **clean** (0 errors)
- **Completion criteria:** sticky header all 3 tables (exact-0 offset while `scrollTop > 0`) ✅ · horizontal scroll unchanged ✅ · ellipsis computed style + actual clip + title/aria reachable ✅ · mobile ≤640px contract ✅ · full suite/E2E/lint/coverage ✅ · atomic commit: **harness commits this phase after the pass** (per pipeline protocol — no git history commands run by me)
- **Documentation:** AGENTS.md phase-156 wording (bounded panels, sticky headers, `.cell-clip` + aria-label idiom) matches the implementation exactly — verified consistent; README.md reviewed in full (pages/URLs vs registered routes, all `.env.example` vars documented, Quick Start, eval/llm scripts, compose prod profile, E2E examples) — verified consistent, no changes needed (README describes no table layout/scroll, so no phase-156 mention required)
- **Deviations:** none — all work was already in the working tree; this pass re-verified every gate and the docs
- **Next pending phase:** `157_chat_chip_cap`
2026-10-04 20:46:57 -04:00
ducoterra 742eef955b phase: 155_folder_summary_voice
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification is complete — the whole project is 100% green. Final report:

**Phase 155 final verification pass — complete.**

**Verified (all tasks 01–04 done in `complete/`):**
- Instruction nudge in `app/rag/folder_summaries.py` (forbidden openers as the pin, 4 constraints preserved, docstring cites the phase)
- `scripts/regen_folder_summaries.py` (import_docs idiom, full mode, cron-safe summary line, fail-loud boot)
- Unit pins, integration suite, mock voice update, dedicated E2E suite

**Test / lint / coverage:**
- `uv run pytest --cov=app -q` → 100% passed (exit 0, zero failures); coverage **99%** on `app/` (>90% ✓)
- `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` → **3 passed** in isolation
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
1. Marker + header prefix byte-identical — ✅ (git vs HEAD + unit pins verbatim, green)
2. Regen script full regeneration — ✅ (live smoke: exit 0, `folders=… limit=7` summary line; misconfig boot → named stderr, exit 1; `updated_at`-advance/no-prune/canned-text semantics proven by the integration suite, green)
3. Sync path untouched — ✅ (0 diffs to `app/api/sync.py` / `app/rag/importer.py`; phase-127 pins green)
4. E2E stored + rendered without forbidden openers — ✅ (3/3, exact canned-string assertions + phase-127 regression guard)
5. Full suite green, E2E isolated green, ruff + pyright clean, coverage >90% — ✅
6. Atomic commit — deferred to the harness (working tree left intact per protocol)

**Documentation:** README.md and AGENTS.md reviewed in full — verified consistent, no changes needed (README already documents the new script in the update workflow; all `.env.example` vars documented; pages table matches routes; 17 colors = 9+8; no stale mock-voice references).

**Decisions:** No live-infrastructure changes (dev-DB read + a 0-row script smoke only). The `TODO.md` working-tree edit is the owner's own, left untouched.

**Next pending phase:** `156_table_ux_polish`.
2026-10-04 16:32:50 -04:00
ducoterra c275735db4 phase: 154_sync_status_freshness
Build and Push Containers / build-and-push-app (push) Successful in 2m26s
Build and Push Containers / build-and-push-db (push) Successful in 16s
All 4 task files exist in `complete/`, and only `00_phase.md` remains in `todo/`. This is the final verification pass. I'll read through the task files and check the current state of the code.
2026-10-04 14:24:33 -04:00
ducoterra 966c4b0540 phase: 153_document_viewer_context
Build and Push Containers / build-and-push-app (push) Successful in 2m45s
Build and Push Containers / build-and-push-db (push) Successful in 19s
That last `--co` (collect-only) run was my probe mistake — it shows 34% because no tests execute under collect-only. The valid coverage run was the real gate command executed earlier: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (exit 0, all 3544 tests executed). All gates are verified.

## Phase 153 — final verification pass report

**Verified (all tasks 01–04 complete in `complete/`):**
- Modal `fullPageUrl(source, path, back = null)` + `openDocumentModal(…, back = null)` with module-level `currentBack`; 5 chat call sites pass `"/"`, sources.js call site unchanged (byte-identical Sources path); `documentUrl` comment now true
- Copy-path control (shared `renderDocument` core, icon+text, text = accessible name) on both surfaces; per-page clipboard helpers with the fallback-field + status-line contract; `#doc-copy-announcer` live region; styles + 44px touch floor
- Unit pins: `&back=%2F` / no-`back` byte-identity, `currentBack` single-writer guard, copy strings byte-pinned, phase-13 regression pins untouched

**Test / lint / coverage results:**
- `uv run pytest` → **3544 passed**, 0 failed (full unit + integration)
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90% gate)
- `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` → **8 passed** in 47.33s, in isolation (DB up, mock LLM)
- `uv run ruff check . && uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** all met — (1) chat→modal→Full page: `href="/"` + "Chat"; Sources→modal→Full page: `href="/sources.html"` + "Sources" with `back=` asserted absent; (2) no-`back` URLs byte-identical to pre-phase; (3) Copy path: "Path copied." on success (clipboard readback), prefilled select-on-focus fallback field + "Path ready — copy it from the field." on rejected clipboard, both surfaces parameterized; (4) copied string exactly `docs/homelab/kubernetes.md`; (5) full suite green, E2E isolated green, lint clean, coverage 99%; (6) commit by harness.

**Documentation:** README.md reviewed in full — phase-153 feature already named in the Chat section; routes, env reference, workflows, deployment all consistent — no changes needed. AGENTS.md reviewed in full — rules, gates, and quick-reference commands verified against the repo (incl. `test_smoke.py`, `.env.example`, gitignore entries) — consistent, no changes needed.

**Notable:** pytest 9.1.1 quirk — a doubled `-q` (pyproject `addopts` + explicit `-q`) swallows the final "N passed" summary line while still exiting 0; the plain `uv run pytest` gate command prints it. No defects found in prior work.

**Next pending phase:** `154_sync_status_freshness`.
2026-10-04 09:23:54 -04:00
ducoterra 12fa69b7ea phase: 152_sync_progress_image_pass
Build and Push Containers / build-and-push-app (push) Successful in 3m34s
Build and Push Containers / build-and-push-db (push) Successful in 23s
All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.

## Final Report — Phase 152 final verification pass

- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
2026-10-03 01:52:41 -04:00
ducoterra c7cfcd26b5 phase: 151_pdf_image_extraction
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.

All verification is complete. Final state:

**Phase 151 — final verification pass: complete, project 100% green.**

- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
- **Next pending phase:** `152_sync_progress_image_pass`
2026-10-02 07:16:08 -04:00
ducoterra edf6048a0d phase: 150_drop_sync_result
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 150 final verification pass — complete, 100% green.** All 6 task files were already in `complete/`, so this was verification + doc pass.

- Verified: `#sync-progress` is the sole sync text surface + sole `aria-live` region (`role="status"`, polite, no `aria-hidden`); `span#sync-result` gone from HTML/JS/CSS/tests (incl. runtime-built `RETIRED_ID` absence pins in the unit suites); `setSyncProgress` single writer with text-driven visibility; running-only box write in `setSyncLabel`; all 9 settle/cancel sites routed; `requestCancel` pre-fetch clear is a NO-OP.
- Tests: `uv run pytest` → **3463 passed**; `uv run pytest --cov=app` → **99% on app/** (>90%); `uv run ruff check .` + `uv run pyright` → clean (0 errors); `tests/e2e/test_sync_progress_box.py --no-cov` → **8 passed in isolation**; all 18 migrated sync/image-PDF E2E files green in isolation (77 tests).
- Completion criteria: 1) grep clean ✅ 2) sole live region, attrs correct ✅ 3) in-flight/settle/cancel/idle box states proven by E2E + unit pins ✅ 4) icon-only running button + restore ✅ 5) full gates green ✅ 6) no completed-phase behavior change (18 family suites + full suite green) ✅ 7) commit — left to harness per pass instructions (working tree untouched, no `git add`/`commit`).
- Documentation: README.md fixed 2 pre-existing inconsistencies (stale `~/Homelab`+`~/Deployments` source model in the intro — retired in phase 61 — now describes the git/local/archive Sources model; "Requires AIPI_KEY" in retrieval-quality now states `BOR_LLM_API_KEY` in `.env` or `$AIPI_KEY`, matching the scripts' real convention). Everything else (URL table, env defaults vs `app/config.py`, sync/phase-150 copy, prod claims) verified accurate. AGENTS.md: verified consistent — no changes needed.
- Notable: no code changes were required — the phase implementation was correct as left; no live infrastructure touched.
- Next pending phase: `151_pdf_image_extraction` in `.agents/phases/todo/`.
2026-10-02 02:35:03 -04:00
ducoterra 9d4c813cae chore(agents): phase roadmap from TODO.md, 3 phases
Converts the three open TODO items into executable phases (Protocol B
append, numbered after the existing 149):

- 150_drop_sync_result — retire span.sync-result; #sync-progress becomes
  the single sync surface and the sole aria-live region (TODO L5)
- 151_pdf_image_extraction — dedupe PDF image occurrences (46 placements
  -> 15 unique in podman-systemd.pdf), composite /SMask so masked images
  are not blank, and skip near-white decorative underlays (TODO L4)
- 152_sync_progress_image_pass — report the PDF vision-image position so
  the sync label stops freezing at the last chunk (TODO L3)

Both reported bugs were reproduced against test_sources/podman-systemd.pdf
before authoring; root causes are recorded in the phase overviews. The
TODO file is cleared now that its items live in .agents/phases/.

LLM-Generated: true
2026-10-01 22:12:10 -04:00
ducoterra dd524246ec phase: 149_sync_progress_box
Build and Push Containers / build-and-push-app (push) Successful in 3m35s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification complete. The `ink-soft`/`--surface` pair the box uses is already pinned AA by the existing theme E2E suites (8.2:1 dark, ~6.9:1 light). Everything is green — no fixes were needed.

**Phase 149 final verification report**

- Verified (no re-work needed): `#sync-progress` box wiring (single `setSyncLabel` writer with `running` flag; `is-running` class travels with the spin; aria-label owned by the writer), the panel/icon-only-button CSS, and the 5-test dedicated E2E; all 6 pre-existing sync E2Es' running pins migrated to the box, settled pins untouched.
- Dedicated E2E: `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` → **5 passed in 198.6 s** (isolation, slow-proxy mock; under the 600 s budget).
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **3463 passed, app/ coverage 99%** (>90% gate).
- Migrated sync E2Es, each in isolation: `test_sync_button` 3 passed, `test_sync_cancel` 3, `test_sync_chunk_progress` 2, `test_sync_label_mobile` 4, `test_sync_summary_visibility` 3, `test_sync_upload_progress` 4 — all green.
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Criteria: in-flight spinner-only button + full untruncated wrapped box (desktop & mobile 390×844, `scrollWidth ≤ clientWidth`) ✓; settle byte-identical ("Synced HH:MM" + `3 added`, box hidden, aria restored) ✓; `#sync-result` sole live region (box `aria-hidden`), mid-run reload re-attaches ✓; WCAG (AA contrast both themes via the pinned ink-soft/surface pair, focus-visible, reduced-motion kill intact at styles.css L2201) ✓; cancel/409/error/upload/status API unchanged (green suites above) ✓.
- Documentation: README.md reviewed in full (sync-states paragraph now matches the shipped UI; env-var table matches `app/config.py` and covers all `.env.example` vars; pages/commands verified) and AGENTS.md reviewed in full — **verified consistent, no changes needed**.
- Notable: no defects found in this or prior work; no live infrastructure touched.
- Next pending phase: none — `todo/` will be empty once the harness moves 149 (only 149 was pending).
2026-10-01 21:14:48 -04:00
ducoterra b233c8d714 phase: 148_pdf_pages_in_sources
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 16s
**Phase 148 — final verification pass: complete, 100% green.**

- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
2026-10-01 16:58:01 -04:00
ducoterra 863519e980 phase: 147_sync_changed_only
Build and Push Containers / build-and-push-app (push) Successful in 2m26s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All defaults match. Verification complete — everything is green with no defects found.

**Phase 147 final verification pass — report**

- All 3 task files found in `complete/147_sync_changed_only/` → verification pass only; no code changes needed.
- Verified: migration 0025 (`documents.file_hash` nullable String(64), no index) + model field; fast path at top of `_index_file` (streamed 1 MiB sha256 → row lookup → `_unchanged_doc`); stamping on all three branches' create/update + shared helper; pre-phase NULL-hash fallthrough; retriever/`content_hash` untouched.
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3438 passed, `app/` coverage **99%** (>90% ✓); `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` → 3 passed in isolation ✓; `uv run ruff check . && uv run pyright` → all checks passed, 0 errors ✓; prior-phase `tests/e2e/test_sync_button.py` → 3 passed (no regression) ✓.
- **Completion criteria:** all 7 pass — (1) column + round-trip ✓ (2) unchanged text/image/garbage-PDF/pictorial-PDF cost zero LLM/extraction/vision, `summary.unchanged` counted, rows byte-still ✓ (3) changed file re-indexes + re-stamps; bytes-changed/content-same stamps only ✓ (4) backfill/D1/D4 + log wordings pinned verbatim ✓ (5) `content_hash`/retriever byte-identical ✓ (6) all gates green ✓ (7) no prior-phase behavior change ✓.
- **Documentation:** full review of README.md (1,031 lines) and AGENTS.md against the code: every `.env.example`/code `BOR_*` var documented, all config defaults match `app/config.py`, quick-start/QA commands and referenced test files exist, `file://` URL + troubleshooting claims accurate; the phase's README "Updating the documents" edit is accurate. **Verified consistent — no changes needed.**
- **Notable:** no live-infrastructure changes (dev Postgres container used for the standard test gates only). Working tree left uncommitted per harness protocol.
- **Next pending phase:** `148_pdf_pages_in_sources`.
2026-10-01 14:23:33 -04:00
ducoterra ab3f5f685b phase: 146_search_tool_scope
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Phase 146 final verification pass complete — all green, tree untouched by me.

**Verified (final pass — both tasks already in `complete/`):**
- Scoped grep streams `{"type":"tool","name":"grep","argument":<pattern>,"path":<source/path>}`; bare grep/read/ls frames dump byte-identical (unit matrix in `test_sse_events.py` + builder pins in `test_chat_gate.py` L1364–1448)
- Tool line `🔎 Searching for <pattern> in <source/path>` (two `<code>` spans, `.grep-scope` sibling, no new emoji); bare line byte-identical (unit Node test + E2E)
- Status label "… is searching for X in Y" scoped-only; `path` persisted additively on the saved tool record and re-rendered on `/?chat=<id>` restore; pre-phase records restore byte-identically (E2E tests 2 & 3)

**Results (exact commands):**
- `uv run pytest --cov=app --cov-report=term-missing` → **3419 passed**, `app/` coverage **99%** (>90% ✓)
- `uv run pytest tests/e2e/test_search_tool_scope.py -v --no-cov` → **3 passed** in isolation ✓
- `uv run ruff check . && uv run pyright` → clean (0 errors) ✓

**Completion criteria:** all 6 ✓ (wire shape, UI line, status label, save/restore, gates, no completed-phase behavior change — full suite green includes all prior-phase pins).

**Documentation:** README.md (incl. its phase-146 section) and AGENTS.md reviewed in full against the codebase — verified consistent, no changes needed (`.env.example` ⊆ README config tables; no stale vars; phase-100 72rem rule in AGENTS.md §5 matches shipped CSS; no-CDN/smoke-DB/gpgsign claims all confirmed).

**Notable:** none — no defects found, no live-infrastructure touched, nothing committed per harness protocol.

**Next pending phase:** `147_sync_changed_only`.
2026-10-01 12:48:04 -04:00
ducoterra 32a3ec4f5e phase: 145_git_source_links
Build and Push Containers / build-and-push-app (push) Successful in 2m23s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 145 final verification pass — all green, no code changes needed**

- **Verified (no defects found):** resolver `origin_for_source` + `_url_basename` (scp/`.git`/trailing-slash, sanitize, ambiguity WARNING); `origin:`/`branch:` lines in all four read shapes via shared `_origin_lines`; HIGH `<tools>` origin rule (verbatim + only-byte-change pins); `DocContent.origin_url/origin_branch` (absent-never-null serializer); `renderDocument` "Original" anchor (copyable verbatim URL, `_blank`/`noopener`, branch in title); mock scripted origin-link flow (disjoint trigger — grep-verified vs all other E2E files)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3402 passed, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_git_source_links.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- **Completion criteria:** (1) git read results carry origin/branch, local byte-identical — ✅ (unit pins + E2E control); (2) HIGH `<tools>` rule pinned, LOW byte-identical — ✅; (3) modal AND /document.html show Original link, branch in label when pinned, nothing for local — ✅; (4) full suite green + coverage + E2E + lint/types — ✅; (5) no completed-phase behavior change (read header line 1, mock prefix, done frame/SourceRef untouched, LOW sha) — ✅
- **Extra regression check:** re-ran shared-mock E2E suites `test_read_line_ranges.py` (2 passed) and `test_document_viewer.py` (8 passed) — no cross-suite regressions from the `mock_llm.py` addition
- **Documentation:** README.md reviewed in full (env table ⊇ `.env.example`, pages/routes, config defaults, scripts, 17-color palette, 300 s guard, 2 s sync poll, phase-145 provenance section matches implementation) and AGENTS.md (72rem container, no-CDN test, smoke test, gitignore, gates) — **verified consistent, no changes needed**
- **Notable:** phase-116 screenshot PNG was regenerated during this phase's E2E runs (byte re-encode only, visually identical, correct no-anchor local-source meta row) — left in tree for the harness commit
- **Next pending phase:** `146_search_tool_scope` (then 147, 148, 149)
2026-10-01 11:13:43 -04:00
ducoterra 486ceb4019 phase: 144_modal_back_button
Build and Push Containers / build-and-push-app (push) Successful in 2m21s
Build and Push Containers / build-and-push-db (push) Successful in 17s
**Phase 144 final verification pass — complete, 100% green.**

- **Verified:** all 3 tasks already implemented in-tree — `document-modal.js` history-aware modal (one state-only entry per open: push on open, adopt on forward re-open, in-place `replaceState(prev)` on button/Escape/backdrop close, `closingViaBack`-guarded back-close, top-level popstate listener), `tests/unit/test_frontend_doc_modal.py` source pins, `tests/e2e/test_modal_back_button.py` (4 tests). No code fixes needed.
- **Tests:** `uv run pytest` → 3371 passed · `uv run pytest --cov=app --cov-report=term-missing` → 3371 passed, **app/ 99%** (gate >90%) · `uv run pytest tests/e2e/test_modal_back_button.py -v --no-cov` → 4 passed in isolation · regressions in isolation: `test_document_back_navigation.py` 3 passed, `test_kb_tree_nav.py` 4 passed · `uv run ruff check .` clean, `uv run pyright` 0 errors.
- **Completion criteria:** back closes modal (URL/view/drill untouched; second back navigates as usual) ✅ · back→forward re-opens same doc, no duplicate entry ✅ · button/Escape/backdrop close leaves no dead back step ✅ (state byte-identity + modal-free back chain; `replaceState` can't shrink `history.length` — documented deviation, locked design forbids `history.back()/go()`) · all gates green ✅ · no behavior change in completed phases (pre-phase pins + both regression E2Es green) ✅.
- **Docs:** README — one feature-description gap fixed (added the phase-144 Back/Forward modal behavior to the Chat section); all other README facts verified against code (routes, every `.env.example` var, all ~40 documented defaults vs `app/config.py`). AGENTS.md — verified consistent, no changes needed.
- **Decisions:** no live-infrastructure work touched; all changes left uncommitted in the working tree per harness protocol.
- **Next pending phase:** `145_git_source_links`.
2026-10-01 08:49:50 -04:00
ducoterra ba4841a811 phase: 143_mobile_keyboard_after_answer
Build and Push Containers / build-and-push-app (push) Successful in 2m32s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 143 final verification pass — 100% green, no code changes needed (all work verified in-tree)**

- **Verified implementation:** `COARSE_POINTER` constant (L395, L391-twin `typeof`-guarded shape) + gated finally-block focus `if (!COARSE_POINTER) input.focus({ preventScroll: true })` in `frontend/assets/app.js`; SETTLE lines/anchor untouched; extended unit pin in `test_frontend_scroll.py`; dedicated two-context E2E `tests/e2e/test_mobile_keyboard_focus.py`
- **C1 desktop byte-identical refocus:** ✅ unit pin passes (gated shape keeps `preventScroll`); desktop E2E asserts composer active; `test_loading_feedback.py` 5 passed (phase-06 regression)
- **C2 mobile keyboard stays down:** ✅ `test_mobile_turn_end_keeps_keyboard_down` passes — `pointer: coarse` sanity assert first, composer NOT active after done
- **C3 gates:** ✅ `uv run pytest --cov=app --cov-report=term-missing` → 3359 passed, `app/` TOTAL 99% (>90%); `uv run pytest tests/e2e/test_mobile_keyboard_focus.py -v --no-cov` → 2 passed in isolation; `uv run ruff check .` clean; `uv run pyright` 0 errors
- **C4 no regression:** ✅ full suite green (all frontend source pins: char counter, send flow, never-scroll); `test_stop_generation.py` 3 passed (single settle)
- **Docs:** README.md fully reviewed — all env vars cross-checked against `app/config.py` defaults, routes/endpoints/tools verified in code; AGENTS.md rules + quick reference verified against repo. Consistent — no changes needed
- **No live-infrastructure changes** (dev Postgres container already running)
- **Next pending phase:** `144_modal_back_button`
2026-10-01 07:03:15 -04:00
ducoterra aa65d2d691 phase: 142_kb_overview_freshness
Build and Push Containers / build-and-push-app (push) Successful in 2m33s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete. Final report:

**Phase 142 final verification pass — all green**

- **Verified implemented** (working tree matches design; app diff is exactly 2 lines of logic): `app/rag/overview.py` — `regenerate_overview` now orders the lite-model document list `created_at` desc, `(source, path)` tie-break (newest docs survive the cap); `app/rag/prompts.py` — `_KB_INTRO` rewritten to the honest intro (recency bias, "map not inventory", verify-with-`ls`/`grep`; "(generated at import time)" gone).
- **Completion criteria:**
  1. Forced-cap outline leads with newest docs — ✅ E2E `test_kb_overview_freshness.py` test 1 (cap 300, digest from `zz_new/z.md`, no "old" tokens) + integration repro (cap 150 → `mydocs zz new 03 md zz new 03`)
  2. New honest intro served in `<knowledge_base>` — ✅ unit pin (verbatim) + E2E test 2 (`build_kb_section` surface, tag/cap byte-stable)
  3. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3359 passed**, `app/` TOTAL **99%** (>90%); `uv run pytest tests/e2e/test_kb_overview_freshness.py -v --no-cov`: **3 passed** in isolation; `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov`: **4 passed** in isolation; `uv run ruff check . && uv run pyright`: **clean, 0 errors**
  4. No behavior change in completed phases — ✅ cap env knob, change-gated sync trigger, `build_kb_section` cap/marker, deflect path all byte-identical (pinned pins pass in the 3359)
- **Documentation:** README.md reviewed in full (every `.env.example` var documented with matching defaults, all endpoints/routes verified against `app/api/`, Quick Start/deploy/troubleshooting claims verified) and AGENTS.md re-reviewed (rules + quick reference accurate) — **verified consistent, no changes needed**.
- **Decisions:** none required — no defects found; no live-infra changes (only the already-running local `db` container was used by tests).
- **Next pending phase:** `.agents/phases/todo/143_mobile_keyboard_after_answer`.
2026-10-01 04:25:03 -04:00
ducoterra a8c0402ad2 phase: 141_read_range_dedupe_fix
Build and Push Containers / build-and-push-app (push) Successful in 2m37s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All gates green and verified. Final report:

**Phase 141 final verification pass — complete**

- **Verified (tasks 01–04 already in `complete/`):** `AgentHolder.read_coverage` span recording (all 4 read outcomes, refusals record nothing); coverage dedupe matrix (range read of uncovered lines proceeds, fully-covered re-read → `ALREADY_IN_CONTEXT` with a now-true claim, partial plain re-read → `REREAD_PARTIAL` naming merged spans, `REREAD_TRUNCATED` retired, fast path retired per task 02); `TOOLS_SECTION` re-read sentence re-cut to coverage-true form (prompt lock re-pinned); owner-scenario integration repro (pure + truncated variants) and mock `read two ranges` flow + dedicated E2E — all match the task specs exactly; no code defects found.
- **Tests/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL 99% (5498 stmts, `app/rag/agent.py` 99%) — run twice, both green; `uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov` → 2 passed in isolation; `uv run ruff check . && uv run pyright` → All checks passed / 0 errors.
- **Completion criteria:** (1) two different ranges both reach the model, re-covered re-read refused truthfully — ✅ (integration + E2E); (2) partial plain re-read gets the `offset`/`limit` teaching, false claim gone both directions — ✅; (3) plain-read + phase-131 result shapes byte-identical, existing pins pass unchanged — ✅; (4) all gate commands green, >90% — ✅; (5) no behavior change in completed phases (grep/ls, truncation notice, `read_docs`/`read_truncations` signals) — ✅ (whole project suite green).
- **Documentation:** README.md — fixed pre-existing phase-140 gap: `BOR_PDF_PICTORIAL` / `BOR_PDF_PICTORIAL_MIN_CHARS` were in `.env.example`/Settings but undocumented (README claims every `.env.example` var is covered); added the pictorial-PDF-pass bullet to the PDF section + two config-reference rows. AGENTS.md — verified consistent (rules, quick-ref commands, phase refs all check out) — no changes needed.
- **Notable:** overview's "fast path survives for holder-state rows" note is superseded by task 02's authoritative decision (every dedupe row resolves — "full" needs the true line count); implementation follows task 02. No phase files touched; changes left in the working tree for the harness commit.
- **Next pending phase:** `142_kb_overview_freshness`.
2026-10-01 01:27:46 -04:00
ducoterra de49a03c4d add NOTES.md
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-30 22:31:54 -04:00
1724 changed files with 195020 additions and 4189 deletions
+41 -4
View File
@@ -84,6 +84,25 @@ A15 — `thinking`, `tool`, unlimited tool rounds under
never-truncated contract is unchanged; the `read`-tool cap is never-truncated contract is unchanged; the `read`-tool cap is
the only exception, owner-permitted 2026-09-10. the only exception, owner-permitted 2026-09-10.
Phase 182 (verbatim text display, owner chat 2026-10-14) adds
**`show_text(path, offset?, limit?)` as the fifth agent tool** — offered
only when `BOR_ANSWER_TEXT=1` (the default), gated exactly like
`show_image`, with its own `<tools>` clause and its own kill switch — and
the **additive `done.answer_text`** field above. The new settings are
`BOR_ANSWER_TEXT` (master switch, ON), `BOR_SHOW_TEXT_MAX_CHARS` (per-block
budget, 12 000) and `BOR_SHOW_TEXT_MAX_BLOCKS` (per-answer budget, 5), all
following the `agent_max_rounds` pattern (`0` inert, negative refuses
startup). The principle in one sentence: **the model points, the server
supplies the bytes** — a ` ```bor source/path 12-40 ` fence or a
`show_text` call names a location (canonical locator `source/path` or
`source/path#start-end`, 1-based and inclusive), and the displayed text is
sliced from `documents.content` server-side, so what the user copies is the
knowledge base, never a re-typed approximation. **A1–A6 and A15 are
otherwise
UNCHANGED:** no new SSE event type, no new table or column, no new model,
endpoint or transport, no change to retrieval, the honesty gate, the
persona text or the embedding dimension.
--- ---
## 3. High-Level Architecture ## 3. High-Level Architecture
@@ -129,7 +148,9 @@ POST /api/chat {message, history?} (auth: require_user)
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7] → cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0): ├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
│ persona + <knowledge_base> + <tuning> + full top-2 <documents> │ persona + <knowledge_base> + <tuning> + full top-2 <documents>
│ + <tools> → agent loop (ls/read/grep; round-capped │ + <tools> → agent loop (ls/read/grep, plus show_image when
│ BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
│ "show the user an artifact" tools, gated independently; round-capped
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed │ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools, └─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
byte-identical direct-stream path) → turbo streamed byte-identical direct-stream path) → turbo streamed
@@ -139,8 +160,20 @@ POST /api/chat {message, history?} (auth: require_user)
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) → SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame `tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[], restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no suggestions[], budget_exhausted?, answer_text[]?}` (terminal). **Phase 182
`query_log` row; a pre-stream DB outage is a plain 503 JSON). extends the `done` frame ADDITIVELY** — `answer_text` is the list of
server-resolved verbatim blocks (`locator`, `source`, `path`, `start`,
`end`, `total_lines`, `text`, `format`, `truncated`) the answer displayed;
there is **no eighth event type**, the key is OMITTED when no block was
displayed, and with `BOR_ANSWER_TEXT=0` it is absent on every turn (the
frame is then byte-identical to pre-phase). The same array persists on the
saved chat record, so restore and the anonymous shared page re-render the
panels from the server's bytes. Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
`tools=None` request never emits a `tool` frame; a round-capped turn
whose final reply carries no visible content answers with the budget
fallback instead of erroring (phase 160, recorded revision of the
phase-71 cap-path terminal, owner-confirmed 2026-10-05).
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars `BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
still counted). LLM retries (A15 extension): retry count and delay are still counted). LLM retries (A15 extension): retry count and delay are
configurable (defaults: 3 retries, 5 s flat delay), **only before a configurable (defaults: 3 retries, 5 s flat delay), **only before a
@@ -268,8 +301,12 @@ scripts.…`, and tests all honor it.
``` ```
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
kb_chars=N history_msgs=N threshold=… deflected=… sources=… kb_chars=N history_msgs=N threshold=… deflected=… sources=…
thinking_chars=… tool_calls=N total_ms=… retries=N scaffold_stripped=N thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
retries=N scaffold_stripped=N verbatim_blocks=N
``` ```
(`verbatim_blocks=` — phase 182 — is the count of verbatim display blocks
actually SENT on the `done` frame, `0` on every turn without them,
deflected included.)
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=` (`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
executed only; a cancelled turn writes neither line nor `query_log` executed only; a cancelled turn writes neither line nor `query_log`
row.) row.)
@@ -0,0 +1,47 @@
# Phase 150 — retire `span.sync-result`: `#sync-progress` is the single sync text surface
**Source:** `TODO.md` L5 — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a (TODO-derived phase, Protocol B append).
**Context:** the Sources page (`frontend/index.html`, the `view-rag` section). Phase 149 added `#sync-progress` (L493–495 — `<div class="sync-progress" id="sync-progress" hidden aria-hidden="true">` + the inner `<span class="sync-progress-text" id="sync-progress-text">`), the bordered panel under the button row that carries the full untruncated live label while a run is in flight. The pre-149 announcer survived beside it: `<span class="sync-result" id="sync-result" role="status" aria-live="polite">` (L511) — today the SOLE aria-live region, which receives (a) the same live label while a run runs (a byte-identical duplicate of the box), (b) the settle counts line (`fmtSyncResult(detail)` — "N added · M updated · …"), (c) the "Sync cancelled" line, (d) the cancel-request failure copy, (e) `""` on the sync-failure / idle / upload-settle paths (plus the settle label "Synced HH:MM" and the idle "Sync sources", written through the shared writer). The single writer is `frontend/assets/sources.js` `setSyncLabel(mainText, countText, chunkText, running)` (L809–843): it writes the button parts, the `title`, the box text + `hidden = !running`, the button `aria-label`, AND `syncResult.textContent = full` (L834). The `syncResult` element handle is L750; the other write sites are L1092 (`applySyncSuccess` — the counts line), L1107 (`applySyncIdle` — clear), L1125 (`applySyncCancelled` — "Sync cancelled"), L1213/L1224 (the upload settles — clear), L1291/L1298/L1310 (the cancel paths — clear + "Couldn't cancel the sync — try again."). CSS: `.sync-progress` L2252, `.sync-progress-text` L2259, `.sync-result` L2284–2290, the mobile `.sync-result` override at L5639, and the "the sync-result shape" comments at L2938 / L3659 (those rules belong to OTHER elements — the upload success line and the action feedback line — and keep their own classes).
**Locked (owner decision, chat — this phase):** `#sync-progress` becomes the **single** sync text surface AND the **sole** aria-live region: `role="status" aria-live="polite"` moves onto the box, `aria-hidden` is dropped, and `span.sync-result` is **deleted** from the template. The box keeps the phase-149 running behavior AND takes over the announcer's content: it is visible while a run is in flight (the live label) and stays visible after a settle to carry the result line (the counts / "Sync cancelled" / cancel-failure copy), hiding only when its text is empty (idle, upload settles, failures — exactly the states where `#sync-result` is empty today). Every string that reaches `#sync-result` today reaches the box instead — no copy changes.
## Objective
One element, one live region: the sync/upload live label, the settle result line, the cancel line and the click copy all render in `#sync-progress`, which announces them exactly once; `#sync-result` no longer exists in the DOM, the JS, or the CSS.
## Dependencies
- `149_sync_progress_box` (complete) — the box, its single-writer wiring, and its CSS are what this phase folds the announcer into. Pipeline order only: no phase is pending in `todo/`.
## Design (shared by all tasks — the executor reads this, not the chat)
- **The element (task 01):** `#sync-progress` keeps its id, class, and inner `#sync-progress-text` span (the textContent target — the house rule: text only, never HTML) and gains `role="status" aria-live="polite`; `aria-hidden="true"` is REMOVED (an aria-live region that is aria-hidden announces nothing — the two attributes are mutually exclusive by intent). It stays ship-hidden (`hidden`) — the anonymous-safe/idle state. `span.sync-result` (L511) and its explanatory comment block (L496–510) are deleted; the box's comment (L483–492) is rewritten to say it is now the SOLE announcer (the phase-149 "presentational (aria-hidden)" sentence is retired — leave a one-line note that phase 150 folded the announcer in, the file's pinned-comment convention).
- **The writer (task 02):** introduce ONE function that owns the box — `setSyncProgress(text)` — which sets `syncProgressText.textContent = text` and `syncProgress.hidden = (text === "")` (the global `[hidden]` rule does the display). Two content modes, decided at the callsites (the same "the callsites are the only decision sites" rule phase 149 established):
* **running mode** — `setSyncLabel(main, count, chunk, running)` keeps writing the box with `full` **only when `running === true`** (exactly today's reveal rule). When `running === false` it does NOT touch the box: the idle/settle button labels ("Sync sources", "Synced HH:MM") must not fill a big panel (today they land on the small `#sync-result` line where they read as a duplicate of the button — that artifact is retired, not migrated).
* **result mode** — the settle paths call `setSyncProgress(text)` explicitly: `applySyncSuccess` → `fmtSyncResult(status.detail)` (the counts line), `applySyncCancelled` → `"Sync cancelled"`, `applySyncFailure` → `""` (the banner + modal carry the error), `applySyncIdle` → `""`, the two upload settles → `""` (the upload result lives on the Sources page — phase 90 A3), `requestCancel`'s network/non-2xx branches → `"Couldn't cancel the sync — try again."` (string unchanged). `requestCancel`'s opening clear (L1291, which today blanks the announcer while the run is still in flight) becomes a NO-OP on the box — the live label stays up until the poll's `cancelled` branch settles it (blanking would hide the running surface for up to one 2 s poll).
The `syncResult` handle (L750) and every `if (syncResult)` guard are deleted. There is no 202/409 click copy: `startSync`'s 202/409 branch enters the running state (L1255–1262), so the box reveals the bare "Syncing…" label through `setSyncLabel` as today.
- **The CSS (task 03):** delete the `.sync-result` rule (L2284–2290) and the mobile `.sync-result` override (L5639); the two "the sync-result shape" comments (L2938, L3659) are reworded to reference `.sync-progress-text` (they describe the mono/soft-ink language those other elements share — the rules themselves and their selectors are untouched). `.sync-progress` needs NO new rule for the settled state (the panel already reads as a calm box; the global `[hidden] { display: none !important }` rule keeps it gone when empty — verify it still applies, the box sets no `display` of its own). Contrast stays AA: `.sync-progress-text` uses `--ink-soft` on `--surface` (8.2:1 built-in dark, AA in both built-ins).
- **The test migration (tasks 04/05):** mechanical selector migration — `#sync-result` → `#sync-progress` (assert on the box's text, i.e. `#sync-progress-text` for textContent, `#sync-progress` for `hidden`/attributes). Settled-state copy assertions must stay byte-identical (the counts line string itself does not change).
- **ASSUMPTION A1 (owner-locked):** the box stays visible after settle to hold the counts line and hides only when its text is empty. The alternative (box running-only, result line elsewhere) was rejected.
- **ASSUMPTION A2:** the settle label "Synced HH:MM" is no longer written to the live region (it stays on the button, visible). The announcer hears the counts line that superseded it in the same tick a moment ago — the merge makes one surface, and a panel reading "Synced HH:MM" under a button reading "Synced HH:MM" is the duplicate this retirement exists to remove.
- **ASSUMPTION A3:** the box's settled appearance needs no new CSS (the same bordered panel reads fine for a result line); only the running-only min-height stays, so the panel does not jump between modes.
## Tasks
1. `01_html_single_surface.md` — delete `span#sync-result`, move `role=status aria-live=polite` onto `#sync-progress`, drop `aria-hidden`, rewrite the two comment blocks.
2. `02_js_single_writer.md` — `sources.js`: add `setSyncProgress`, route all 9 announcer sites through it, delete the `syncResult` handle, change the box visibility rule to text-driven.
3. `03_css_retire_sync_result.md` — `styles.css`: remove `.sync-result` + its mobile override, reword the two shape comments.
4. `04_unit_test_migration.md` — `tests/unit/test_sync_button.py` (20 refs) + `tests/unit/test_frontend_sync_upload.py` (26 refs) migrated to the box.
5. `05_e2e_sync_family.md` — the 12 sync-family E2E files' selectors migrated + the dedicated `tests/e2e/test_sync_progress_box.py` extended with the single-live-region + settled-line assertions.
6. `06_e2e_image_pdf_family.md` — the 6 image/PDF E2E files migrated (their sync-done barriers move to the box) + the a11y/no-CDN/full-gate pass and the phase commit.
## Testing & Quality
- Unit: the two frontend-parsing suites (`test_sync_button.py`, `test_frontend_sync_upload.py`) pin the writer contract on the box — running label visible + announced, settle counts line, "Sync cancelled", the idle/upload clears, the "Couldn't cancel" copy — and assert `#sync-result` is ABSENT from the template (the retirement is pinned, not implied).
- E2E: `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` in isolation (the slow-LLM proxy recipe already in that file) proves in-flight → settle → idle on the real sync path; every pre-existing sync/image/PDF E2E that touched the announcer passes after migration (12 sync-family files + 6 image/PDF files).
- Coverage: **>90%** on `app/` (this phase touches no `app/` logic — the floor must simply not regress).
- Accessibility gate (AGENTS.md rule 5): exactly ONE `aria-live` region in the sync area; the box is a `status` region (polite — no alert spam); the button keeps its accessible name in both states; focus-visible and contrast unchanged.
## Completion Criteria
- [ ] `grep -rn "sync-result\|syncResult" frontend/ tests/` returns NOTHING (the element, its class, and its JS handle are gone from source and tests)
- [ ] `#sync-progress` carries `role="status" aria-live="polite"` and no `aria-hidden`; it is the only `aria-live` element in the sync area
- [ ] in flight: the box shows the full untruncated live label (wrapping, never clipped); at settle: the box shows the counts line byte-identical to the pre-phase `#sync-result` string ("N added · …") and "Sync cancelled" for a cancel; at idle / upload-settle / sync-failure: the box is `hidden` and empty
- [ ] the button is still the icon-only spinner while running and restores "Sync sources" on settle (phase 149 behavior unchanged)
- [ ] `uv run pytest` green; `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] no behavior change in completed phases (the cancel control, the 409 double-trigger, the error banner/modal, the upload run's status, the sync status API)
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,28 @@
# Task 01 — `index.html`: delete `span#sync-result`, make `#sync-progress` the live region
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
The template ships ONE sync text surface: `#sync-progress`, which is now the `role="status"` `aria-live="polite"` announcer; `span.sync-result` no longer exists in the DOM.
## Work
1. `frontend/index.html` (the `view-rag` Sources section, L483–511):
- `#sync-progress` (L493) becomes
`<div class="sync-progress" id="sync-progress" role="status" aria-live="polite" hidden>`
— `aria-hidden="true"` is REMOVED (an aria-live region must not be aria-hidden), `role`/`aria-live` are ADDED, `hidden` stays (ship-hidden = the idle state, anonymous-safe). The inner `<span class="sync-progress-text" id="sync-progress-text"></span>` (L494) is unchanged (it stays the textContent target).
- DELETE the `#sync-result` span (L511) and its comment block (L496–510, the "#sync-result is the aria-live announcer …" paragraph).
- REWRITE the box's own comment (L483–492) so it states the phase-150 contract: the box carries the full untruncated live label while a run is in flight AND the settle result line after it; it is the SOLE aria-live announcer for the sync area (one live region, no double announce); the button is the spinner while running (`.is-running` hides the label parts). Keep the file's comment voice (phase-prefixed, rationale-first).
2. Nothing else in the template changes: `#sync-btn` and its label parts, `#sync-cancel-btn`, `#sync-error-banner` / `#sync-error-text`, the sync modal, and the Sources gate are untouched.
3. No CDN/external references are introduced (AGENTS.md rule 6 — the page stays fully self-hosted).
4. ASSUMPTION: the box keeps its current position (between `.page-head` and the error banner) — the owner asked to remove the span, not to move the box.
## Testing & Quality
- Unit: this task's own gate is the template parse — the pre-existing suites that read `index.html` still find `#sync-progress` (with the new attributes) and no longer find `#sync-result`. Tasks 04/05 do the full migration; if a suite fails ONLY on the missing `#sync-result`, note it for task 04 rather than editing test expectations here beyond what the parse requires.
- Coverage: n/a (template task — `app/` coverage floor unchanged).
## Completion Criteria
- [ ] `grep -n "sync-result" frontend/index.html` returns nothing
- [ ] `#sync-progress` has `role="status"`, `aria-live="polite"`, `hidden`, and NO `aria-hidden`
- [ ] the sync area contains exactly one `aria-live` element
- [ ] `uv run pytest tests/unit -q` is green or failing ONLY on the `#sync-result` expectations tasks 04/05 own
@@ -0,0 +1,40 @@
# Task 02 — `sources.js`: one box writer (`setSyncProgress`), the announcer handle deleted
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
Every string that used to land on `#sync-result` lands on `#sync-progress` instead, written by one function; the `syncResult` handle is gone and the box's visibility is owned by that single writer.
## Work
1. `frontend/assets/sources.js` — add the box writer next to `setSyncLabel` (L809):
```js
function setSyncProgress(text) {
if (!syncProgressText || !syncProgress) return;
syncProgressText.textContent = text;
syncProgress.hidden = text === "";
}
```
(textContent only — the house rule, never HTML; the `hidden` attribute drives `display:none` through the global `[hidden]` rule.)
2. `setSyncLabel` (L809–843): replace the phase-149 pair `syncProgressText.textContent = full;` + `syncProgress.hidden = !running;` (L838–839) with `if (running) setSyncProgress(full);` — the running label still flows through the single writer, and a non-running call leaves the box alone (the idle/settle button labels "Sync sources" / "Synced HH:MM" must NOT fill the panel; the panel is the live label + the result line, nothing else). The `title`, the `aria-label` (`running ? full : "Sync sources"`), and the label parts are untouched. Delete `if (syncResult) syncResult.textContent = full;` (L834) — the box IS the announcer now.
3. Delete the element handle `const syncResult = root.querySelector("#sync-result");` (L750) and every `if (syncResult)` guard. Migrate the eight remaining announcer sites to `setSyncProgress(...)`, at the SAME position in the same order:
- `applySyncSuccess` (L1092) → `setSyncProgress(fmtSyncResult(status.detail))` (the counts line — the string builder is unchanged);
- `applySyncFailure` (L1107) → `setSyncProgress("")` (the banner + modal carry the error; the box hides);
- `applySyncCancelled` (L1125) → `setSyncProgress("Sync cancelled")`;
- `applySyncIdle` (L~1130) → `setSyncProgress("")` (the true idle — retry-ready, nothing to announce);
- the upload-success settle (L1213) and the upload-failed settle (L1224) → `setSyncProgress("")` (the upload's result line lives on the Sources page — phase 90 A3);
- `requestCancel`'s network-failure branch (L1298) and its non-2xx tail (L1310) → `setSyncProgress("Couldn't cancel the sync — try again.")` (string byte-identical);
- `requestCancel`'s opening clear (L1291) becomes a **no-op on the box** — the run is still in flight at that moment, so blanking would hide the running surface for up to one 2 s poll; the poll's `cancelled` branch settles it. Leave a short comment saying why the clear was dropped (the phase-134 "never stale" contract is kept by the control's immediate disable).
4. Update the module/function comments that name `#sync-result` as the announcer (the file cites it in ~15 comment lines — L160, 321, 375, 670, 677, 700, 706, 723, 744, 789, 859, 894, 949, 1006, 1120, 1209, 1256, 1274, 1286): rename the references to `#sync-progress` and keep the rationale (one live region, the untruncated label, A4). Do not change any behavior while doing it.
5. ASSUMPTION A2 (owner-visible): the settle label "Synced HH:MM" is no longer written to the live region — it stays on the button; the counts line is the announced result.
6. ASSUMPTION: the re-attach path (`initSyncButton`, the page-load mid-run entry) needs no new code — it goes through `enterSyncRunningState` → `setSyncLabel(..., true)`, which reveals the box (the never-stale contract holds).
## Testing & Quality
- Unit: task 04 migrates the two frontend suites; this task's own gate is `uv run pytest tests/unit -q` failing only on the expectations task 04 owns, and `grep -n "syncResult" frontend/assets/sources.js` returning nothing.
- Coverage: n/a (frontend task).
## Completion Criteria
- [ ] `grep -n "syncResult\|sync-result" frontend/assets/sources.js` returns nothing
- [ ] `setSyncProgress` is the only writer of the box's text/visibility; `setSyncLabel` writes it only when `running`
- [ ] the counts line, "Sync cancelled", and "Couldn't cancel the sync — try again." all render in `#sync-progress`; the idle/upload-settle/sync-failure states leave it hidden and empty
- [ ] the running label, the button `title`, and the button `aria-label` behavior are byte-identical to pre-phase
@@ -0,0 +1,27 @@
# Task 03 — `styles.css`: retire `.sync-result`, keep the box correct in both modes
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
No CSS rule targets the deleted element, and `#sync-progress` reads correctly as both the running label and the settled result line.
## Work
1. `frontend/assets/styles.css`:
- DELETE the `.sync-result` rule (L2284–2290 — `display:block; color:var(--ink-soft); font-family:var(--mono); font-size:0.8rem; white-space:nowrap; padding-block:0.5rem;`) and its "Last-result announcer" comment.
- DELETE the mobile `.sync-result` override inside the phone media query (L5639) — read it first and confirm it only tweaks the deleted element (if it also styles a surviving selector, keep that part).
- REWORD the two comments that reference "the sync-result shape" (L2938 — the upload success line `role=status`; L3659 — the action feedback line `role=status`) to name `.sync-progress-text` as the shared mono/soft-ink language. The RULES themselves (their own selectors) are untouched — only the cross-reference text changes.
- `.sync-progress` (L2252) / `.sync-progress-text` (L2259) need no new rule: the panel already carries the border/surface/radius/padding and the `min-height: 4rem`; the box sets no `display` of its own, so the global `[hidden] { display: none !important }` rule keeps it gone when empty (verify by reading that global rule — if it is scoped to something narrower, add `.sync-progress[hidden] { display: none; }`).
- Update the `.sync-progress-text` comments that say "the .sync-result mono language / size / ink" (L2261–2266) so they no longer cite a dead class (keep the contrast note: `--ink-soft` on `--surface` is a pinned AA pair).
2. No new colors, no new fonts, no external assets (AGENTS.md rule 6). Contrast stays ≥4.5:1 in both built-in themes (the ink/surface pairs are unchanged).
3. ASSUMPTION A3: the settled result line uses the same panel styling as the running label (no `.is-settled` variant) — the owner asked for one surface, not two looks.
## Testing & Quality
- Unit/E2E: the migrated suites (tasks 04/05) assert the box's visibility states; the theme suites (`tests/unit/test_theme*.py`, the custom-theme contrast gate) pass unchanged — no theme token changed.
- Coverage: n/a (CSS task).
## Completion Criteria
- [ ] `grep -n "sync-result" frontend/assets/styles.css` returns nothing
- [ ] `#sync-progress` is `display:none` while `hidden` and a bordered panel when shown, at desktop and 360 px
- [ ] the upload success line and the action feedback line still render exactly as before (their own rules untouched, only comments reworded)
- [ ] `uv run pytest tests/unit -q` green
@@ -0,0 +1,29 @@
# Task 04 — unit suites: migrate the `#sync-result` source pins to the box
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
The source-level suites that pin the sync UI no longer reference `#sync-result` / `.sync-result`, and they pin the new contract instead: one live region, box-driven.
## Work
1. `tests/unit/test_sync_button.py` (20 references) — the pins are regex/string reads of `frontend/index.html`, `frontend/assets/sources.js`, `frontend/assets/styles.css`:
- L232–241 (`test …sync-result…`): the "polite live region right after the button" pin moves to `#sync-progress` — assert the div carries `role="status"` and `aria-live="polite"`, does NOT carry `aria-hidden`, and sits after the button row; rename the test accordingly;
- ADD a pin that the retired element is gone: `assert 'id="sync-result"' not in index_html` (the retirement is pinned, not implied);
- L747–751 (the `.sync-result` CSS block pin): retarget to `.sync-progress-text` (assert the mono family, the `--ink-soft` color, the `overflow-wrap: anywhere`) and assert `.sync-result` has no rule in `styles.css`;
- L1984 (the DOM-order pin `main_end < cancel_start < sync-result`): re-anchor on `id="sync-progress"`;
- L371, 384, 465, 652, 2067 and the module docstring (L11, 43, 56): reword the "title + #sync-result" language to "title + #sync-progress" (the A4 untruncated-label contract is unchanged — it just names the surviving surface);
- the cancel-request pin (~L2067) that asserted the announcer clears on click: update to the new contract — the box is NOT blanked at cancel-request time (the live label stays until the poll settles), the control disables immediately.
2. `tests/unit/test_frontend_sync_upload.py` (26 references) — same migration: the `fmtSyncLabel` / `enterSyncRunningState` pins that assert the full untruncated label reaches the announcer now assert it reaches `#sync-progress` (via `setSyncProgress` / the `running` branch of `setSyncLabel`); the A3 pin ("the settle never renders upload counts into #sync-result") becomes "the upload settles write an empty string to the box" (`setSyncProgress("")`); the docstring references (L9, 12, 61, 79) are reworded.
3. Keep every assertion's INTENT: same strings, same states, one different selector. Do not weaken a pin to make it pass — if a behavior genuinely changed (the "Synced HH:MM" announcer write, the cancel-request clear), pin the NEW behavior explicitly and say why in the test docstring.
4. ASSUMPTION: no new test file is created for the unit layer — the retirement is a migration of existing pins plus the two "element is gone" assertions.
## Testing & Quality
- `uv run pytest tests/unit/test_sync_button.py tests/unit/test_frontend_sync_upload.py -v` green.
- `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage stays >90% (no `app/` change in this phase).
- Coverage: n/a for frontend source; the two suites must not lose assertions (compare the collected test count before/after — it must not drop).
## Completion Criteria
- [ ] `grep -rn "sync-result\|syncResult" tests/unit/` returns nothing
- [ ] both suites assert the box is the `role="status" aria-live="polite"` region and that `#sync-result` is absent from the template and from `styles.css`
- [ ] test count is not lower than before the migration; both files green
@@ -0,0 +1,31 @@
# Task 05 — E2E migration (sync family) + the dedicated `test_sync_progress_box.py` assertions
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
Every sync-flow Playwright suite drives and asserts the box instead of the deleted span, and the phase's dedicated suite proves the merged contract end to end (one live region, running label + settled result line).
## Work
1. Migrate the sync-family suites (the `sync-result` reference count in parentheses) — replace the selector `#sync-result` with the box (`#sync-progress` for visibility/attributes, `#sync-progress-text` for `text_content()`), keeping every expected STRING byte-identical:
- `tests/e2e/test_sync_button.py` (2), `tests/e2e/test_sync_cancel.py` (2 — the "Sync cancelled" line and the cancel lifecycle), `tests/e2e/test_sync_model_down.py` (2), `tests/e2e/test_sync_upload_progress.py` (2), `tests/e2e/test_upload_no_scan.py` (2), `tests/e2e/test_sync_summary_visibility.py` (3), `tests/e2e/test_sync_changed_only.py` (3), `tests/e2e/test_sync_label_mobile.py` (1), `tests/e2e/test_git_source_dates.py` (1), `tests/e2e/test_git_source_branch.py` (4), `tests/e2e/test_force_push_resync.py` (4), `tests/e2e/test_sync_chunk_progress.py` (8 — the "· chunk n/m" announcer pins).
- Where a suite asserted "the announcer is empty" it now asserts `expect(page.locator("#sync-progress")).to_be_hidden()`; where it asserted announcer text it asserts the box's text. A suite that asserted the announcer was VISIBLE while running keeps that assertion on the box (it is visible while running — phase 149).
2. `tests/e2e/test_sync_progress_box.py` (11 references) — the dedicated suite for this phase (A16: one file per story, run in isolation). Extend it, keeping the existing running-state proofs:
- **template pin:** exactly one `[aria-live]` element in the sync area, and it is `#sync-progress`; `#sync-result` is absent (`page.locator("#sync-result").count() == 0`);
- **running:** unchanged (icon-only button + the full untruncated label in the box, the long `source/relative/path` wrapping at 360 px and desktop);
- **settled success:** the box is VISIBLE and its text equals the counts line the pre-phase announcer produced (the `fmtSyncResult` shape — "N added · …"), the button reads "Synced HH:MM";
- **settled cancel:** the box reads "Sync cancelled" (the phase-134 neutral result — no banner, no modal);
- **settled failure:** the box is hidden/empty and the error banner + modal carry the error (unchanged);
- **idle re-attach:** a page load with no run leaves the box hidden.
3. Keep the suites MOCK-ONLY (the `tests/e2e/slow_llm.py` delay-proxy recipe already in the file) and run each migrated file in isolation while editing it: `uv run pytest tests/e2e/test_<name>.py -v --no-cov` (DB up: `podman compose up -d db`).
4. ASSUMPTION: no new E2E file — the phase's behavioral proof belongs to the existing `test_sync_progress_box.py` (the phase-149 dedicated suite is the box's suite; this phase extends it rather than forking a near-duplicate).
## Testing & Quality
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green IN ISOLATION (the phase's gate file).
- Every migrated sync-family file green in isolation; then the full `uv run pytest` green.
- Coverage: >90% on `app/` (unchanged — no `app/` edit).
## Completion Criteria
- [ ] `grep -rln "sync-result" tests/e2e/` returns nothing after task 06 also lands (this task clears the sync family)
- [ ] the dedicated suite proves: one live region, running label in the box, counts line visible at settle, "Sync cancelled" on cancel, hidden on failure/idle
- [ ] `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation
@@ -0,0 +1,30 @@
# Task 06 — E2E migration (image / PDF family) + the no-CDN and a11y gate pass
**Phase:** `150_drop_sync_result` · **Source:** `TODO.md:5` — "Get rid of the span sync-result now that we have the div sync-progress"
**Story:** n/a
## Objective
The remaining suites that touch the announcer — the image and PDF stories, which wait on sync result text — drive the box, and the whole gate passes with the span gone.
## Work
1. Migrate the image/PDF-family suites (reference count in parentheses), same rule as task 05 — selector changes only, expected strings byte-identical:
- `tests/e2e/test_image_documents.py` (2), `tests/e2e/test_image_anchor_context.py` (2), `tests/e2e/test_answer_images.py` (2), `tests/e2e/test_pdf_support.py` (2), `tests/e2e/test_pdf_image_context.py` (2), `tests/e2e/test_pictorial_pdf.py` (2).
- These use the announcer mostly as a SYNC-DONE barrier (wait for the counts line before querying the KB). Keep them as barriers, just on the box: wait for `#sync-progress-text` to contain the expected result text (or for the box to become visible with the counts line) — do NOT replace them with `wait_for_timeout`.
2. Confirm the no-CDN integration test still passes (AGENTS.md rule 6 — the index page has no external `<script src=>`/`<link href=>`): the template edit added no external reference.
3. Accessibility pass on the merged surface (AGENTS.md rule 5, PLAN §7): exactly one `aria-live` region in the sync area; `role="status"` (polite — the failure path keeps its `role="alert"` banner); the button keeps an accessible name in both states (`aria-label` full label while running, "Sync sources" otherwise); `:focus-visible` on the button + Cancel untouched; `--ink-soft` on `--surface` contrast ≥4.5:1 in both built-in themes (unchanged tokens).
4. Full gate: `uv run pytest` (unit + integration), `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright` clean, and the phase's E2E file in isolation.
5. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit, `--no-gpg-sign`, e.g.
`feat(ui): retire #sync-result; #sync-progress is the single sync surface + live region`
(includes `.agents/phases/todo/150_drop_sync_result/` moved to `.agents/phases/complete/`).
## Testing & Quality
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` green in isolation (the phase gate file).
- The six migrated image/PDF suites green in isolation, then the full suite green.
- Coverage: `app/` >90% (no `app/` change).
## Completion Criteria
- [ ] `grep -rn "sync-result\|syncResult" frontend/ tests/ app/` returns NOTHING anywhere in the project
- [ ] the six image/PDF suites still gate on a real sync-done signal (no sleeps introduced)
- [ ] one `aria-live` region in the sync area; contrast/focus-visible unchanged; no external CDN references
- [ ] `uv run pytest` green, `uv run ruff check . && uv run pyright` clean, coverage >90%
- [ ] the phase directory is in `.agents/phases/complete/` and committed with `--no-gpg-sign`
@@ -0,0 +1,56 @@
# Phase 151 — no more blank white image documents from PDFs (dedupe + soft masks + a blank gate)
**Source:** `TODO.md` L4 — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a (TODO-derived phase, Protocol B append).
**Context:** `app/rag/pdf.py` — `extract_pdf_images()` (the module's image extractor: `page.get_image_info(xrefs=True)` per page via `_page_image_entries`, then `doc.extract_image(xref)` per entry, one `PdfImage(page, index, width, height, data, mime)` per **occurrence**) and its consumer `app/rag/importer.py` `_index_pdf_embedded_images()` (L1924 — the per-PDF embedded-image pass: `_pdf_image_context_inputs` once, then `extract_pdf_images(full_path)` and one `_index_one_embedded_image()` (L2030) per image: mime→ext gate through `_MIME_TO_EXT` (L1340 — png/jpg/gif/webp/bmp/tiff; anything else maps to `.bin` and is skipped), sha256-digest skip, one vision describe call (`_describe_or_skip`, L2907), the `image_dir` copy, chunk + embed, one transaction per image). The `images` toggle (`BOR_IMAGES`, `settings.images`, default OFF) gates the whole pass (LOCKED A3). `ImportSummary` (L390–475) carries the log-only counters (`images_failed`, `pdfs_screenshotted`, `pdfs_pictorial`, `pdf_pictorial_pages`, `image_context_backfilled`) — the phase-133 stats log-only convention: counters land in the `import: summary …` line, never in the sync `detail` API shape.
**Reproduced on the owner's file (`test_sources/podman-systemd.pdf`, 50 pages) — three distinct defects:**
1. **Duplicates:** `get_image_info` reports one entry per *placement*, so the file yields **46 occurrences of only 15 unique xrefs** (each image is placed 2–4 times). Every occurrence becomes its own document (`…#page=N#img=i.<ext>`), so the same image is vision-described and embedded up to 4×.
2. **Soft masks ignored:** every one of the 15 images carries an `/SMask` (`extract_image` reports `smask=<xref>`, `get_image_info` reports `has-mask=True`). `extract_image()` returns the **base** sample stream only — the mask (which carries the visible shape) is dropped, so the stored bytes are not what a PDF viewer shows.
3. **Decorative underlays indexed:** those images are table-shading underlays — `DeviceGray`, `mean=238/255`, **8 unique sample values**, **99.5% of samples within ±8 of the mean** (verified: composited with their mask they are still a near-white block; the table text a viewer shows is real PDF text drawn on top). They are indexed as documents, vision-described, and returned as image sources.
Measured comparison (the same probe, `fitz.Pixmap(...).samples`): the podman underlays `uniq=8`, near-uniform(±8) `0.95–0.995`; `test_sources/pdf_horse_test.pdf`'s photo `uniq=256`, `0.14`; `test_sources/players_handbook.pdf` page art `uniq≥188`, `0.10–0.59`. The gate separates them cleanly.
**Locked (owner decisions, chat):** blank/decorative images are **skipped entirely** (never indexed, never described — not stored-and-flagged); JPEG2000 (`jpx`) images stay **skipped** as they are today (the unsupported-extension gate keeps running on the RAW embedded format, so compositing to PNG must not newly admit the 586 `jpx` images in `players_handbook.pdf`).
## Objective
A PDF's embedded-image pass yields one document per genuinely distinct, non-decorative image, with the bytes a viewer would show (soft mask applied) — no blank white documents, no triplicate images, and far fewer vision calls per PDF.
## Dependencies
- `150_drop_sync_result` (todo) — pipeline order only (that phase touches the Sources-page sync UI; this phase touches `app/rag/pdf.py` + the importer's image pass). No shared files.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Dedupe (task 01):** in `extract_pdf_images`, keep the FIRST occurrence of each distinct image and skip the repeats — key = the `xref` when non-zero, else the sha256 of the final `data` (a digest set also catches byte-identical images stored under different xrefs). The kept `PdfImage` carries the first occurrence's `page`/`index` (the synthetic path `…#page=N#img=i.<ext>` is therefore stable for the first placement). One INFO line per PDF when duplicates were dropped (`pdf: deduped N duplicate image occurrence(s) in <path>`), DEBUG per image.
- **Soft masks (task 02):** when the info/`extract_image` dict reports a non-zero `smask`, build the VISIBLE image instead of the raw base: `base = fitz.Pixmap(doc, xref)`, `mask = fitz.Pixmap(doc, smask_xref)`, `visible = fitz.Pixmap(base, mask)` (PyMuPDF's alpha-combining constructor) → `data = visible.tobytes("png")`, `mime = "image/png"`. Fail-soft: any exception (or a `Pixmap` with a colorspace PyMuPDF cannot combine) falls back to the raw `extract_image` bytes + the raw mime and logs a warning — the image is still indexed, exactly as today. **Ordering rule (owner-locked A4):** the raw-format gate runs FIRST — an image whose raw format is unsupported (`jpx` → `.bin`) is left exactly as today (raw bytes, raw mime, skipped downstream by the importer's ext gate); compositing happens only for images whose raw mime already maps through `_MIME_TO_EXT`. A `jpx` image with an smask therefore stays skipped.
- **Blank gate (task 03):** a NEW pure predicate in `app/rag/pdf.py`, `is_blank_image(samples: bytes, *, mean_min: int = 235, tolerance: int = 12, fraction: float = 0.98) -> bool` — pure (no PDF, no I/O — the `is_usable_pdf_text` / `pictorial_pages` house style, so it is unit-testable with hand-built sample buffers): blank when `len(samples)` is non-empty AND `mean(samples) >= mean_min` AND at least `fraction` of the samples lie within `±tolerance` of the mean. Applied in `extract_pdf_images` to the samples of the image that will actually be stored (the composited pixmap when a mask was applied, else the base pixmap): a blank image is **skipped** (not returned), with one INFO line per skipped image (`pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.0f uniform=%.3f`) and one summary line per PDF. Fail-soft: a pixmap that cannot be built skips the gate (the image is kept — never lose a real image to a decode problem).
- **The knob (task 04):** `Settings.pdf_image_skip_blank: bool = True` (`BOR_PDF_IMAGE_SKIP_BLANK`) — the kill switch, exactly the `pdf_pictorial` (L433) pattern: a `bool` with a `#:` docstring naming the env var, read by the importer and passed down, `False` restores the pre-phase behavior (every image kept, gate off). Dedupe and mask compositing are NOT knobed (they are correctness fixes). `.env.example` gains the documented entry next to the other PDF/image vars; `README.md` gains one line in the PDF/image settings section.
- **Counters (task 03):** two log-only `ImportSummary` fields — `pdf_images_deduped` and `pdf_images_blank_skipped` — appended to `ImportSummary.log()`'s line (the phase-133 stats log-only convention: the sync `detail` API shape is untouched). They count occurrences dropped per run across all PDFs.
- **ASSUMPTION A1 (owner-locked):** blank images are skipped entirely (no row, no copy, no vision call) rather than indexed-and-flagged.
- **ASSUMPTION A2 (owner-locked):** thresholds `mean ≥ 235`, `≥98%` of samples within `±12` — pinned by unit tests against the three real PDFs in `test_sources/` (podman underlays skipped; the horse photo and the handbook page art kept).
- **ASSUMPTION A3:** the dedupe changes the synthetic path for repeat occurrences, so image docs created pre-phase for a 2nd/3rd placement are pruned on the next sync and the first placement's doc is kept (idempotent from then on). No migration code.
- **ASSUMPTION A4 (owner-locked):** `jpx` (JPEG2000) images stay unsupported/skipped; compositing never newly admits them.
- **ASSUMPTION A5:** the gate lives in the parser (`extract_pdf_images`), not the importer, so every caller of the parser gets the fix and the parser stays the single place that decides what "an embedded image" is.
## Tasks
1. `01_dedupe_image_occurrences.md` — one `PdfImage` per distinct image (xref, else digest); duplicate-drop log + counter.
2. `02_composite_soft_masks.md` — `/SMask` composited to an alpha PNG (raw-format gate first, fail-soft to raw bytes).
3. `03_blank_image_gate.md` — the pure `is_blank_image()` predicate + the skip in the extractor + the two log-only counters.
4. `04_settings_knob.md` — `BOR_PDF_IMAGE_SKIP_BLANK` (default on) threaded parser → importer, `.env.example` + README.
5. `05_unit_tests.md` — PyMuPDF-built fixtures for dedupe/mask/blank + the real-PDF regression pins.
6. `06_e2e_pdf_image_extraction.md` — the dedicated Playwright suite (upload a PDF with a duplicate placement + a blank underlay + a real image → exactly one image doc, no blank docs).
## Testing & Quality
- Unit (`tests/unit/test_pdf.py` + a new `tests/unit/test_pdf_image_filter.py`): dedupe (N placements of one image → 1 result, first occurrence's page/index kept, counter), digest-level dedupe across xrefs, smask compositing (an RGBA image round-trips with alpha; the raw-format gate leaves `jpx`-style unsupported formats untouched; the fail-soft path returns raw bytes), `is_blank_image` as a pure function (uniform white → blank; noise → not blank; dark-uniform → NOT blank because the mean gate is about near-white underlays; empty samples → not blank), the knob off → every image kept.
- Real-file pins (skip-if-missing, the `HANDBOOK`/`HORSE_PDF` pattern already in `tests/unit/test_pdf.py` L1734–1737): `podman-systemd.pdf` → 15 unique images in, **0 image docs out** (all are decorative underlays); `pdf_horse_test.pdf` → its photo kept; `players_handbook.pdf` → its images stay skipped by the ext gate (unchanged, `jpx`).
- Integration: the importer's embedded-image pass indexes exactly the extractor's returned list (a blank/duplicate image never reaches `_index_one_embedded_image` — assert with a fake `Embedder` counting vision calls, the existing image-test idiom).
- Coverage: **>90%** on new/modified code (`app/rag/pdf.py`, the importer's pass, `app/config.py`).
- E2E: one dedicated Playwright file, run in isolation (`uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov`).
## Completion Criteria
- [ ] syncing a PDF like `podman-systemd.pdf` creates **no** blank white image documents and **no** duplicate image documents (verified by the E2E and the real-file unit pins)
- [ ] an image with an `/SMask` is stored as the composited image (alpha PNG), not the blank base; an unsupported raw format (`jpx`) is still skipped
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` restores the pre-phase behavior (every extracted image indexed) — pinned by a unit test
- [ ] `import: summary … pdf_images_deduped=N pdf_images_blank_skipped=M …` appears in the sync log; the sync `detail` API shape is unchanged
- [ ] no behavior change in completed phases (the phase-137 document-context block, the phase-125 markdown enrichment, the digest skip, the `images` toggle semantics, the phase-133 fallback, the phase-140 pictorial pass)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,29 @@
# Task 01 — one `PdfImage` per distinct image (duplicate placements dropped)
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
`extract_pdf_images()` returns one entry per DISTINCT image, so a PDF that places the same image 4× produces one document instead of four.
## Work
1. `app/rag/pdf.py` — `extract_pdf_images()`: keep the per-page/occurrence walk (it is what gives `page`/`index`), but skip anything already seen:
- maintain `seen_xrefs: set[int]` and `seen_digests: set[str]` (sha256 hex of the final `data`) outside the page loop;
- after `extract_image` succeeds and `data` is non-empty: `if xref in seen_xrefs: dropped += 1; continue`, then `digest = hashlib.sha256(data).hexdigest()`; `if digest in seen_digests: dropped += 1; continue`; record both before appending (the digest set catches byte-identical images stored under different xrefs);
- the FIRST occurrence wins: the appended `PdfImage` keeps that occurrence's `page` (1-based) and `index` (0-based on the page) — the importer's synthetic path `…#page=N#img=i.<ext>` therefore stays stable for the first placement (ASSUMPTION A3: repeat-placement docs created pre-phase are pruned on the next sync, no migration code);
- when `dropped`: one INFO line after the page loop — `pdf: dropped %d duplicate image occurrence(s) from %s (%d unique)` — and a DEBUG line per drop (`pdf: duplicate image xref=%d page=%d index=%d of %s — skipped`);
- `hashlib` is already imported in this module? If not, add `import hashlib` (check first — `app/rag/pdf.py` currently imports `logging`, `re`, `collections.abc`, `dataclasses`, `pathlib`, `typing`, `pymupdf`).
2. Update the module docstring's `extract_pdf_images` bullet: "one :class:`PdfImage` per DISTINCT embedded image (page order, first placement wins) — repeated placements and byte-identical copies are deduplicated (phase 151 task 01)".
3. `app/rag/importer.py` — `_index_pdf_embedded_images()` needs no logic change (it indexes exactly what the extractor returns), but its docstring's "each embedded image … in page order then 0-based index" sentence gains the dedupe note. Add the two `ImportSummary` counters' plumbing HERE only if task 03 has not landed yet — otherwise leave the counters to task 03 (avoid a duplicate field).
4. ASSUMPTION: dedupe is per-PDF (the sets are local to one `extract_pdf_images` call), never across PDFs — two PDFs sharing a logo still each get their own image doc (the synthetic path is per-PDF, so cross-PDF dedupe would break the upsert key).
## Testing & Quality
- Unit (`tests/unit/test_pdf.py`, the PyMuPDF-built-fixture convention): a PDF with the SAME image inserted on 3 pages (or 3× on one page) → `len(extract_pdf_images(...)) == 1` and it carries the first occurrence's `page`/`index`; a PDF with two genuinely different images → 2; two different xrefs holding byte-identical bytes → 1 (digest path); the INFO line emitted (caplog) only when a duplicate was dropped.
- Real-file pin (skip-if-missing): `test_sources/podman-systemd.pdf` → the extractor returns **15** entries (46 placements pre-phase).
- Coverage: **>90%** on the new branches (both dedupe paths + the no-duplicate path).
## Completion Criteria
- [ ] `extract_pdf_images(Path("test_sources/podman-systemd.pdf"))` returns 15 entries (was 46)
- [ ] the first occurrence's page/index are preserved; the digest path is covered by a unit test
- [ ] the duplicate-drop INFO line appears once per PDF with duplicates
- [ ] `uv run pytest tests/unit/test_pdf.py -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,37 @@
# Task 02 — apply the `/SMask`: store the image a viewer would show
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
An image stored in the PDF with a soft mask is extracted as the composited (masked) image, not as the blank base sample stream `extract_image()` returns today.
## Work
1. `app/rag/pdf.py` — `extract_pdf_images()`, inside the per-image block, AFTER the raw-format decision and BEFORE appending:
- read the mask reference: `smask = int(extracted.get("smask") or 0)` (the `extract_image` dict reports it; `get_image_info`'s `has-mask` is the cross-check, not the source of truth);
- **raw-format gate FIRST (owner-locked A4):** compute `mime` from the raw format exactly as today (`_mime_from_image_info({**info, "ext": extracted.get("ext") or ""})`). If that mime is `application/octet-stream` (i.e. a format the importer's `_MIME_TO_EXT` gate will reject — `jpx`/JPEG2000, the 586 images in `players_handbook.pdf`), DO NOT composite: keep the raw bytes + raw mime, so the behavior is byte-identical to pre-phase for unsupported formats (compositing must never newly admit them);
- otherwise, when `smask` is non-zero: build the visible image
```python
base = fitz.Pixmap(doc, xref)
mask = fitz.Pixmap(doc, smask)
visible = fitz.Pixmap(base, mask) # base + alpha-from-mask
data = visible.tobytes("png")
mime = "image/png"
```
wrapped in `try/except Exception` → on failure log a warning (`pdf: could not composite smask=%d for image page=%d index=%d of %s — %s; using raw bytes`) and fall back to the raw `extracted["image"]` bytes + the raw mime (fail-soft: the image is still indexed, exactly as pre-phase);
- keep `width`/`height` from `extracted` (unchanged — the composited pixmap has the same dimensions);
- a DEBUG line when a mask was applied (`pdf: composited smask=%d for image page=%d index=%d of %s`).
2. `app/rag/pdf.py` module docstring: extend the `extract_pdf_images` bullet — masked images are composited to an alpha PNG (phase 151 task 02), unsupported raw formats are left untouched so the importer's extension gate still decides.
3. `app/rag/importer.py`: no code change — `_index_one_embedded_image()` already derives the copy extension from `img.mime` via `_MIME_TO_EXT`, and `image/png` → `.png` is in `settings.image_extension_set` by default. Verify the digest-skip consequence in a comment: a masked image's bytes change, so pre-phase rows for masked images re-describe ONCE on the first post-phase sync (the sha256 digest differs) — expected, self-limiting, no migration.
4. ASSUMPTION: `fitz.Pixmap(base, mask)` is the correct combinator (verified against `podman-systemd.pdf` xrefs 202/744/1633 — it produces an alpha PNG of the same dimensions); a CMYK/Indexed base that PyMuPDF refuses to combine takes the fail-soft path.
## Testing & Quality
- Unit (`tests/unit/test_pdf.py`): build a PDF with an RGBA PNG inserted (`page.insert_image(rect, stream=png_bytes)` — PyMuPDF writes the `/SMask`) and assert (a) `extract_image`'s raw bytes ≠ the extractor's `data`, (b) the extractor's `mime == "image/png"`, (c) re-opening the extractor's bytes as a `fitz.Pixmap` reports `alpha == 1` (the mask survived), (d) a PDF whose image has no mask is returned byte-identical to pre-phase (no regression for the common case); the fail-soft path via a monkeypatched `fitz.Pixmap` that raises → raw bytes returned + the warning logged (caplog).
- Real-file pin (skip-if-missing): every image extracted from `test_sources/podman-systemd.pdf` carries an smask pre-phase; post-phase the extractor's output for it is either composited PNG or (blank-gated away by task 03) absent — assert the composited bytes differ from the raw base for at least one xref when the gate is off.
- Coverage: **>90%** on the new branches (mask applied / mask failed / unsupported format untouched / no mask).
## Completion Criteria
- [ ] a masked image extracted through `extract_pdf_images` yields alpha-PNG bytes (`mime="image/png"`), not the blank base stream
- [ ] an unsupported raw format (`jpx`) is untouched — raw bytes + raw mime, still skipped downstream
- [ ] a compositing failure falls back to the raw bytes with a warning (no image lost, no exception raised)
- [ ] `uv run pytest tests/unit/test_pdf.py -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,42 @@
# Task 03 — the blank/decorative gate: never index a near-white underlay
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
A deterministic, pure predicate decides "this image carries no visible content" and the extractor drops those images — no row, no `image_dir` copy, no vision call.
## Work
1. `app/rag/pdf.py` — add the pure predicate (the `is_usable_pdf_text` / `pictorial_pages` house style: no PDF, no I/O, no LLM):
```python
BLANK_IMAGE_MEAN = 235 #: near-white floor
BLANK_IMAGE_TOLERANCE = 12 #: "same as the mean" band
BLANK_IMAGE_FRACTION = 0.98 #: fraction of samples that must sit in the band
def is_blank_image(samples: bytes, *, mean_min: int = BLANK_IMAGE_MEAN,
tolerance: int = BLANK_IMAGE_TOLERANCE,
fraction: float = BLANK_IMAGE_FRACTION) -> bool:
...
```
Blank when `samples` is non-empty AND `sum(samples)/len(samples) >= mean_min` AND `sum(1 for s in samples if abs(s - mean) <= tolerance) / len(samples) >= fraction`. Empty samples → `False` (never blank a thing we cannot measure). Document the measured basis in the docstring (podman underlays: mean 238, uniq 8, uniform(±8) 0.95–0.995 → blank; the horse photo: mean 123, uniq 256, uniform 0.14 → kept; handbook page art: mean 24–221, uniq ≥188, uniform 0.10–0.59 → kept). Note explicitly that a DARK uniform block is NOT blank by this rule (the gate targets near-white page underlays — a solid dark block is unusual and cheaper to keep than to explain away).
2. `app/rag/pdf.py` — `extract_pdf_images()`: after the dedupe + mask steps, decide blank-ness on the samples of the image that would actually be stored:
- build `gate_pixmap` = the composited pixmap when a mask was applied, else `fitz.Pixmap(doc, xref)`; when `gate_pixmap.alpha`, drop the alpha first (`fitz.Pixmap(gate_pixmap, 0)`) so the mean is over COLOR samples only (an alpha channel would drag the mean and produce false blanks);
- `if settings-gate and is_blank_image(gate_pixmap.samples): blank_skipped += 1; continue` (the gate is skipped entirely when the caller passes `skip_blank=False` — task 04's knob);
- one INFO line per skipped image: `pdf: skipped blank/decorative image page=%d index=%d of %s — mean=%.1f uniform=%.3f`, and one INFO per PDF when `blank_skipped` (`pdf: skipped %d blank/decorative image(s) in %s`);
- fail-soft: if the pixmap cannot be built for the gate, DO NOT skip the image (keep it, DEBUG why) — a decode problem must never delete a real image;
- the extractor signature gains `skip_blank: bool = True` (keyword, default on) so callers/tests can turn it off without settings; the dedupe and mask steps are NOT knobed.
3. `app/rag/importer.py` — `ImportSummary` gains two log-only counters (the phase-133 stats log-only convention: the sync `detail` API shape is untouched), each with a `#:` docstring citing this phase: `pdf_images_deduped: int = 0` and `pdf_images_blank_skipped: int = 0`, appended to `ImportSummary.log()`'s format string (`… pdf_images_deduped=%d pdf_images_blank_skipped=%d formats=%s`). The extractor returns the counts alongside the images so the importer can record them — the minimal clean shape is a small result object or a 2-tuple; prefer keeping `extract_pdf_images` returning `list[PdfImage]` for its other callers and add the counters via an optional out-param-free route: give the function a sibling `extract_pdf_images_with_stats(pdf_path, *, skip_blank=True) -> tuple[list[PdfImage], PdfImageStats]` (a `@dataclass(frozen=True) PdfImageStats` with `duplicates`, `blank_skipped`, `total_found`) and make `extract_pdf_images` a thin wrapper returning `stats[0]`. `_index_pdf_embedded_images()` calls the `_with_stats` variant and adds to `summary.pdf_images_deduped` / `summary.pdf_images_blank_skipped`.
4. ASSUMPTION: the counters are log-only (no API change) — the owner reads them in the sync log; the UI's sync result line is untouched.
## Testing & Quality
- Unit (`tests/unit/test_pdf_image_filter.py`, new file): `is_blank_image` as a pure function — all-250 bytes → blank; random noise → not blank; all-0 (dark) → NOT blank; empty → not blank; a 97%-uniform-with-a-few-dark-pixels buffer → not blank (just under the fraction); the boundary values pinned exactly at `mean_min` and `fraction`; the extractor-level behavior with PyMuPDF-built PDFs — a near-white 200×200 image inserted → 0 results + the INFO line + `stats.blank_skipped == 1`; a colorful noise image → 1 result; `skip_blank=False` → both returned.
- Integration: the importer's pass with a fake `Embedder` (the existing image-test idiom) — a PDF with one blank + one real image produces exactly ONE image document and ONE vision call; `import: summary … pdf_images_blank_skipped=1` in the log (caplog).
- Real-file pin (skip-if-missing): `test_sources/podman-systemd.pdf` → `extract_pdf_images` returns `[]` (all 15 unique images are decorative underlays) with `stats.duplicates == 31` and `stats.blank_skipped == 15`; `test_sources/pdf_horse_test.pdf` → its photo is returned.
- Coverage: **>90%** on the new code.
## Completion Criteria
- [ ] `extract_pdf_images(Path("test_sources/podman-systemd.pdf"))` returns no images (pre-phase: 46 blank-white docs)
- [ ] `extract_pdf_images(..., skip_blank=False)` still returns them (the kill switch at the function level)
- [ ] `import: summary … pdf_images_deduped=… pdf_images_blank_skipped=…` appears in the sync log; the `/api/sync/status` `detail` shape is unchanged
- [ ] a dark-uniform image and a real photo are both kept (unit-pinned)
- [ ] `uv run pytest tests/unit/test_pdf_image_filter.py -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,29 @@
# Task 04 — `BOR_PDF_IMAGE_SKIP_BLANK`: the kill switch, wired end to end
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
The blank gate is a documented setting, default ON, with a supported way to turn it off — the `pdf_pictorial` pattern.
## Work
1. `app/config.py` — add to `Settings`, next to `pdf_pictorial` / `pdf_pictorial_min_chars` (L433–443):
```python
#: Skip blank/decorative PDF-embedded images (phase 151, task 04,
#: ``BOR_PDF_IMAGE_SKIP_BLANK``; ``0``/``false`` = off) …
pdf_image_skip_blank: bool = True
```
The docstring names the env var, states the default, and says what turning it off means (every extracted image is indexed again — the pre-phase behavior; dedupe and soft-mask compositing are unaffected, they are correctness fixes). No validator needed (a plain bool like `pdf_pictorial` / `images`).
2. `app/rag/importer.py` — `_index_pdf_embedded_images()`: pass the setting into the extractor call (`extract_pdf_images_with_stats(full_path, skip_blank=settings.pdf_image_skip_blank)`), so the knob is read in exactly one place. Update the pass's docstring bullet list with the gate + the knob.
3. `.env.example` — add the entry in the PDF/image block (next to `BOR_PDF_PICTORIAL` / `BOR_IMAGES`), commented in the file's existing voice: what it does, the default, and when to set it to `false` (a KB of scanned/decorative pages where a near-white image still matters).
4. `README.md` — one line in the PDF/image settings list (the settings table/section the phase-126/140 entries live in): `BOR_PDF_IMAGE_SKIP_BLANK` (default `true`) — skip blank/decorative images embedded in PDFs.
5. ASSUMPTION: no admin-UI toggle (the phase-91 Theme Tab precedent shows knobs live in `.env`; the owner asked for the bug fixed, not a settings page).
## Testing & Quality
- Unit: `Settings` default is `True`; `BOR_PDF_IMAGE_SKIP_BLANK=false` (env override, the existing config-test idiom) yields `False`; an integration test with the fake `Embedder` proves the off path indexes a blank image again (the inverse of task 03's assertion) — the kill switch is proven, not assumed.
- Coverage: **>90%** on the touched lines.
## Completion Criteria
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` restores indexing of blank PDF-embedded images (test-proven)
- [ ] the setting is documented in `.env.example` and `README.md`
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,31 @@
# Task 05 — unit + integration tests for dedupe, masks, and the blank gate
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
Every new branch of the extractor is pinned without a browser: PyMuPDF-built fixtures for the synthetic cases, the three real PDFs in `test_sources/` for the regression proof.
## Work
1. `tests/unit/test_pdf_image_filter.py` (new file; the `tests/unit/test_pdf.py` conventions — PyMuPDF-built temp PDFs, no binary fixtures committed, `tmp_path`):
- **dedupe:** one image inserted on 3 pages → 1 result with the FIRST page/index; two distinct images → 2; byte-identical bytes under two xrefs → 1 (digest path); `stats.duplicates` counts; the INFO line via `caplog`;
- **soft mask:** an RGBA PNG inserted (`page.insert_image(rect, stream=…)`) → the extractor's `mime == "image/png"`, its bytes ≠ `doc.extract_image(xref)["image"]`, and `fitz.Pixmap(stream=extracted_bytes).alpha == 1`; an unmasked image → byte-identical to pre-phase; a monkeypatched `fitz.Pixmap` that raises → raw bytes returned + the warning (fail-soft);
- **blank gate:** `is_blank_image` as a pure function (uniform white → True; noise → False; uniform dark → False; empty → False; the exact `mean_min`/`fraction` boundaries); extractor-level — a near-white 200×200 image → 0 results + `stats.blank_skipped == 1` + the INFO line; a colorful noise image → 1 result; `skip_blank=False` → both returned;
- **unsupported format untouched:** an image whose raw ext is unsupported (build one by monkeypatching `extract_image` to report `ext="jpx"` + a non-zero `smask`) → the extractor returns raw bytes + the unmapped mime (no compositing) — the owner-locked A4 pin.
2. `tests/unit/test_pdf.py` — extend the existing extractor tests (they already cover `extract_pdf_images` and the fail-soft contract): the dedupe/mask steps must not break the "unreadable file → []", "no xref → skipped", "empty data → skipped" pins; add the real-file pins with the file's existing skip-if-missing idiom (see `HANDBOOK` / `HORSE_PDF`, L1734–1737):
- `test_sources/podman-systemd.pdf` → `extract_pdf_images(...) == []`, `stats.duplicates == 31`, `stats.blank_skipped == 15` (the owner's exact file — the bug's tombstone test);
- `test_sources/pdf_horse_test.pdf` → exactly 1 image, not blank;
- `test_sources/players_handbook.pdf` → images are returned by the parser but their raw mime is unmapped (`jpx`) → the importer's ext gate still skips them (assert the mime mapping, not the importer).
Each guarded by `pytest.mark.skipif(not path.exists(), …)` — `test_sources/*.pdf` are untracked fixtures.
3. Integration (`tests/unit/test_image_documents.py` or a new `tests/integration/test_pdf_image_pass.py`, whichever matches the existing image-pass tests' home): with `tests/fakes.py::FakeEmbedder` (the `_VisionMockLLM` idiom — a vision-capable fake that counts `describe_image` calls), a PDF containing one blank underlay + one real image + a duplicate placement indexes EXACTLY ONE image document and makes exactly ONE vision describe call; `summary.pdf_images_deduped` / `summary.pdf_images_blank_skipped` carry the counts; the `import: summary …` line includes them (caplog).
4. Run the whole suite after each addition: `uv run pytest -q`, then `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/rag/pdf.py`, `app/rag/importer.py`, `app/config.py`).
## Testing & Quality
- Coverage: **>90%** on the new/modified code paths (dedupe ×2, mask applied/failed/unsupported, blank skip/keep/knob-off, the stats plumbing).
- No network, no real LLM (the fakes only); the real-PDF pins are pure CPU (PyMuPDF) and must stay under a few seconds each.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_pdf_image_filter.py tests/unit/test_pdf.py -v` green
- [ ] the podman-systemd pin asserts 0 images out of 46 placements (the reported bug, pinned)
- [ ] the `skip_blank=False` and unsupported-format (A4) paths are both covered
- [ ] `uv run pytest -q` fully green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,36 @@
# Task 06 — the dedicated E2E: a PDF sync produces no blank, no duplicate image docs
**Phase:** `151_pdf_image_extraction` · **Source:** `TODO.md:4` — "For the same podman pdf document, it extracted a series of blank white images for some reason. There don't appear to be any blank images in the document, is this a bug?"
**Story:** n/a
## Objective
Through the real product path (upload a PDF → press Sync → inspect the KB), a PDF containing a decorative underlay, a duplicate placement, and one real image yields exactly one image document — the owner's complaint, proven fixed in the browser.
## Work
1. `tests/e2e/test_pdf_image_extraction.py` (new; the `tests/e2e/test_pdf_support.py` / `test_pdf_image_context.py` per-suite-app pattern — a module-scoped app with `BOR_IMAGES=true` + the `pdf` import token, the shared E2E Postgres, the autouse truncate, the mock LLM only):
- **fixture PDF built in the suite with PyMuPDF** (the house convention — no binary committed): 2 text pages, page 1 carrying (a) a near-white 200×200 "decorative underlay" image, (b) the SAME real image inserted twice (the duplicate placement), (c) one genuinely distinct colorful image; page 2 carrying one more placement of the real image (a cross-page duplicate — the podman case);
- **seed** it through the real admin upload flow + the RAG page's **Sync** button (the same user path the owner used), then wait for the sync to settle on the sync result surface (post-phase-150: `#sync-progress-text`);
- **assertions:**
* the catalog contains exactly ONE image document for the PDF (the distinct real image, first placement) — count the `is_image` docs whose path starts with the PDF's synthetic path;
* NO document's stored image bytes are a blank white image (fetch the served copy / read the row's `image_path` and assert the file's decoded pixels are not near-uniform — the same predicate the app uses, imported from `app.rag.pdf.is_blank_image`, so the test cannot drift from the product);
* the synthetic path of the kept doc is the FIRST placement's (`…#page=1#img=<i>.png`) — the dedupe's stability contract;
* the mock's vision call count for image description is 1 (the duplicate and the blank never reached the model — the cost the owner paid in "several minutes");
* a follow-up re-sync (nothing changed) reports `unchanged` and creates nothing new (idempotence after the fix).
- **negative control:** a second module app (or a second seed source) with `BOR_PDF_IMAGE_SKIP_BLANK=false` — the same PDF now yields the blank doc too (the knob is proven live in the product, not just in unit tests). If a second app is too heavy for the file's structure, drive the knob through the same env-fallback mechanism the other suites use for per-suite settings.
2. Run it in isolation until green: `uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` (DB up: `podman compose up -d db`).
3. Regression run: the pre-existing PDF/image suites must stay green unedited (they pin the behaviors this phase must not disturb): `tests/e2e/test_pdf_support.py`, `test_pdf_image_context.py`, `test_pictorial_pdf.py`, `test_image_documents.py`, `test_image_anchor_context.py`, `test_answer_images.py`.
4. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit with `--no-gpg-sign`, e.g.
`fix(rag): dedupe PDF images, apply soft masks, skip blank/decorative images`
(moves `.agents/phases/todo/151_pdf_image_extraction/` to `complete/`).
## Testing & Quality
- The dedicated E2E green IN ISOLATION (the phase gate) and the six pre-existing PDF/image E2Es green.
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- Mock-only suite (no real vision calls), deterministic (no sleeps as barriers — wait on the sync result surface).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` green in isolation
- [ ] exactly one image document per distinct real image; zero blank-white documents; one vision call
- [ ] `BOR_PDF_IMAGE_SKIP_BLANK=false` demonstrably restores the old behavior in the product
- [ ] the six pre-existing PDF/image E2Es green unedited; full suite green; coverage >90%
- [ ] the phase directory moved to `.agents/phases/complete/` and committed with `--no-gpg-sign`
@@ -0,0 +1,48 @@
# Phase 152 — the sync label keeps moving while images are described and embedded
**Source:** `TODO.md` L3 — "The sync status on the RAG page gets \"stuck\" when it's embedding images. For example, when syncing test_sources/podman-systemd.pdf the status was stuck on `Syncing… podman-systemd.unit — Podman documentation/podman-systemd.unit — Podman documentation.pdf (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a (TODO-derived phase, Protocol B append).
**Context:** the sync status machine `app/api/sync.py` — `SyncStatus` (L276–300: `state`, `current_file`/`files_done`/`files_total` (phase 64), `phase`/`current_summary`/`summaries_done`/`summaries_total` (phase 98), `current_chunk`/`chunks_total` (phase 129)), the `GET /api/sync/status` payload (L330–360), and `_run_sync`'s paired progress closures `_hook` (L553 — names the file) and `_chunk_hook` (L563 — the in-flight file's embed-batch position, fed by `LLMClient.embed`'s `on_progress`). The importer threads the chunk hook as `import_sources(on_chunk_progress=…)` → `_index_file(on_chunk=…)` → `_index_pdf_file(on_chunk=…)` → `_replace_content_chunks(on_chunk=…)`. The importer's own docstring states the gap this phase closes (L684–690): *"Only the content-chunk embeds report … the PDF's embedded-image docs (separate documents, not the walked file) do not report either."*
**Reproduced:** the owner's `podman-systemd.pdf` is the LAST file of a 2,540-file run. Its text chunks finish (the label's last write: `· chunk 125/126`), then the PDF's vision spans run — `_index_pdf_embedded_images` (46 image occurrences pre-phase-151, each a `describe_image` chat call + an embed + a summary), `_pictorial_pdf_pass` (L1715 — one render + one describe call per sparse page), `_fallback_page_screenshots` (L1635 — one describe call per page of a garbage-text PDF). **None of them reports progress**, so the label sits frozen on the last chunk position for minutes while the run is very much alive. The phase-149 box made the freeze plainly visible (it is now the big surface the user reads).
**Locked (owner decision, chat):** the fix is a real progress signal for the vision spans (not a spinner/heartbeat that hides the stall), reported as a per-image position alongside the existing chunk position.
## Objective
During a PDF's image spans the status reports `image n/m`, the UI renders it, and the label advances once per vision call — a long sync is visibly progressing, and a genuinely stalled run is distinguishable from a busy one.
## Dependencies
- `151_pdf_image_extraction` (todo) — its dedupe + blank gate shrink the span this phase reports on, and its `extract_pdf_images_with_stats` is the call that establishes the span's `total`. Report order matters: the totals this phase reports are post-dedupe/post-gate counts.
- `150_drop_sync_result` (todo) — the label surface is `#sync-progress` (the phase-149 box is the only place the running label renders; the button is the spinner).
## Design (shared by all tasks — the executor reads this, not the chat)
- **The hook (task 02):** `import_sources(..., on_image_progress: Callable[[int, int], None] | None = None)`, threaded exactly like `on_chunk_progress`: `import_sources` → `_index_file(on_image=…)` → `_index_pdf_file(on_image=…)` → the three vision spans (`_index_pdf_embedded_images`, `_pictorial_pdf_pass`, `_fallback_page_screenshots`). Contract (the phase-129 pairing rule, extended): the hook is called **immediately before each vision describe call** with `(position_1based, total)` where `total` is that span's count — the extracted-image count for the embedded pass (post-dedupe, post-blank-gate: the work actually about to happen), the candidate-page count for the pictorial pass, the rendered-page count for the screenshot fallback. The three spans of ONE file are reported as ONE continuous counter: the pass that runs second starts where the first ended (`base = images_done_so_far`), so the label climbs `1/N … N/N` across the file's whole vision span instead of restarting. `None` (every existing caller — the CLI, the chat path) is a zero-cost guarded no-op and the run is byte-identical (the phase-129 rule).
- **The status fields (task 01):** `SyncStatus` gains `images_done: int = 0` / `images_total: int = 0`, reported by `GET /api/sync/status` as `images_done` / `images_total`. Clear/reset rules mirror the chunk position (a sub-position of a file): cleared at the `phase = "overview"` / `phase = "summaries"` transitions and in every terminal (success/failed/cancelled), and **reset to 0/0 by the per-file `_hook`** — the pairing rule is "a `progress` call names the file and clears the image position; every image position until the next `progress` call belongs to it". (The chunk position does not need this because every file embeds chunks; a file with no images must not inherit the previous PDF's position.)
- **The label (task 03):** `sources.js` `fmtSyncPhaseLabel` appends `· image n/m` when `status.images_total > 0` (the phase-129 chunk-part language: a position reads as data, mono in the box). `setSyncLabel` gains a fifth parameter `imageText = ""` appended to `full` (after the chunk part) — it rides the title, the box, and the running `aria-label`. **No new button span** is added: while running the label parts are hidden (phase 149 makes the button the spinner), so the parts DOM stays as-is and the position lives only in the composed `full` string. The chunk part keeps its own slot (the two positions are distinct and both may show: `… (2540/2540) · chunk 125/126 · image 3/15`).
- **ASSUMPTION A1 (owner-locked):** a real per-image position, not an elapsed-time heartbeat.
- **ASSUMPTION A2:** counts only, not the image identity — the file label already names the PDF and the synthetic `…#page=N#img=i.png` path is noise in a label.
- **ASSUMPTION A3:** the three PDF vision spans share ONE counter per file (continuous, not three restarts) — the user cares that the file is progressing, not which span it is in.
- **ASSUMPTION A4:** standalone image files (one image per file) get no sub-position — `1/1` per file would be noise; the file count already covers them.
- **ASSUMPTION A5:** no new `phase` value — the image spans are inside `phase: "import"` (the phase machine stays as phase 98 defined it).
## Tasks
1. `01_status_image_fields.md` — `SyncStatus.images_done/images_total`, the reset/clear rules, the `/api/sync/status` payload + docstring.
2. `02_importer_image_hook.md` — `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter.
3. `03_label_and_wiring.md` — the `_image_hook` closure in `_run_sync`, the `sources.js` label part + title/aria, the phase-129-style comment updates.
4. `04_unit_tests.md` — the status machine pins (idle 0/0, mid-run position, the per-file reset, every terminal clears) + the importer hook pairing.
5. `05_e2e_image_progress.md` — the dedicated Playwright suite: the label advances through a PDF's image span (slow-LLM proxy), never frozen.
## Testing & Quality
- Unit: `tests/unit/test_sync_button.py`'s full-status-dict pin gains the two keys (idle `0/0`); the terminal-clears and per-file-reset rules pinned through the runner's own closures (the phase-64/129 idiom); the importer hook pinned with `tests/fakes.py::FakeEmbedder` — a PDF with N images calls the hook N times with `(1,N)…(N,N)`, the three spans continue one counter, `None` costs nothing (the hook-less run is byte-identical).
- Integration: a sync run over a fixture PDF reports a monotonically increasing `images_done` across polls (no backwards jumps, never above `images_total`).
- Coverage: **>90%** on new/modified code (`app/api/sync.py`, `app/rag/importer.py`, the JS is pinned by the source-level suites + E2E).
- E2E: `uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov` in isolation (the `tests/e2e/slow_llm.py` delay proxy — every describe call outlives the 2 s poll, so each position is observed deterministically).
## Completion Criteria
- [ ] during a PDF's image span the status reports `images_done`/`images_total` climbing 1→N and the UI shows `· image n/m` in the box (and the button `title`/`aria-label`)
- [ ] the label is never frozen for more than one vision call's duration on the fixture PDF (the E2E asserts at least 3 distinct positions observed)
- [ ] the image position clears at the overview/summaries transitions and in every terminal; a file with no images never shows a stale position
- [ ] `GET /api/sync/status` is additive only — every pre-existing key unchanged (the full-dict pin)
- [ ] a hook-less caller (the CLI import path) is byte-identical to pre-phase
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,34 @@
# Task 01 — `SyncStatus` gains `images_done` / `images_total`
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a
## Objective
The sync status carries the in-flight file's vision-image position, with the same clear/reset discipline the chunk position already has.
## Work
1. `app/api/sync.py` — `SyncStatus` (L276–300): add
```python
# Phase 152 (task 01): the in-flight file's VISION-image position —
# the 1-based index of the image about to be described and the
# file's total images for this run (the importer's
# on_image_progress hook; the three PDF vision spans share ONE
# continuous counter per file). 0/0 before the first image, during
# the prelude/overview/summaries phases, and in terminal states.
images_done: int = 0
images_total: int = 0
```
and extend the dataclass docstring with a phase-152 paragraph in the file's established voice (it is a SUB-position of a file, like the chunk position: the keep-final-counts convention covers FILES only, so this clears with the file).
2. Clear/reset rules — exactly the points `current_chunk`/`chunks_total` clear today (grep `_status.current_chunk = 0` — the `phase = "overview"` transition ~L597, the `phase = "summaries"` transitions ~L650/671, the terminal paths ~L713/717/752/755/770/774, and the run start ~L462): set the two new fields to `0` at the SAME sites. In addition, the per-file `_hook` closure (L553) resets them to `0/0` — the pairing rule is "a `progress` call names the file and clears the image position; every image position until the next `progress` call belongs to it" (unlike the chunk position, a file may have NO images at all, and it must not inherit the previous PDF's position).
3. `GET /api/sync/status` (the payload dict, L330–360): add `"images_done"` / `"images_total"` beside `current_chunk`/`chunks_total`, with the same style of inline comment; extend the endpoint docstring with the phase-152 sentence (additive only — every pre-existing key keeps its name, type, and meaning).
4. Module docstring: add the phase-152 paragraph after the phase-129 chunk-position paragraph (the file documents each phase's contribution in order).
5. ASSUMPTION A5: no new `phase` value — the image spans live inside `phase: "import"`.
## Testing & Quality
- Unit: the full idle-dict pin in `tests/unit/test_sync_button.py` gains the two keys (`0`/`0`); a terminal-clears test and a per-file-reset test (drive the runner's closures directly, the phase-64/129 idiom).
- Coverage: **>90%** on the new lines.
## Completion Criteria
- [ ] `GET /api/sync/status` returns `images_done` / `images_total` (idle `0`/`0`) and every pre-existing key is unchanged
- [ ] the fields clear at the overview/summaries transitions and in success/failed/cancelled, and reset per file
- [ ] `uv run pytest tests/unit/test_sync_button.py -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,32 @@
# Task 02 — the importer's `on_image_progress` hook across the PDF vision spans
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a
## Objective
Every vision describe call in a file's import reports `(position, total)` before it happens, so the caller can show the span moving.
## Work
1. `app/rag/importer.py` — thread the callback exactly like `on_chunk_progress` (the phase-129 route):
- `import_sources(..., on_image_progress: Callable[[int, int], None] | None = None)` (L636–650) — document it in the docstring next to the `on_chunk_progress` paragraph, and CORRECT that paragraph's closing sentence ("the PDF's embedded-image docs … do not report either") — it no longer holds;
- `_index_file(..., on_image: Callable[[int, int], None] | None = None)` (L1050) → `_index_pdf_file(..., on_image=…)` (L1372);
- the three spans, each called with the file's continuous counter (ASSUMPTION A3 — one counter per file, spans continue each other):
* `_index_pdf_embedded_images(..., on_image=…)` (L1924): `total = len(images)` from `extract_pdf_images_with_stats` (phase 151 — post-dedupe, post-blank-gate: the work actually about to happen); call `on_image(base + i + 1, base + total)` **immediately before** each `_describe_or_skip` (so the label moves at the START of each slow call, not after it), where `base` is the count already reported for this file;
* `_pictorial_pdf_pass(..., on_image=…)` (L1715): `total = len(candidate_pages)` (the deterministic sparse-page gate's list, already computed before the loop); same call site rule;
* `_fallback_page_screenshots(..., on_image=…)` (L1635): `total = len(rendered pages)`; same rule;
- the counter handoff: `_index_pdf_file` keeps a small local `reported = 0` and passes it down / adds each span's returned count (each span returns the number of images it reported — reuse its existing return shape or return the count; the executor picks the minimal clean shape). A span that is skipped (toggle off, no candidates, usable text) reports nothing and does not advance the counter;
- `None` stays a zero-cost guarded no-op: every call site is `if on_image is not None:` (the phase-129 rule — a hook-less run is byte-identical, so the CLI and chat paths are untouched).
2. Update the module docstring's progress paragraph (L287–291) to name the image hook, and the `_index_pdf_embedded_images` docstring's bullet list.
3. ASSUMPTION A4: standalone image files (`_index_image_file`, L2989) get NO sub-position — one image per file, `1/1` would be noise; the file count already covers them.
4. ASSUMPTION A2: counts only — the hook carries no image identity (the file label already names the PDF).
## Testing & Quality
- Unit (`tests/unit/test_importer.py` / the image-pass tests' `tests/fakes.py::FakeEmbedder` idiom): a fixture PDF with N embedded images → the hook fires N times with `(1,N)…(N,N)`; a PDF with a pictorial span + an embedded span → ONE continuous counter (no restart, never above the combined total); a text-only PDF → the hook never fires; `on_image_progress=None` → the import result is byte-identical to pre-phase (the regression pin).
- The hook never raises into the pipeline: a hook that raises is a caller bug (the phase-64 rule — "the hooks in this repo only assign fields", no try/except), so do NOT add defensive handling; keep the closure assignment-only.
- Coverage: **>90%** on the new branches (each span, the skip paths, the `None` path).
## Completion Criteria
- [ ] the hook fires once per vision describe call, 1-based, with the span's total, before the call
- [ ] the three PDF spans share one continuous per-file counter
- [ ] a hook-less import is byte-identical to pre-phase
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,35 @@
# Task 03 — wire the hook to the status and render `· image n/m` in the label
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a
## Objective
The importer's image hook lands on the status object, and the Sources page renders the position — the label advances once per vision call.
## Work
1. `app/api/sync.py` — `_run_sync`: add the third paired closure beside `_hook` (L553) and `_chunk_hook` (L563):
```python
def _image_hook(done: int, total: int) -> None:
_status.images_done = done
_status.images_total = total
```
pass it into the import call (L580): `import_sources(..., on_image_progress=_image_hook)`; comment it in the file's established voice (the three hooks are paired: `progress` names the file and resets the image position, `on_chunk_progress` reports the embed-batch position, `on_image_progress` reports the vision-image position).
2. `frontend/assets/sources.js`:
- `fmtSyncPhaseLabel(status)` (L896–919): the import branch gains `image` — `status.images_total > 0 ? \`· image ${status.images_done}/${status.images_total}\` : ""` — and the `overview` / `summaries` branches return `image: ""` (the position belongs to the import phase only);
- `enterSyncRunningState` (L964): destructure `{ main, count, chunk = "", image = "" }` and pass the new argument to the writer;
- `setSyncLabel(mainText, countText = "", chunkText = "", running = false, imageText = "")` (L809, post-phase-150): the composed `full` string becomes `main + (count ? " " + count : "") + (chunk ? " " + chunk : "") + (image ? " " + image : "")` — so the box, the button `title`, and the running `aria-label` all carry `Syncing… <file> (2540/2540) · chunk 125/126 · image 3/15` (the phase-149 box is the surface that shows it; the button's label parts stay as they are — no new span, the running button is the spinner);
- update the function's docstring + the phase-129/149 comment blocks that enumerate the label parts (the file cites the "· chunk n/m" contract in ~10 comments — name the image part beside it, do not restate the whole contract).
3. No CSS change: the box already wraps mono text with `overflow-wrap: anywhere`; the extra part is just more text.
4. ASSUMPTION A2/A3: counts only, one continuous per-file counter (the importer owns the continuity — the UI just renders what it is told).
5. ASSUMPTION: no new button label span — while running the parts are hidden (phase 149), so the position rides the composed string only.
## Testing & Quality
- Unit: the source-level pins in `tests/unit/test_sync_button.py` / `test_frontend_sync_upload.py` gain the image part (the `fmtSyncPhaseLabel` contract: `images_total > 0` → the part present, `0` → absent; the `overview`/`summaries` branches never carry it; the composed `full` string pinned byte-for-byte for a status with both positions).
- Integration: a mocked sync run over a fixture PDF shows `images_done` increasing across two polls of `GET /api/sync/status` (the phase-129 mid-run pin idiom).
- Coverage: **>90%** on the `app/` change.
## Completion Criteria
- [ ] while a PDF's images are described, `GET /api/sync/status` reports the climbing position and `#sync-progress-text` shows `· image n/m`
- [ ] the chunk part and the image part can show together and each disappears independently
- [ ] the prelude / overview / summaries labels are byte-identical to pre-phase
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,34 @@
# Task 04 — unit + integration tests for the image progress contract
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a
## Objective
The new position is pinned at both ends: the importer emits it, the status machine carries and clears it, the label formats it.
## Work
1. `tests/unit/test_sync_button.py` — the status-machine pins (the phase-64/98/129 idiom: drive the runner's closures, assert the endpoint dict):
- idle: the full-dict pin includes `images_done: 0` / `images_total: 0` and every pre-existing key unchanged;
- mid-run: after `_image_hook(3, 15)` the endpoint reports `3/15` while `current_file` / the chunk position keep their own values (the three positions are independent);
- per-file reset: a `_hook(...)` call for the NEXT file resets the image position to `0/0` (the pairing rule — a file with no images never shows a stale position);
- every terminal (success / failed / cancelled) clears the image position while the FILE counts keep their final values (the phase-64 convention, restated for the new sub-position);
- the `phase = "overview"` and `phase = "summaries"` transitions clear it (the same sites as the chunk position).
2. `tests/unit/test_importer.py` (or the image-pass home that already uses `tests/fakes.py::FakeEmbedder`) — the hook contract:
- a fixture PDF with N embedded images → the hook fires exactly N times, `(1,N) … (N,N)`, each call BEFORE the corresponding describe call (assert the interleaving by recording events in the fake);
- a PDF with a pictorial span AND an embedded span → one continuous counter across both spans, never exceeding the combined total, never restarting;
- a text-only PDF → the hook never fires;
- `on_image_progress=None` → the `ImportSummary` and the stored rows are identical to the hook-less baseline (the byte-identical regression pin);
- the blank-gate interaction (phase 151): images skipped by the dedupe/blank gate are NOT counted in `total` (the total is the work about to happen).
3. `tests/unit/test_sync_button.py` (frontend-source pins) — `fmtSyncPhaseLabel`'s image part: present iff `images_total > 0`; the composed `full` string pinned for a status carrying both positions (`… (2540/2540) · chunk 125/126 · image 3/15`); the `overview`/`summaries` labels unchanged.
4. Integration (`tests/integration/`): a full sync run over a fixture KB containing one image-bearing PDF, polling `GET /api/sync/status` from the test while the mock import runs — assert the observed `images_done` sequence is monotonically non-decreasing and ends at `images_total` (no backwards jumps, no overshoot).
5. Run the gates: `uv run pytest -q`, then `uv run pytest --cov=app --cov-report=term-missing` (>90%).
## Testing & Quality
- Coverage: **>90%** on `app/api/sync.py` + `app/rag/importer.py` new lines (every clear/reset site, each span, the `None` path).
- Deterministic: no sleeps, no real LLM (the fakes), no reliance on wall-clock timing.
## Completion Criteria
- [ ] the status pins cover idle / mid-run / per-file reset / all three terminals / both phase transitions
- [ ] the importer hook pins cover the N-image case, the continuous multi-span counter, the no-image case, and the hook-less byte-identical baseline
- [ ] the label formatter's image part is pinned byte-for-byte
- [ ] `uv run pytest -q` green; coverage >90%; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,37 @@
# Task 05 — the dedicated E2E: the label advances through a PDF's image span
**Phase:** `152_sync_progress_image_pass` · **Source:** `TODO.md:3` — "The sync status on the RAG page gets \"stuck\" when it's embedding images … the status was stuck on `Syncing… … (2540/2540) · chunk 125/126` for several minutes for some reason"
**Story:** n/a
## Objective
In the browser, during a PDF sync whose images take time to describe, the live label shows the image position MOVING — the owner's "stuck for several minutes" is impossible on the fixture.
## Work
1. `tests/e2e/test_sync_image_progress.py` (new; the `tests/e2e/test_sync_chunk_progress.py` recipe — a module-scoped app on its own port, `BOR_IMAGES=true` + the `pdf` import token, the `tests/e2e/slow_llm.py` delay proxy in front of the mock so EVERY LLM call (vision describes included) takes `SLOW_DELAY_S = 1.5 s` ≫ the UI's 2 s poll, the autouse truncate, mock-only — `E2E_REAL_LLM` unsupported):
- **fixture:** a source (local dir or `file://` git repo, the suite's existing idiom) with ONE PyMuPDF-built PDF carrying 4 genuinely distinct, non-blank images (post-phase-151 they all survive the gate) and enough text to produce ≥2 embed batches (so the chunk part and the image part are both exercised);
- **the observation loop:** press Sync, then poll the page (not the API) and collect the DISTINCT `#sync-progress-text` values seen while the run is in flight;
- **assertions:**
* at least three distinct labels containing `· image ` are observed (e.g. `· image 1/4`, `· image 2/4`, `· image 3/4`) — the freeze the owner reported is a single repeated label, so "≥3 distinct image positions" IS the bug's negation;
* the observed image positions are non-decreasing and never exceed the total;
* the chunk part appeared earlier in the run and the image part appears during the image span (the two positions are distinct);
* the button `title` / `aria-label` carry the same full string (the A4 untruncated contract);
* at settle the box shows the counts line and the image part is gone (no stale position in a terminal state);
* a page reload mid-run re-attaches with the current image position visible (the never-stale contract).
2. Keep the suite deterministic: the slow proxy makes each position outlive the poll; no `wait_for_timeout` as a barrier — wait on the observed text values.
3. Run in isolation until green: `uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov` (DB up: `podman compose up -d db`).
4. Regression run of the neighbouring label suites (they must stay green unedited): `tests/e2e/test_sync_progress_box.py`, `test_sync_chunk_progress.py`, `test_sync_label_mobile.py`, `test_pdf_support.py`, `test_image_documents.py`.
5. Commit the phase per AGENTS.md rule 8 — one atomic Conventional Commit with `--no-gpg-sign`, e.g.
`feat(sync): report PDF image-describe progress so the sync label never stalls`
(moves `.agents/phases/todo/152_sync_progress_image_pass/` to `complete/`).
## Testing & Quality
- The dedicated E2E green IN ISOLATION (the phase gate) + the four neighbouring suites green.
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- Mock-only, deterministic, no real vision calls.
## Completion Criteria
- [ ] ≥3 distinct `· image n/m` label values observed during one sync (the anti-freeze proof)
- [ ] positions are monotonic, cleared at settle, and re-attached on a mid-run reload
- [ ] the chunk part's behavior is unchanged
- [ ] full suite green, coverage >90%, lint + types clean
- [ ] the phase directory moved to `.agents/phases/complete/` and committed with `--no-gpg-sign`
@@ -0,0 +1,43 @@
# Phase 153 — the document viewer carries its origin, and the path is copyable
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "document.html's back link always says '← Sources' even when you arrived from Chat" + optimization "a 'copy path' affordance next to the path chip would speed up the copy-paste-to-terminal workflow."
**Story:** `.agents/user_stories/document-viewer.md` (phase 10, complete) + `.agents/user_stories/document-back-navigation.md` (phase 13, complete) — this phase extends both: the phase-13 `back` contract exists on the DIRECT chip link (`app.js::documentUrl`, L447–457) but the phase-26 same-page modal's "Full page" escape hatch (`document-modal.js::fullPageUrl`, L94–97) drops it, so a chat-originated document opened through the modal lands on "← Sources" in the standalone viewer.
**Context:** `openDocumentModal(source, path, trigger = null)` (document-modal.js L167) — called from the chat view (app.js L1700, L1762, L1804, L1825, L1908 — source chips, nearby-doc links, source-image figures) and from the RAG folder table (sources.js L2023). The "Full page" anchor (L202) and the no-skeleton `window.open` fallback (L172) both use `fullPageUrl`. The standalone back-link resolution (document.js, phase 13) honors a same-origin relative `back` param (`/` prefix, not `//`) and falls back to `/sources.html` otherwise; its label matches the resolved target (Chat / Sources) — already shipped, untouched by this phase. The clipboard idiom is established (app.js L216–221, L2375–2384; history.js L297–301): `navigator.clipboard.writeText` in a try — a non-secure (http) homelab origin rejects the clipboard, so the failure path renders an inline fallback field (an input-like node that selects its content on focus) + a status line.
## Objective
A document opened from the in-page modal carries its origin to the standalone viewer (chat → back to chat, Sources → back to Sources — the phase-13 resolution, finally fed on every path), and the owner can copy the document path (`source/path`) from both the modal's meta and the standalone titlebar with one click, with the established clipboard-fallback contract.
## Dependencies
- `13_document_back_navigation` (complete) — the `back` param + same-origin validation + label-match contract this phase feeds.
- `10_story_document_viewer` (complete) — the viewer's meta row (source/format chips, mono path) the Copy-path control joins.
- `144_modal_back_button` (complete) — the modal's history-entry model; this phase adds no navigation, only a URL param on the escape hatch.
## Design (shared by all tasks — the executor reads this, not the chat)
- **The origin param (task 01):** `openDocumentModal(source, path, trigger = null, back = null)` and `fullPageUrl(source, path, back = null)` — `back` is appended as `&back=` + `encodeURIComponent(back)` ONLY when truthy (a null/absent `back` keeps the URL byte-identical to pre-phase: the Sources path is unchanged). The modal remembers the `back` of the CURRENT open (a module-level variable set at each `openDocumentModal` — the modal is one-at-a-time by its own contract, so a single variable is sound). The five chat-view call sites (app.js) pass `"/"` — chat is the shell's home route, the same value `documentUrl` already uses; the sources.js L2023 call site omits it (null → no param → the viewer's `/sources.html` default). The no-skeleton fallback (L172) uses the same `back`. The app.js L447–457 comment ("this URL is also what the modal's 'Full page' link points at") becomes TRUE — update the comment where it cites the contract, do not restate it.
- **Copy path (task 02):** ONE control shape on both surfaces — an icon + "Copy path" text button (visible text, `aria-label` "Copy the document path" is redundant with the text so it carries the plain text; icon `aria-hidden`), placed in the standalone titlebar's meta row after the mono path and in the modal's meta block. The copied string is `source + "/" + path` (ASSUMPTION A1 — the exact `source/path` form the `read` tool, the chips, and the Sources table use; the paste target is the owner's own tooling). Clipboard contract (the share-link-fallback idiom, app.js L216–221): `navigator.clipboard.writeText` in a try → success: a transient status "Path copied." (visible near the control + the page's live region; auto-clears ~4 s, the toast idiom) → failure: an inline fallback field (input-like, prefilled with the path, selects its content on focus — the `.share-link-fallback` pattern) + the status line "Path ready — copy it from the field." (the owner-locked fallback wording, adapted for the path). Per-page duplication of the small helper is the established idiom (history.js L301) — document.js and document-modal.js each carry their own; no new shared module.
- **A11y (both tasks):** the standalone page already has the landmarks/label frame (phase 10/13); the new button is a labeled control (WCAG 1.3.1/4.1.2); the status line is `role="status"`; the fallback field is a labeled input-like node (the share-fallback precedent). No new color-alone state (B5).
## Tasks
1. `01_modal_back_context.md` — the `back` param through `openDocumentModal`/`fullPageUrl` + the six call-site updates + the comment correction.
2. `02_copy_path.md` — the Copy-path control on the standalone titlebar + the modal meta, clipboard idiom + fallback field + status line.
3. `03_unit_pins.md` — the URL-contract pins (with/without `back`), the modal param default, the copy-path strings + fallback contract.
4. `04_e2e_viewer_context.md` — the dedicated Playwright suite: chat→modal→Full page back-to-chat, Sources→modal→Full page back-to-Sources, Copy path success + fallback.
## Testing & Quality
- Unit: `tests/unit/` source-level pins — `fullPageUrl` byte-identical without `back`, `&back=%2F` appended with `back = "/"` (encoded); `openDocumentModal`'s default is null (the Sources path unchanged); the copy-path status strings pinned byte-for-byte (success + fallback); the phase-13 resolution pins (a garbage `back` still falls back to `/sources.html` — the regression guard).
- Integration: the `/api/documents/content` + `GET /document.html` contract is untouched (no new server surface — the phase is frontend-only; the server pins are regression guards only).
- Coverage: **>90%** on modified code (JS pinned by the source-level suites + the E2E).
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
- E2E: `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py` — a citation-bearing answer renders the chips).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the Copy-path affordance is a user-visible feature — name it where the viewer is described).
## Completion Criteria
- [ ] chat view → source chip → modal → "Full page": the standalone back link is `href="/"` with the "Chat" label; Sources table → modal → "Full page": `href="/sources.html"` with the "Sources" label (E2E)
- [ ] a direct Sources table chip (no `back` param) and the pre-phase URL are byte-identical — no regression on the phase-13/26 paths
- [ ] Copy path on both surfaces: clipboard success reads "Path copied."; a rejected clipboard renders the prefilled fallback field + "Path ready — copy it from the field." (E2E pins both branches)
- [ ] the copied string is exactly `source/path` (the `read`-tool form)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,24 @@
# Task 01 — the modal's "Full page" escape hatch carries the origin `back`
**Phase:** `153_document_viewer_context` · **Story:** `.agents/user_stories/document-back-navigation.md`
## Objective
A document opened from the in-page modal reaches the standalone viewer with the same `back` context the direct chip link already carries (phase 13) — chat-originated opens land back on chat, Sources-originated opens keep the `/sources.html` default.
## Work
1. `frontend/assets/document-modal.js`:
- `fullPageUrl(source, path, back = null)` (L94): append `"&back=" + encodeURIComponent(back)` when `back` is truthy — the encoding contract is the one `app.js::documentUrl` (L447–457) already documents (percent-encoded query values; `back` is a same-origin relative URL, never a URL with its own query).
- `openDocumentModal(source, path, trigger = null, back = null)` (L167): store `back` in a module-level variable (the modal is one-at-a-time by its own contract — a single variable is sound; name it `currentBack` and reset it at the top of every open); the no-skeleton fallback (L172) and the "Full page" anchor (L202: `openEl.href = fullPageUrl(source, path, currentBack)`) both read it.
- Update the L39 and L93–97 comment blocks where the "Full page" URL is described — the contract now matches `documentUrl`'s (cite it, do not restate it).
2. `frontend/assets/app.js` — the five chat-view call sites pass `"/"` (chat is the shell's home route, the value `documentUrl` already uses): L1700 (`openDocumentModal(s.source, s.path, chip)`), L1762 (`… link)`), L1804 (`… fig)`), L1825 (`… chip)`), L1908 (`… fig)`).
3. `frontend/assets/sources.js` L2023 (`openDocumentModal(d.source, d.path, link)`) — UNCHANGED (null → no param → the viewer's `/sources.html` default). Do not touch it; the byte-identity is the contract.
4. `frontend/assets/app.js` L447–457: correct the comment where it says the chip URL "is also what the modal's 'Full page' link points at" — after this task that is true on the chat path (the `back` param rides along); name the fix, keep the existing contract language.
## Testing & Quality
- Unit: the source-level pins in the frontend test suite gain: `fullPageUrl("s", "p")` byte-identical to pre-phase (no `back`); `fullPageUrl("s", "p", "/")` === `"/document.html?source=s&path=p&back=%2F"`; `openDocumentModal` with no `back` arg leaves the Sources path byte-identical; the no-skeleton fallback uses the stored `back`.
- Coverage: **>90%** on the modified JS (source-level pins).
## Completion Criteria
- [ ] chat chip → modal → "Full page" URL carries `&back=%2F`; Sources table → modal → "Full page" URL is byte-identical to pre-phase
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
- [ ] no behavior change in the phase-13/26/144 paths (the direct chip link, the context-menu escape hatch, the modal back-close)
@@ -0,0 +1,25 @@
# Task 02 — Copy path on the standalone titlebar and the modal meta
**Phase:** `153_document_viewer_context` · **Story:** `.agents/user_stories/document-viewer.md`
## Objective
The owner copies the document path with one click on both surfaces — the standalone viewer's titlebar meta row and the modal's meta block — with the established clipboard + inline-fallback contract (a non-secure homelab origin rejects `navigator.clipboard`, so the failure path must always work).
## Work
1. `frontend/document.html` — the titlebar meta row (phase 10/13, L108–113 area): after the mono path cell, add the Copy-path control — an icon (`aria-hidden`) + visible "Copy path" text (the text IS the accessible name; no redundant `aria-label` needed). Follow the icon+text button house pattern (the Sources table's "Ignore paths"/"Remove" buttons are the shape reference).
2. `frontend/assets/document.js` — the control's behavior (a page-local helper, the per-page duplication idiom — history.js L301):
- the copied string is `source + "/" + path` (ASSUMPTION A1 — the exact `source/path` form the `read` tool and the chips use; the same two values the page already renders in the meta row);
- `navigator.clipboard.writeText(…)` in a try: success → a transient visible status "Path copied." near the control (auto-clears ~4 s, the toast idiom — the node is visual-only) + the page's live region reads the same text (one a11y announcer, the share pattern: app.js L219–221); failure → an inline fallback field (an input-like node, prefilled with the path, selects its full content on focus — the `.share-link-fallback` pattern) + the status line "Path ready — copy it from the field." (the owner-locked fallback wording, path-adapted). A failed copy NEVER clears the previous state; re-clicking re-attempts.
- the live region: reuse the page's existing `role="status"` surface where one exists; if the standalone page has none, add one visually-hidden node (the a11y frame is otherwise phase-10's — do not restructure it).
3. `frontend/assets/document-modal.js` — the modal's meta block: the SAME control shape + the same page-local helper (per-page duplication — the modal module is separate). The modal already has `statusLine(text)` (L100) for its loading states — the success/failure status reuses that node shape (`role="status"`), and the fallback field is placed in the meta block next to the control.
4. `frontend/assets/styles.css` — the control's styles (icon+text button, the house pattern) + the fallback field (input-like, select-on-focus is JS; the styling mirrors `.share-link-fallback`). Theme-aware (built-in variables only — no new palette entry, B3/B4: an all-default theme stays byte-identical in the served HTML… the control is in `document.html`/`index.html` markup — it is ALWAYS present, so check the B4 byte-identical contract: the control is a new static node in the built-in pages; B4's contract is about THEME no-ops (no `ui_settings` row → no `#bor-theme` tag), not about page content — the new static node is fine, but the unit test that parses `document.html`/`index.html` structure (if any) must gain the node).
## Testing & Quality
- Unit: source-level pins — the copied string is `source + "/" + path` (pinned byte-for-byte for a path with spaces + slashes); the success status string "Path copied."; the fallback field's prefill + select-on-focus contract; the failure status string "Path ready — copy it from the field."; a rejected-clipboard E2E-mock (the suite's http-origin idiom) renders the fallback field.
- Coverage: **>90%** on the modified JS (source-level pins + E2E).
## Completion Criteria
- [ ] both surfaces: click → clipboard success → "Path copied." (visible + live region); clipboard rejected → prefilled fallback field + "Path ready — copy it from the field."
- [ ] the fallback field's content selects on focus (the share-link-fallback idiom)
- [ ] the a11y frame is intact: labeled control, `role="status"` announcer, no color-alone state
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,23 @@
# Task 03 — unit pins for the URL contract and the copy-path strings
**Phase:** `153_document_viewer_context` · **Story:** n/a
## Objective
The two contracts of this phase — the `back` param on the modal's full-page URL, and the copy-path status/fallback strings — are pinned at the source level so a later refactor cannot silently drop the origin or break the fallback contract.
## Work
1. `tests/unit/` — locate the existing frontend source-level pin suite (the `document-modal.js` / `document.js` / `app.js` pins — the phase-13/144 suites are the precedent) and extend it:
- `fullPageUrl`: no-`back` byte-identical to pre-phase; `back = "/"` → `&back=%2F` appended (the exact string pinned); `back = null` and absent are identical;
- `openDocumentModal`: default `back` is null; a chat open (`back = "/"`) is what the "Full page" anchor and the no-skeleton fallback use;
- copy-path: the composed copy string for a `source`/`path` pair with spaces + slashes (pinned byte-for-byte); the success string; the fallback field prefill; the failure string;
- regression guard: the phase-13 resolution pins stay green (a missing/garbage/protocol-relative `back` falls back to `/sources.html`; a same-origin relative `back` wins — do NOT modify those pins, only add beside them).
2. If `document.html`'s structure is pinned by a unit test (check the existing suite — the no-CDN test pins `index.html` per AGENTS.md rule 6; check whether a `document.html` pin exists), gain the Copy-path node.
## Testing & Quality
- Unit: the pins above; the suite's existing pins all stay green (no pin was deleted or loosened — additions only).
- Coverage: the pins exercise the modified branches; `app/`-side code is untouched (frontend-only phase).
## Completion Criteria
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
- [ ] the no-`back` URL pin is byte-identical to the pre-phase string (the byte-identity contract, the phase-13 precedent)
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,25 @@
# Task 04 — the dedicated Playwright suite: origin back + copy path
**Phase:** `153_document_viewer_context` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_document_viewer_context.py`)
## Objective
An isolated E2E proves the two user-facing contracts: a document opened through the modal from Chat lands back on Chat (and from Sources on Sources) after "Full page", and Copy path works on both surfaces — including the rejected-clipboard fallback branch.
## Work
1. `tests/e2e/test_document_viewer_context.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the neighboring suites: the mock-LLM server by default, the DB truncate/re-import per module):
- **test_chat_origin_full_page_backs_to_chat:** the mock-LLM answer renders a citation-bearing bubble (the mock's standard grounded turn); the chat view's source chip is clicked (the phase-26 intercept opens the modal); "Full page" is clicked (new tab — the Playwright context's second page); on the standalone page: `#doc-back`'s `href === "/"` AND its text is "Chat" (the phase-13 label-match contract, now actually fed).
- **test_sources_origin_full_page_backs_to_sources:** the RAG folder table's path link is clicked (the modal); "Full page"; the standalone `#doc-back` `href === "/sources.html"` + text "Sources" (the default — the URL carries NO `back` param; assert the param's ABSENCE too).
- **test_direct_chip_link_still_carries_back (regression guard):** a context-menu / middle-click on the chat chip (the `documentUrl` escape hatch, pre-phase behavior) still lands with `&back=%2F` — the phase-13 path is untouched.
- **test_copy_path_clipboard_success:** the standalone titlebar's Copy path is clicked with the clipboard permitted (the suite's https-origin idiom, or a granted clipboard mock): the status "Path copied." is visible + the live region reads it; the clipboard content is exactly `source/path`.
- **test_copy_path_fallback:** the clipboard is rejected (the suite's http-origin idiom — the established pattern): the fallback field appears, prefilled with the path, and focuses+selects its content on focus; the status reads "Path ready — copy it from the field."; the modal surface branch is covered by the same two tests run against the modal's control (parameterize the surface: standalone / modal).
- **test_direct_open_without_back_defaults (regression guard):** `GET /document.html?source=…&path=…` (no `back`, e.g. a shared-chat document link) still resolves to `/sources.html` + "Sources" — the phase-13 fallback.
2. The suite boots exactly like the neighboring story suites (the app under the mock LLM; a saved chat with a citation-bearing answer is seeded the same way `test_document_viewer.py` / `test_document_back_navigation.py` do — read those two files for the fixture idiom and mirror it, do not invent a new one).
## Testing & Quality
- E2E: the file above, run in isolation: `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` (DB up: `podman compose up -d db`).
- The suite is DETERMINISTIC: mock-LLM answers only (no `E2E_REAL_LLM` dependency), fixed fixture document (the test-KB document with a known `source/path` — the copied-string assertion is a byte-for-byte pin).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_document_viewer_context.py -v --no-cov` green in isolation
- [ ] every contract of the phase's Completion Criteria is asserted by the suite (the origin back on both surfaces, the label match, the `back` absence on the Sources path, the copy string, both clipboard branches)
- [ ] the suite adds no real-LLM dependency and no fixture beyond the established idiom
@@ -0,0 +1,44 @@
# Phase 154 — sync + upload stamps say WHEN (date-qualified freshness)
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "RAG 'Synced 16:06' badge is time-only — ambiguous the next day" + "Sources 'Uploaded — press Sync sources to import it.' has no timestamp — it's server-persisted and re-rendered on every mount; after that archive is imported by a sync, the line can sit there indefinitely and mislead."
**Story:** n/a (audit-derived phase)
**Context:** `sources.js::fmtSyncTime(iso)` (L814–819) — `HH:MM` only; its consumer `applySyncSuccess` (L1195–1196) settles the RAG sync button as `Synced ${time}` (or bare "Synced" when the ISO is unparseable — the app-restart case: the sync status is in-memory, A12). The label parts writer (L821 onward) carries the same text in the button `title` + the running `aria-label` — the qualified stamp rides all three (the phase-128 parts-writer idiom: write once, three surfaces). `git-sources.js::fmtUploadResult(detail, name)` (L1128–1133) — the no-count contract (phase 90): `Uploaded <name> — press Sync sources to import it.` (fresh 202 settle, the page-local `lastUploadName`) or the nameless `Uploaded — press Sync sources to import it.` (the boot re-render, `initUploadStatus` L1279–1304, where the name is unknown after a reload). `GET /api/git-sources/upload/status` (git_sources.py L704–729) ALREADY exposes `started_at` / `finished_at` (ISO) + `detail` — the server change is ZERO; this phase is frontend-only. The per-page duplication idiom is established (history.js L301's copyText, sources.js vs app.js) — the date-qualifier helper is duplicated into the two modules, not shared.
## Objective
Both freshness stamps tell the owner *when*: a same-day stamp stays `HH:MM` (byte-identical to today's behavior); a non-same-day stamp carries the date (`M/D, h:MM AM/PM` — the table-date family); the Sources upload line carries its completion stamp on every render path (fresh settle AND boot re-render), so a stale "Uploaded" line is self-dating.
## Dependencies
- `150_drop_sync_result` (complete) — the settled sync button is the sole "Synced" surface (the phase-149 box is the running label; the button settles it).
- `64_sync_upload_progress` (complete) — the upload 202/poll/re-attach state machine this phase's line change rides on.
- `90_upload_no_scan` (complete) — the no-count result-line contract (phase 90, A2) this phase EXTENDS (a stamp is added; no counts return).
## Design (shared by all tasks — the executor reads this, not the chat)
- **The date-qualifier rule (ASSUMPTION A1, owner-locked):** the stamp is computed from the ISO's LOCAL date (the page's clock — the table dates already use local time). Same local year as now AND same calendar day → `HH:MM` (byte-identical to the current `fmtSyncTime` output — a same-day badge is unchanged). Otherwise, same year → `M/D, h:MM AM/PM` (e.g. `10/2, 3:14 PM`); different year → `M/D/YYYY, h:MM AM/PM` (e.g. `1/3/2025, 9:12 PM`). An unparseable/missing ISO → the existing fallback (sync: bare "Synced"; upload: no stamp — the pre-phase strings). `h:MM` uses the locale 12-hour form WITHOUT leading zero on the hour (matching the table dates' `5:34:06 PM` family — drop the seconds), month/day without leading zero (the tables' `M/D/YYYY` shape).
- **RAG sync badge (task 01):** `fmtSyncTime` becomes the date-qualifier (keep the name — its contract widens; or add `fmtStamp` and delegate — the executor's choice, the pins fix the output). `applySyncSuccess` is unchanged in shape (`settleSyncButton(time ? \`Synced ${time}\` : "Synced")` — the qualifier lives inside the formatter). The parts writer's comment block (L821 onward) names the qualified stamp in the label parts where the `HH:MM` contract is cited (cite, don't restate).
- **Sources upload line (task 02):** `fmtUploadResult(detail, name, when)` — `when` is the precomputed stamp string (the caller computes it from `status.finished_at`, falling back to `status.started_at`; the success state always has a `finished_at` on the server, the fallback is the defensive guard for a legacy/partial payload). Shapes (the phase-90 contract extended, name present/absent): `Uploaded <name> (<stamp>) — press Sync sources to import it.` / `Uploaded <stamp> — press Sync sources to import it.`; `when` empty (running state, legacy payload) → BYTE-IDENTICAL to the current strings (the no-stamp fallback). Call sites: the poll settle (L1239–1244, the page-local `lastUploadName`) and `initUploadStatus`'s success branch (L1279–1304, nameless) — both read `status.finished_at` from the same `/upload/status` payload. The L78–96 and L1125–1133 comment blocks name the stamp in the contract (cite, don't restate).
- **A11y:** the stamps are text in existing `role="status"` / button-label surfaces — no new nodes, no color-alone state (B5).
## Tasks
1. `01_rag_sync_stamp.md` — the date-qualifier in `sources.js` (`fmtSyncTime`'s contract widens; same-day byte-identical).
2. `02_upload_line_stamp.md` — the `when` param on `fmtUploadResult` + both call sites reading `finished_at`/`started_at`.
3. `03_unit_pins.md` — the qualifier's boundary pins + both stamp shapes (with/without name, with/without stamp).
4. `04_e2e_status_freshness.md` — the dedicated Playwright suite.
## Testing & Quality
- Unit: source-level pins — the qualifier (same-day `HH:MM` byte-identical; yesterday `M/D, h:MM AM/PM`; last-year `M/D/YYYY, h:MM AM/PM`; garbage/missing ISO → the pre-phase fallbacks — a fixed clock in the test, not wall-clock math on `new Date()`); `fmtUploadResult`'s four shapes (name×stamp, name×no-stamp byte-identical, no-name×stamp, no-name×no-stamp byte-identical); the `applySyncSuccess` settle string for a non-same-day `finished_at`.
- Integration: ZERO server change — the `/upload/status` payload is untouched (regression guard: the full-dict pin, if one exists, stays byte-identical).
- Coverage: **>90%** on the modified JS (source-level pins + E2E).
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
- E2E: `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the stamp format is a visible UI change — name it where the sync/upload surfaces are described).
## Completion Criteria
- [ ] a same-day sync settles as `Synced HH:MM` — byte-identical to pre-phase (the pin + the E2E)
- [ ] a non-same-day `finished_at` (the unit pins; the E2E asserts the same-day shape on a real mock sync — the date branch is unit-pinned with a fixed clock, the E2E never depends on the wall clock)
- [ ] the upload line carries the stamp on BOTH render paths (the fresh 202 settle with the name; the boot re-render without it) — the E2E seeds a prior upload via the API and reloads the page
- [ ] a running/legacy upload payload renders the pre-phase strings byte-identically (no partial-state regression)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,23 @@
# Task 01 — the RAG sync badge is date-qualified
**Phase:** `154_sync_status_freshness` · **Story:** n/a
## Objective
The settled RAG sync button reads `Synced HH:MM` when the run finished today and `Synced M/D, h:MM AM/PM` (or `M/D/YYYY, …` across years) otherwise — a badge that no longer lies about its age.
## Work
1. `frontend/assets/sources.js`:
- `fmtSyncTime(iso)` (L814–819): the contract widens to the date-qualifier (the phase design's A1 rule). Keep the function name if the delegation is clean; otherwise add `fmtStamp(iso)` and let `fmtSyncTime` delegate — the pins fix the OUTPUT, the executor's choice is internal. The parse guard (`Number.isNaN(d.getTime()) → ""`) is UNCHANGED — an unparseable ISO still yields the bare "Synced" settle.
- the qualifier compares LOCAL calendar dates (the page's clock — the tables' `Updated` columns already render local dates; the idiom is there in `history.js`'s updated-cell formatter — mirror its `toLocaleDateString`/manual shape, the `M/D(/YYYY), h:MM AM/PM` family with no leading zeros).
- `applySyncSuccess` (L1195–1196) is UNCHANGED in shape — the qualifier lives inside the formatter; the settle string for a same-day run is byte-identical to pre-phase.
- the parts-writer comment block (L821 onward) names the qualified stamp where the `HH:MM` label contract is cited (cite, don't restate — the block already lists the label parts; the badge is one of them).
2. No CSS change (the label part ellipsizes; the longer stamp is just more text in the same part).
## Testing & Quality
- Unit: the source-level pins gain the qualifier's boundary set with a FIXED clock (the test's fake clock — never `new Date()` in the test body): same-day → `HH:MM` (byte-identical to the pre-phase pin); yesterday (same year) → `M/D, h:MM AM/PM`; last-year same M/D → `M/D/YYYY, h:MM AM/PM`; hour formatting (`3:14 PM` — no leading zero on the hour; `12:05 AM` edge); garbage ISO → `""` (unchanged).
- Coverage: **>90%** on the modified JS.
## Completion Criteria
- [ ] same-day settle is byte-identical to pre-phase (`Synced HH:MM`)
- [ ] the boundary pins pass on a fixed clock; the wall clock never appears in a test assertion
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,28 @@
# Task 02 — the Sources upload line carries its completion stamp on every render path
**Phase:** `154_sync_status_freshness` · **Story:** n/a
## Objective
`Uploaded — press Sync sources to import it.` becomes self-dating — `Uploaded 21:14 — …` (or `Uploaded 10/2, 3:14 PM — …`) on BOTH render paths (the fresh 202 settle with the page-local name, the boot re-render without it) — so a line that outlives its upload can no longer masquerade as fresh.
## Work
1. `frontend/assets/git-sources.js`:
- `fmtUploadResult(detail, name, when)` (L1128–1133): the third param is the PRECOMPUTED stamp string (the phase design's A1 format, computed by a page-local `fmtUploadStamp(status)` helper — the per-page duplication idiom; it reads `status.finished_at` first, falls back to `status.started_at`, and yields `""` when both are missing/absent). Shapes:
- `detail.message === "uploaded"` + `name` + `when`: `Uploaded ${name} (${when}) — press Sync sources to import it.`
- `detail.message === "uploaded"` + no `name` + `when`: `Uploaded ${when} — press Sync sources to import it.`
- `when === ""` (any name state): BYTE-IDENTICAL to the current strings (the legacy/partial-payload guard — the phase-90 contract for a payload without timestamps).
- the non-`uploaded` branch (`"The upload finished."`) is unchanged.
- the poll settle (L1239–1244): `fmtUploadResult(detail, lastUploadName, fmtUploadStamp(status))` — the poll tick already holds the full status object.
- `initUploadStatus`'s success branch (L1279–1304): the same third arg from the boot-fetched status (the name stays null — the page-local `lastUploadName` is null after a reload; the stamp comes from the SERVER payload, so the nameless line is still self-dating after a reload — that is the point).
- the L78–96 and L1125–1133 comment blocks name the stamp in the no-count contract (cite, don't restate — the blocks already document the nameless re-render; add the stamp to the described string shape).
2. NO server change: `GET /api/git-sources/upload/status` already exposes `started_at` / `finished_at` (git_sources.py L704–729) — do not touch `app/` in this task.
## Testing & Quality
- Unit: the source-level pins gain the four `fmtUploadResult` shapes (name×stamp, name×no-stamp → byte-identical to the pre-phase string, no-name×stamp, no-name×no-stamp → byte-identical) + `fmtUploadStamp` (finished wins over started; started when finished is missing; `""` when both absent; a garbage ISO → `""`).
- Coverage: **>90%** on the modified JS.
## Completion Criteria
- [ ] the four shape pins pass; the two no-stamp shapes are byte-identical to pre-phase
- [ ] `initUploadStatus`'s success branch passes the stamp (a boot re-render of a prior upload shows the stamp even with the name unknown)
- [ ] `app/` is untouched in this task (a `git diff --name-only` scoped check)
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,22 @@
# Task 03 — unit pins for the qualifier and the four stamp shapes
**Phase:** `154_sync_status_freshness` · **Story:** n/a
## Objective
The date-qualifier rule (A1) and both stamp line shapes are pinned at the source level on a fixed clock — a later refactor cannot silently revert to time-only or break the byte-identity fallbacks.
## Work
1. `tests/unit/` — locate the existing `sources.js` / `git-sources.js` source-level pin suite (the phase-64/90/150/152 suites are the precedent — read `test_frontend_sync_upload.py` and its siblings for the idiom) and extend it:
- the qualifier (task 01's boundary set — fixed clock; the byte-identity of the same-day output against the PRE-PHASE pinned string — pin the old string verbatim in the test so the byte-identity is assertable, not assumed);
- `fmtUploadResult`'s four shapes (task 02 — including the two byte-identity fallbacks pinned against the pre-phase strings);
- `fmtUploadStamp` (finished/started/missing/garbage).
2. The `GET /api/git-sources/upload/status` payload pin (if the suite pins the full dict — the phase-64 precedent): UNCHANGED — this phase adds no server keys; the pin stays byte-identical (a regression guard that task 02 really touched nothing in `app/`).
## Testing & Quality
- Unit: the pins above; every pre-existing pin stays green (additions only — no pin deleted or loosened).
- Coverage: the pins exercise the modified branches.
## Completion Criteria
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
- [ ] the two byte-identity fallback pins assert against the verbatim pre-phase strings
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,22 @@
# Task 04 — the dedicated Playwright suite: date-qualified stamps
**Phase:** `154_sync_status_freshness` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_sync_status_freshness.py`)
## Objective
An isolated E2E proves the two visible contracts on the live UI: a fresh mock sync settles the RAG button as `Synced HH:MM` (same-day shape), and the Sources upload line carries its stamp on BOTH render paths — the fresh settle (with the file name) and the boot re-render after a reload (nameless, still stamped from the server payload).
## Work
1. `tests/e2e/test_sync_status_freshness.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the neighboring sync suites — `test_sync_image_progress.py` (phase 152) and the phase-90/64 suites are the precedents; read one and mirror its boot, do not invent a new fixture):
- **test_fresh_sync_settles_same_day_shape:** a mock sync run (the suite's sync fixture — a small fixture source; the mock-LLM makes the run fast) → the RAG sync button settles; assert the label matches `Synced \d{2}:\d{2}` (the same-day shape — the test runs "now", so the qualifier must yield `HH:MM`; a regex pin, NOT a wall-clock comparison — the suite never asserts on the actual time value).
- **test_upload_line_fresh_settle_carrys_name_and_stamp:** a real small fixture archive is uploaded through the UI (the phase-90 suite's upload idiom — a generated zip in `tmp_path`) → the 202 toast → the poll settles → the ready-for-sync line reads `Uploaded <name> (<stamp>) — press Sync sources to import it.` (the name is the fixture's safe name; the stamp matches the `(\d{2}:\d{2}|\d{1,2}/\d{1,2}(?:/\d{4})?, \d{1,2}:\d{2} [AP]M)` shape pin — same-day in practice, the shape pin covers both).
- **test_upload_line_boot_rerender_stamped_nameless:** continue from the previous state (or re-seed): RELOAD the Sources view/page → the boot re-render (`initUploadStatus`'s success branch) shows the NAMELESS line WITH the stamp: `Uploaded <stamp> — press Sync sources to import it.` (the name is gone after the reload — the phase-90 contract — but the stamp survives because it comes from the server payload: THIS is the regression the phase fixes).
- **test_running_upload_line_unchanged (regression guard):** a running upload (the poll's processing state) never renders the ready-for-sync line (the phase-90 contract — the line appears only on settle; assert the bare "Processing…" label during the run with the suite's slow-LLM/delay idiom if a deterministic in-flight observation is needed, otherwise the settle-only assertion).
2. The suite is DETERMINISTIC: mock-LLM only, no `E2E_REAL_LLM` dependency; time assertions are SHAPE pins (regex), never value comparisons against the wall clock.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_status_freshness.py -v --no-cov` green in isolation
- [ ] the boot-re-render test proves the nameless line carries the stamp (the phase's actual fix)
- [ ] the suite asserts shapes, not wall-clock values; no real-LLM dependency
@@ -0,0 +1,42 @@
# Phase 155 — folder summaries in varied, natural language
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "All `lite` folder descriptions open with 'This folder contains…' — formulaic; a prompt nudge for variety would help."
**Story:** n/a (audit-derived phase; the folder-summary machinery is phase 94's — `94_ls_tree_drilldown` (complete))
**Context:** `app/rag/folder_summaries.py::FOLDER_SUMMARY_INSTRUCTION` (L102–108) — the current neutral instruction: "From the document list below, write a 1-3 sentence plain-text summary of what this folder's documents cover, in natural language. Do not use markdown. Do not invent anything that is not in the list." The `FOLDER_SUMMARY_MODE` marker (L91) and the `FOLDER_HEADER_PREFIX` (`"Folder: "`) are the E2E mock's keys — `tests/e2e/mock_llm.py` keys on the marker + the header line (a canned reply that is a pure function of the request), NOT on the instruction text — an instruction change is mock-safe as long as the marker + header prefix stay byte-identical. `generate_folder_summaries(db, llm, *, skip, only_missing, changed_docs, on_progress)` (L440) — `changed_docs=None` is the FULL regeneration (every candidate folder; the `scripts/import_docs.py` CLI's default); the sync path is change-gated (phase 127). PLAN's "persona text changes through the plan, not in code" does NOT bind this constant — it is a one-shot prompt constant (the phase-94/127 idiom), not the chat persona; the marker contract is the load-bearing piece and it is untouched. LLM calls are free (A21) — a full regeneration's cost is not a constraint.
## Objective
New and regenerated folder summaries open in varied, natural language — no more catalogue of "This folder contains…" — via an instruction nudge, plus a one-shot `scripts/regen_folder_summaries.py` that refreshes every stored summary against the new instruction (the live RAG catalogue's 15+ source-root rows currently all carry the formulaic opener).
## Dependencies
- `94_ls_tree_drilldown` (complete) — the folder-summary machinery (grouping, prompt, upsert, prune).
- `127_sync_incremental_folder_summaries` (complete) — the `changed_docs` mode contract the new script's FULL mode (`changed_docs=None`) must not disturb.
## Design (shared by all tasks — the executor reads this, not the chat)
- **The nudge (task 01):** `FOLDER_SUMMARY_INSTRUCTION` gains the variety direction: keep the 1–3 sentence / plain-text / no-markdown / no-invention constraints UNCHANGED; add: "Open in a varied, natural way — state what the folder is for, the way a person would describe it. Do NOT begin with 'This folder contains' or 'The <folder name> folder contains'." (ASSUMPTION A1, owner-locked — the exact wording above; the executor may polish grammar but not the two forbidden openers, which are the pin.) `FOLDER_SUMMARY_MODE` and `FOLDER_HEADER_PREFIX` stay byte-identical (the mock contract — a unit pin asserts it).
- **The regen script (task 02):** `scripts/regen_folder_summaries.py` — the `scripts/import_docs.py` entrypoint idiom (argparse; the app settings/`get_settings()` fail-loud boot; the DB session construction; the `LLMClient` construction — read `scripts/import_docs.py`'s main path and mirror its exact setup, the `--limit N` debug passthrough included if import_docs has one — check and match). Body: `await generate_folder_summaries(session, llm)` — FULL mode (`changed_docs=None` the default), `only_missing=False` (a REGEN — every stored row is refreshed, not just gaps), `skip=False`. End: the importer's cron-safe single summary log line idiom (one line: counts out of the returned stats dict) + the per-`uv run` exit code 0 only on success. NO new API surface, NO schema change, NO change to the sync path's change-gated behavior (phase 127's contract is untouched — the script is a standalone maintenance entrypoint).
- **The mock (task 04's suite prerequisite):** `tests/e2e/mock_llm.py`'s CANNED folder-summary reply may evolve (it is a pure function of the request — the marker/header keys are unchanged); its canned text for a folder is updated to the new voice (e.g. "Documentation for <folder>, covering <first-doc-theme> …" — a natural opener, never the forbidden ones) so the E2E can assert the stored text's shape. The mock's OTHER markers/contracts are untouched.
## Tasks
1. `01_prompt_nudge.md` — the instruction update + the marker/header byte-identity pins.
2. `02_regen_script.md` — `scripts/regen_folder_summaries.py` (full mode, the import_docs idiom).
3. `03_unit_pins.md` — the instruction pin, the mock-contract guard, the script wiring.
4. `04_e2e_folder_voice.md` — the dedicated Playwright suite (the mock's new canned voice + the RAG table rendering it).
## Testing & Quality
- Unit: `FOLDER_SUMMARY_INSTRUCTION` pinned (the two forbidden openers present as the constraint; the 1–3 sentence/plain-text/no-markdown/no-invention words preserved); `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX` byte-identical (the mock contract); the script's wiring (a fake LLM + a fixture catalogue: `generate_folder_summaries` called with `changed_docs=None`, `only_missing=False`, `skip=False` — the stats dict surfaces in the summary line; a bad settings boot fails loud — the app's validator idiom).
- Integration: a full-mode regeneration over a fixture catalogue upserts every candidate row (the phase-94 idiom) and prunes nothing (no catalogue change — the prune rule is count-driven; a pure regen touches only `updated_at` + text).
- Coverage: **>90%** on new/modified code (`app/rag/folder_summaries.py`, the script).
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
- E2E: `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py`).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the new script is a user-visible command — it joins the README's import/update workflow list beside `import_docs`; the summary voice is a visible content change).
## Completion Criteria
- [ ] `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX` are byte-identical to pre-phase (the mock contract pin)
- [ ] `uv run python -m scripts.regen_folder_summaries` runs a FULL regeneration (the stats summary line logs; exit 0; every candidate row's `updated_at` advances, unchanged folders' text is the mock's new voice)
- [ ] the sync path's change-gated behavior is byte-identical (phase 127's pins stay green — no `app/api/sync.py` change)
- [ ] the E2E: a mock-LLM folder summary is stored + rendered in the RAG table WITHOUT the forbidden openers
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,21 @@
# Task 01 — the instruction nudge (varied openers, marker untouched)
**Phase:** `155_folder_summary_voice` · **Story:** n/a
## Objective
`lite` stops opening every folder summary with "This folder contains…" — the instruction gains the variety direction while the E2E mock's keys (the `FOLDER_SUMMARY_MODE` marker + the `FOLDER_HEADER_PREFIX`) stay byte-identical.
## Work
1. `app/rag/folder_summaries.py` — `FOLDER_SUMMARY_INSTRUCTION` (L102–108): keep the existing four constraints (1–3 sentence; plain text; no markdown; no invention) and ADD the variety direction (the phase design's A1 wording — the two forbidden openers are the pin): "Open in a varied, natural way — state what the folder is for, the way a person would describe it. Do NOT begin with 'This folder contains' or 'The <folder name> folder contains'." (grammar may be polished; the two forbidden openers and the "varied, natural way" direction must survive).
2. `FOLDER_SUMMARY_MODE` (L91) and `FOLDER_HEADER_PREFIX` (`"Folder: "`) — UNTOUCHED (byte-identical — the mock's keys; a unit pin asserts it).
3. `app/rag/folder_summaries.py` module docstring — where the instruction's contract is described, name the nudge (cite the phase, don't restate the whole contract).
4. `tests/e2e/mock_llm.py` — the CANNED folder-summary reply (the pure function of the request): update its text to the new voice (a natural opener naming the folder — e.g. "Documentation for <folder> …" — NEVER the two forbidden openers). The marker + header keys are untouched; the reply stays a pure function of the request (the same inputs → the same output — the determinism contract).
## Testing & Quality
- Unit: the instruction pin (the two forbidden openers present as the constraint; the four preserved constraint words present); the marker + header prefix byte-identical pins; the mock's canned-reply shape (a folder's reply names the folder, avoids the forbidden openers, is a pure function of the request — two identical requests → identical replies).
- Coverage: **>90%** on the modified module (the pins exercise the changed constant).
## Completion Criteria
- [ ] the marker + header prefix pins are byte-identical to pre-phase
- [ ] the mock's canned folder reply avoids the forbidden openers and stays deterministic
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,28 @@
# Task 02 — `scripts/regen_folder_summaries.py` (one-shot full regeneration)
**Phase:** `155_folder_summary_voice` · **Story:** n/a
## Objective
A standalone maintenance entrypoint refreshes EVERY stored folder summary against the (new) instruction in one `uv run` — the live catalogue's 15+ source-root rows all currently carry the formulaic opener, and the sync path's change-gated mode (phase 127) never re-touches unchanged folders, so a one-shot is the only way to re-voice the whole catalogue.
## Work
1. `scripts/regen_folder_summaries.py` — the `scripts/import_docs.py` entrypoint idiom (READ that file first and mirror its exact setup — do not invent a second pattern):
- `build_parser()` (argparse) — `--limit N` (the debug passthrough, same meaning as import_docs': cap the walk/generation; the summary line names it) if import_docs' parser has it — match import_docs' actual flags, do not guess;
- `main(argv: list[str] | None = None) -> int` + the `if __name__ == "__main__": raise SystemExit(main())` tail (match import_docs' tail);
- `get_settings()` + `configure_logging(settings.log_level)` + `configure_debugging()` (the phase-8 debug idiom — `DEBUGPY=1` honored, the AGENTS.md rule 7 contract);
- `llm = LLMClient()`; the short-lived DB session (the phase-53 convention — the script's own session commits; the generator only flushes — read import_docs' session handling and mirror it);
- the body: `stats = await generate_folder_summaries(session, llm)` — FULL mode (`changed_docs=None` is the default — do NOT pass it), `only_missing=False` (a REGEN — every stored row is refreshed, not just gaps; do NOT pass it either — the defaults), `skip=False` (do not pass);
- the cron-safe SINGLE summary line (the importer's idiom — one greppable line: `regen_folder_summaries: folders=<n> …` out of the returned stats dict) on the logger;
- exit 0 on success; non-zero + a named stderr line on failure (the fail-loud idiom — a missing admin secret / DB outage names the failure, never a silent partial).
2. NO new API surface; NO schema change; NO change to `app/api/sync.py` or the importer (phase 127's change-gated contract is untouched — the script is a consumer of the existing `changed_docs=None` full mode).
## Testing & Quality
- Unit: the script's wiring (a fake LLM + a fixture catalogue: `generate_folder_summaries` is called with the FULL-mode defaults — assert the call's kwargs; the stats dict surfaces in the summary line; a settings/boot failure exits non-zero with the named line — the app validator idiom).
- Integration: a full-mode regeneration over a fixture catalogue upserts EVERY candidate row (text + `updated_at` advance for all of them — a pure regen, no prune: the catalogue is unchanged, so the count-driven prune rule fires for nothing).
- Coverage: **>90%** on the script (the wiring pins exercise main()'s paths).
## Completion Criteria
- [ ] `uv run python -m scripts.regen_folder_summaries` runs the full regeneration against a fixture DB: every candidate row's `updated_at` advances, the single summary line logs, exit 0
- [ ] `DEBUGPY=1 uv run python -m scripts.regen_folder_summaries` honors the listener (the debugging idiom — the unit test for `configure_debugging`'s contract, the phase-8 precedent)
- [ ] `app/api/sync.py` + `app/rag/importer.py` are untouched (`git diff --name-only` scoped check)
- [ ] `uv run pytest -q` green (the script's pins); `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,22 @@
# Task 03 — unit pins: the instruction, the mock contract, the script wiring
**Phase:** `155_folder_summary_voice` · **Story:** n/a
## Objective
The load-bearing pieces are pinned: the instruction carries the two forbidden openers as its constraint (the four old constraints preserved), the mock's keys are byte-identical, the mock's canned reply is deterministic and opener-free, and the script calls the generator in full mode.
## Work
1. `tests/unit/` — the folder-summaries pin suite (the phase-94/96/127 suites are the precedent — locate `test_folder_summaries.py` or its equivalent and extend it):
- `FOLDER_SUMMARY_INSTRUCTION`: the two forbidden openers present as the constraint; the four preserved constraint words (1–3 sentence; plain text; no markdown; no invention) present;
- `FOLDER_SUMMARY_MODE` + `FOLDER_HEADER_PREFIX`: byte-identical to the pre-phase values (pin the verbatim strings — the mock contract);
- the mock's canned folder reply (in `test_mock_llm.py` or the mock's own suite — the phase's idiom): names the folder, avoids the two forbidden openers, is a pure function of the request (two identical requests → identical replies; different folders → different replies keyed on the header line).
2. The script's wiring pins (task 02's list — a new test module beside the import_docs' script pins, mirroring that idiom).
## Testing & Quality
- Unit: the pins above; every pre-existing pin stays green (additions only).
- Coverage: the pins exercise the changed constant + the script's paths.
## Completion Criteria
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
- [ ] the marker/header pins assert the verbatim pre-phase strings
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,21 @@
# Task 04 — the dedicated Playwright suite: the new voice, end to end
**Phase:** `155_folder_summary_voice` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_folder_summary_voice.py`)
## Objective
An isolated E2E proves the voice change end to end: a KB change triggers the folder regeneration under the mock LLM, the STORED summary carries the mock's new-voice canned text (no forbidden opener), and the RAG catalogue renders it.
## Work
1. `tests/e2e/test_folder_summary_voice.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-127/94 suites — read `test_incremental_sync.py` / the folder-summary E2E for the boot + the KB-change fixture, mirror it, do not invent a new one):
- **test_regen_summary_avoids_forbidden_openers:** the fixture KB change (a new document under a fixture source folder — the suites' idiom) triggers the sync's folder regeneration (the change-gated path, phase 127) → the stored `folder_summaries` row for that folder equals the mock's canned reply for that folder (a pure function — assert the EXACT canned string) AND does not start with "This folder contains" / "The <folder> folder contains" (the shape pin);
- **test_rag_table_renders_the_stored_voice:** the RAG view's folder table (the phase-97 catalogue) shows the stored text in the DESCRIPTION cell for the regenerated folder (the text is user-derived — `textContent` only, the house rule — assert the visible cell text equals the canned reply, truncated display aside: the cell's `title`/full text attribute carries the full string if the display ellipsizes — check the kb-table cell's idiom in the phase-97 suite and assert accordingly);
- **test_unchanged_folders_keep_their_rows (phase-127 regression guard):** a KB change in folder A does not touch folder B's stored row (text + `updated_at` unchanged — the phase-127 contract; assert via the API/DB the way the phase-127 suite does).
2. The suite is DETERMINISTIC: mock-LLM only (the canned reply is a pure function — no `E2E_REAL_LLM` dependency).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_folder_summary_voice.py -v --no-cov` green in isolation
- [ ] the stored-text assertion is an exact match against the mock's canned reply (deterministic)
- [ ] the phase-127 regression guard is asserted (unchanged folders keep their rows)
@@ -0,0 +1,43 @@
# Phase 156 — tables: sticky headers + ellipsis on clipped labels
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "sticky table headers (RAG/Sources/History — the tables are long; the header scrolls away)" + "At ~1,009px window width the rightmost table columns start clipped with no ellipsis hint on the 'Hidden'/'Edit' labels (the wrapper scrolls fine — just subtle)".
**Story:** n/a (audit-derived phase; the table machinery is phase 97's `97_kb_tree_catalog` + phase 86's `86_history_page_width`, both complete)
**Context:** `styles.css:: .table-wrap` (L2528–2534: `position: relative; background; border; radius; box-shadow; overflow-x: auto`) wraps `.docs-table` (`width: 100%; min-width: 640px`) — the RAG folder table (the `kb-folders-table` class), the Sources table (`#git-sources-table-wrap`, L3390), and the History table (the phase-50/86 width contract, L5583). The thead already has a TINTED background (L530 comment: "the thead tinted") — the sticky design needs it OPAQUE (rows must not show through on scroll). The clipped cells: the Sources table's `Hidden` checkbox label + `Ignore paths`/`Remove` buttons, the RAG table's `Edit` button, the History table's `Copy`/`Unshare`/`Create link`/`Delete` buttons — today a clipped label hard-cuts (no `text-overflow: ellipsis`). **The sticky constraint (the design's load-bearing fact):** a `position: sticky` `th` sticks to its NEAREST SCROLLING ANCESTOR — with `overflow-x: auto`, the wrapper's computed `overflow-y` is `auto` too, so the wrapper IS the scroll container; but it has no height bound, so it never scrolls vertically — the PAGE does. A sticky header in an unbounded wrapper therefore scrolls away with the page. The header sticks only if the wrapper itself is the vertical scroll region (a bounded `max-height`) — the standard scrollable-table-with-sticky-header pattern. That is the UX trade this phase makes (ASSUMPTION A1): the tables become bounded internal-scroll panels; the page no longer carries their height.
## Objective
On the three catalogue tables (RAG folders, Sources, History): the header row stays pinned while the user scrolls the table's own (now bounded) vertical scroll, and a clipped cell label shows `…` instead of a hard cut — the full text remains reachable (title/aria, the established idiom).
## Dependencies
- `97_kb_tree_catalog` (complete) — the RAG folder table + the `.table-wrap` card.
- `86_history_page_width` (complete) — the History table's width contract (the sticky design must keep it — the table stays full-width inside the bounded panel).
- `63_unambiguous_document_listing` (complete) — the Sources table's column idiom (the `Hidden` label + per-row actions).
## Design (shared by all tasks — the executor reads this, not the chat)
- **A1 (owner-locked):** the bounded-panel design — `.table-wrap` gains `max-height` (ASSUMPTION A2: `min(70vh, 64rem)` — the phase picks this; it must keep a 15-row table scrolling while a 3-row one fits without a scrollbar: the `min()` of a viewport fraction and a fixed cap does exactly that) + `overflow: auto` (superseding `overflow-x: auto` — the horizontal scroll contract is UNCHANGED: the table's `min-width: 640px` still triggers the wrapper's horizontal scroll at narrow widths; `overflow: auto` keeps both axes). The `th` sticky: `position: sticky; top: 0; z-index: 1;` on `.table-wrap table thead th` with an OPAQUE background (the existing tint made opaque — `background: var(--surface)` or the existing tint token, whichever the L530 comment names; check the current value and make it non-transparent; do NOT add a new palette variable — B3: the 17-variable contract is untouched, and an all-default theme stays byte-identical in the served CSS — the rule changes apply to BOTH themes equally).
- **A3 (owner-locked):** the ellipsis hint — the clipped cell LABELS get `max-width` + `overflow: hidden; text-overflow: ellipsis; white-space: nowrap;` (a `.cell-clip` class or a direct selector on the action-cell text nodes — the executor's choice; the `docs-table`'s existing cell padding is untouched). Coverage: the Sources `Hidden` checkbox label + `Ignore paths` + `Remove`; the RAG `Edit`; the History action buttons (`Copy`/`Unshare`/`Create link`/`Delete`). The FULL text stays reachable: every clipped label's control already carries its accessible name (the `Remove` button's `aria-label` names the full source — the phase-89 idiom; CHECK each of the other controls and add a `title`/`aria-label` where one is missing — the audit: the RAG `Edit` button, the History action buttons, the `Ignore paths` button — add the missing labels in the same task).
- **Scope guard:** the CHAT view's `.md-table-wrap` (L539 — the markdown-rendered tables) is a DIFFERENT class — UNTOUCHED (it has no card, no height bound; the design is for the catalogue tables only). The mobile ≤640px contract (PLAN §7.1) is re-verified in the E2E (the hamburger nav + the tables' horizontal scroll at ≤640px still work — the `max-height` must not break the mobile scroll model).
## Tasks
1. `01_sticky_header.md` — the bounded-panel `max-height`/`overflow: auto` + the opaque sticky `th` (all three catalogue tables share the class — one rule).
2. `02_ellipsis_hints.md` — the clipped-label ellipsis + the missing title/aria audit (the five control families).
3. `03_unit_pins.md` — the CSS contract pins (the existing styles-pinning idiom — the `:root`-drift test parses `styles.css`; extend the same suite).
4. `04_e2e_table_polish.md` — the dedicated Playwright suite (narrow viewport: the sticky offset + the computed ellipsis style + the mobile contract).
## Testing & Quality
- Unit: the CSS pins — `.table-wrap` carries `max-height: min(70vh, 64rem)` + `overflow: auto`; `.table-wrap table thead th` carries `position: sticky; top: 0; z-index: 1` + an opaque `background` (parse the stylesheet — the styles-pinning idiom, the `:root`-drift test's precedent); the `.md-table-wrap` rule is byte-UNCHANGED (the scope guard).
- Integration: none (CSS-only phase — the server pins stay byte-identical as regression guards).
- Coverage: **>90%** on modified code (CSS pinned by the source-level suite + the E2E).
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
- E2E: `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the bounded-panel scroll is a visible UX change — name it where the pages' layout is described, if anywhere).
## Completion Criteria
- [ ] all three catalogue tables: after scrolling the wrapper's vertical scroll, the `thead` stays at the panel top (the E2E asserts the `th`'s offsetTop ≈ 0 while the wrapper's scrollTop > 0)
- [ ] the horizontal-scroll contract is unchanged (narrow viewport: the table's `min-width: 640px` still scrolls inside the wrapper — the E2E)
- [ ] a clipped action label shows `text-overflow: ellipsis` (the E2E asserts the COMPUTED style on a forced-narrow cell — deterministic, no pixel assertions) and its full text is reachable (title/aria present on every clipped control)
- [ ] the mobile ≤640px contract holds (the hamburger nav + the table scroll — the E2E)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,22 @@
# Task 01 — the bounded-panel sticky header (one rule, three tables)
**Phase:** `156_table_ux_polish` · **Story:** n/a
## Objective
The catalogue tables' header rows stay pinned while the table's own vertical scroll moves — the standard scrollable-table-with-sticky-header pattern, shared by all three tables through the one `.table-wrap` class.
## Work
1. `frontend/assets/styles.css`:
- `.table-wrap` (L2528–2534): `overflow-x: auto` → `overflow: auto` (both axes — the horizontal contract via the table's `min-width: 640px` is unchanged) + `max-height: min(70vh, 64rem)` (the phase design's A2 value). Keep `position: relative` (the sticky's containing context) — the rest of the block (background/border/radius/shadow) is untouched.
- `#git-sources-table-wrap` (L3390) — check whether it OVERRIDES the base `.table-wrap` (a height/overflow of its own): if it does, harmonize it (the same `max-height`/`overflow: auto`; do not leave a divergent rule that would make the Sources table unbounded while the others are bounded — the one-rule contract).
- the sticky rule: `.table-wrap table thead th { position: sticky; top: 0; z-index: 1; }` + an OPAQUE background. Check the existing thead background (L530 comment: "the thead tinted" — the `.docs-table thead th` rule at L542 is the MARKDOWN table (`md-table-wrap`), a DIFFERENT class — the catalogue tables' thead background is set elsewhere; locate the catalogue thead's current background and make it opaque (the tint token as-is if it is already opaque; otherwise the surface token). NO new palette variable (B3); the rule applies equally under a themed page (B4 — a themed page gets the same sticky behavior, the `#bor-theme` override of the background token handles the color).
- the `.md-table-wrap` block (L539) is UNTOUCHED (the scope guard — a pin asserts it).
2. Comment the rule in the file's established voice (the phase-97/86/50 comment blocks that describe the scrollable `.table-wrap` — name the sticky + the bounded panel, cite the phase, don't restate the whole contract).
## Testing & Quality
- Unit: the CSS pins (task 03's suite) — the `max-height` value, `overflow: auto`, the sticky th rule, the opaque background, the `.md-table-wrap` byte-unchanged.
- Coverage: **>90%** (CSS pinned by the source-level suite).
## Completion Criteria
- [ ] the CSS pins pass; the `#git-sources-table-wrap` override is harmonized (no divergent unbounded rule)
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,24 @@
# Task 02 — ellipsis on clipped labels + the title/aria audit
**Phase:** `156_table_ux_polish` · **Story:** n/a
## Objective
A clipped cell label shows `…` instead of a hard cut, and every clipped label's full text stays reachable (title/aria) — the five control families across the three catalogue tables.
## Work
1. `frontend/assets/styles.css` — the ellipsis rule (the phase design's A3): the clipped cell LABELS get `max-width` + `overflow: hidden; text-overflow: ellipsis; white-space: nowrap;`. The executor's choice of selector shape (a `.cell-clip` class added to the label nodes, or direct selectors) — the constraint: the `docs-table` cell padding is untouched, and a cell that FITS shows its full text (the `max-width` must be generous enough that `Edit`/`Hidden`/`Remove` at full width are never clipped — the ellipsis is for the CLIPPED case only; the `max-width` is on the label span, not the cell).
2. The label nodes (markup — `frontend/index.html` + `frontend/assets/git-sources.js` + `frontend/assets/history.js` + `frontend/assets/sources.js` — the tables' renderers build their cells with the DOM APIs, the house rule): wrap the clipped labels in the `.cell-clip` span (or the chosen selector's class) — the five families:
- the Sources table: the `Hidden` checkbox LABEL (the visible text — the checkbox's own `aria-label` already names the full source, the phase-105 idiom), the `Ignore paths` button text, the `Remove` button text (its `aria-label` names the full source — KEEP it; the visible text clips);
- the RAG folder table: the `Edit` button text;
- the History table: the `Copy`/`Unshare`/`Create link`/`Delete` button texts (the `Delete` button's `aria-label` already names the chat title, the phase-66 idiom — CHECK the other three and add the missing `aria-label`s: `Copy the share link for <title>` / `Unshare <title>` / `Create a share link for <title>` — the audit's additions).
3. The missing title/aria (the audit, one task): for every clipped control that has NO accessible name beyond the visible text, add the `aria-label`/`title` — the `Remove` (Sources) and `Delete` (History) precedents are the idiom; the additions must name the ROW's identity (the source URL / the chat title — the values the renderers already have in hand).
4. Comment the change in the file's established voice (cite the phase; the B5 "text + color, never color alone" idiom is already satisfied — the labels are text).
## Testing & Quality
- Unit: the source-level pins — the renderers' cell builders carry the `.cell-clip` span (or the chosen class) on the five label families; the added `aria-label`s are present on the audited controls (assert the attributes the renderers set — the source-pinning idiom).
- Coverage: **>90%** on the modified JS/markup (the pins).
## Completion Criteria
- [ ] the five control families carry the ellipsis class (the pin) and the computed style yields `text-overflow: ellipsis` on a forced-narrow viewport (the E2E, task 04)
- [ ] every clipped control's full text is reachable: the audit's `aria-label` additions are present (the pin + the E2E)
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,24 @@
# Task 03 — CSS contract pins (the styles-pinning idiom)
**Phase:** `156_table_ux_polish` · **Story:** n/a
## Objective
The design's load-bearing CSS facts are pinned at the source level — a later "cleanup" cannot silently unstick the header, unbound the panel, or transparent the thead.
## Work
1. `tests/unit/` — the styles-pinning suite (the `:root`-drift test that PARSES `styles.css` is the idiom — the phase-91/92 theme suites extended it; locate the stylesheet-parsing test module and extend it):
- `.table-wrap`: `max-height: min(70vh, 64rem)` (the exact value pinned) + `overflow: auto`;
- `.table-wrap table thead th`: `position: sticky` + `top: 0` + `z-index: 1` + an OPAQUE `background` (the parse asserts the background is a non-`transparent` token — the exact token is the phase's choice, the OPACITY is the pin);
- `#git-sources-table-wrap`: no divergent unbounded override (if the rule exists, it carries the same `max-height`/`overflow: auto` — the pin asserts harmony);
- `.md-table-wrap`: byte-UNCHANGED (the scope guard — pin the rule's exact pre-phase text);
- the ellipsis rule: the clipped-label selector carries `text-overflow: ellipsis` + `overflow: hidden` + `white-space: nowrap`.
2. The `:root`-drift test itself is UNCHANGED (the theme contract is untouched — this phase adds no palette variable; a pin asserts the 17-variable set is byte-identical).
## Testing & Quality
- Unit: the pins above; every pre-existing pin stays green (additions only).
- Coverage: the pins exercise the changed CSS.
## Completion Criteria
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
- [ ] the `.md-table-wrap` pin asserts the verbatim pre-phase rule
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,23 @@
# Task 04 — the dedicated Playwright suite: sticky + ellipsis + mobile
**Phase:** `156_table_ux_polish` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_table_ux_polish.py`)
## Objective
An isolated E2E proves the visible contracts on the real DOM: the header stays pinned while the table's own scroll moves, a forced-narrow viewport shows the ellipsis computed style on a clipped label with its full text reachable, and the mobile ≤640px contract (hamburger nav + horizontal table scroll) survives the bounded panel.
## Work
1. `tests/e2e/test_table_ux_polish.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-97/86 suites — read one of their E2E files for the boot + the table fixtures, mirror it, do not invent a new one):
- **test_sticky_header_on_vertical_scroll (×3 tables, parameterized — RAG folders / Sources / History):** a tall fixture (the suites' idiom — enough rows to exceed the panel's `max-height`); scroll the wrapper (`el.scrollTop = <a large value>`); assert `wrapper.scrollTop > 0` AND the `thead th`'s `offsetTop ≈ 0` relative to the panel (the offset assertion — the Playwright `boundingBox` delta, the standard sticky assertion; NO pixel-tolerance games: the sticky offset is exactly 0 while scrolling).
- **test_horizontal_scroll_contract_unchanged:** a narrow viewport (the E2E browser resizes — the suite's idiom) → the table's `min-width: 640px` still scrolls horizontally inside the wrapper (`wrapper.scrollWidth > wrapper.clientWidth` with a table wider than the panel) — the phase-97/86 contract.
- **test_ellipsis_on_clipped_label:** force the action column's clip (a narrow viewport that clips the rightmost cells — the audit's ~1,009px case, or a tighter fixture width): assert the clipped label's COMPUTED style `text-overflow === "ellipsis"` (deterministic — the computed style, not a pixel), AND `el.scrollWidth > el.clientWidth` (the label is actually clipped), AND its full text is reachable: the control's `aria-label`/`title` (the audit's additions) contains the row identity (the source URL / the chat title).
- **test_mobile_contract:** ≤640px viewport (the PLAN §7.1 contract — the phase-88 suite's idiom): the hamburger nav opens + the catalogue table's horizontal scroll works inside the bounded panel + the `thead` still pins (the mobile regression guard — the `max-height` must not break the mobile scroll model).
2. The suite is DETERMINISTIC: fixed fixture row counts (the suites' idiom), computed-style assertions (never pixel/offset tolerance beyond the exact-0 sticky offset), no real-LLM dependency.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_table_ux_polish.py -v --no-cov` green in isolation
- [ ] the sticky assertion is the exact-0 offset while `scrollTop > 0` (all three tables)
- [ ] the ellipsis assertion is the computed style + the actual-clip check + the reachable full text
- [ ] the mobile contract test passes (the phase-88 idiom)
@@ -0,0 +1,41 @@
# Phase 157 — hero suggestion chips cap their display at 90 characters
**Source:** owner UX audit 2026-10-03 (live interactive-browser pass): "hero suggestion chips — consider a char cap — long opening questions get hard-truncated mid-word… the chips stretch near full container width before clipping."
**Story:** n/a (audit-derived phase; the chip machinery is phase 80's `80_history_suggestion_chips` + phase 104's `104_chip_sizing_question_cap`, both complete)
**Context:** `app.js::renderChips(container, items, onSelect)` (L1272–1315) — the ONE shared chip component (the hero `#suggestions` + the deflection "Maybe try" chips). Phase 104's contract (owner-locked 2026-09-12, in-code): the single-line chip CLIPS long questions (CSS), `title` is the hover reveal, `aria-label` is the clipped-case accessible name (the source-chip pattern), and the click submits the FULL `text` (`submitSuggestion(text)` + `onSelect(text, btn)`). Today there is NO display-length cap — a 300-char opening question stretches its chip to near full container width before the CSS clip engages (the audit's observation: the hero's third chip ran the full reading-column width). Phase 104 also capped the COMPOSER (the input) — a different surface, untouched here. The chips' content is the 3 newest saved chats' opening questions (phase 80/103) — owner-derived, `textContent`-only (the house rule), so a display cap is a pure presentation change: the click payload stays full.
## Objective
The chip's DISPLAYED text caps at 90 characters with a clean `…` (the chip stays compact — no more full-width hero chips), while the `title` hover reveal, the clipped-case `aria-label`, and the click payload all keep the FULL question (phase 104's contract, extended).
## Dependencies
- `80_history_suggestion_chips` (complete) — the hero chips' content contract (the 3 newest opening questions).
- `104_chip_sizing_question_cap` (complete) — the single-line CSS clip + the `title`/`aria-label` idiom this phase EXTENDS (the cap makes the CSS clip a secondary guard, not the primary one).
## Design (shared by all tasks — the executor reads this, not the chat)
- **The cap (ASSUMPTION A1, owner-locked):** `CHIP_CAP = 90` — a named constant in `app.js` (the module's top, beside the other chip constants). Applied to the DISPLAYED text ONLY in `renderChips`: `btn.textContent = text.length > CHIP_CAP ? text.slice(0, CHIP_CAP).trimEnd() + "…" : text`. `slice(0, 90).trimEnd()` avoids the trailing-space-before-`…` artifact when char 90 is mid-word (the `trimEnd` is the pin). The `title` is UNCHANGED (the full `text` — the hover reveal, phase 104). The `aria-label` logic is UNCHANGED (the clipped-case accessible name is the full `text` — with the cap, a long chip is ALWAYS "clipped" in the visual sense, but phase 104's `scrollWidth > clientWidth` check still governs the aria-label: a 91-char display still overflows its single-line box at typical widths, so the aria-label lands; a SHORT string (< 90) is byte-identical to pre-phase in every respect — the no-cap case is unchanged).
- **The click payload (the load-bearing invariant):** `submitSuggestion(text)` + `onSelect(text, btn)` take the FULL `text` (the un-capped value) — UNCHANGED. The cap is display-only: a capped chip sends the full question (the user gets their full question submitted — the phase-80 contract). A unit pin asserts the handler receives the FULL string for a >90-char item.
- **Scope:** `renderChips` is shared (hero + Maybe-try) — the cap applies to both (a single code path; the Maybe-try chips are title-derived and short — the cap is a no-op there, and the byte-identity of short strings is the guard). NO other file changes (the CSS clip stays as the secondary guard; the composer's phase-104 cap is a different surface — untouched).
## Tasks
1. `01_chip_cap.md` — the `CHIP_CAP` constant + the display-only cap in `renderChips` + the comment in the phase-104 voice.
2. `02_unit_pins.md` — the boundary pins (89/90/91), the full-text invariant, the short-string byte-identity.
3. `03_e2e_chip_cap.md` — the dedicated Playwright suite.
## Testing & Quality
- Unit: source-level pins — the 90-char boundary (an 89-char display is unchanged; a 90-char display is unchanged (the cap is `>`); a 91-char display is `slice(0,90).trimEnd() + "…"` — the exact string pinned); the `title` is the full text in all three cases; the click/`onSelect` payload is the FULL `text` for a >90-char item (the load-bearing invariant); a <90-char item is byte-identical to pre-phase in textContent/title/aria-label (the short-string guard).
- Integration: the `/api/suggestions` contract is untouched (no server change — a regression guard only).
- Coverage: **>90%** on the modified JS (the pins).
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's).
- E2E: `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` in isolation (DB up: `podman compose up -d db`; the deterministic `tests/e2e/mock_llm.py`).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the cap is a visible UI change — name it where the onboarding chips are described, if anywhere).
## Completion Criteria
- [ ] a >90-char opening question renders a ≤91-char display (the cap + the `…`) — the E2E
- [ ] clicking the capped chip submits the FULL question (the user bubble shows the full text — the E2E's load-bearing assertion)
- [ ] the `title` hover reveal + the clipped-case `aria-label` carry the full text (the E2E)
- [ ] a <90-char question is byte-identical to pre-phase (the pin)
- [ ] `uv run pytest` green; the dedicated E2E green in isolation; `uv run ruff check . && uv run pyright` clean; `app/` coverage >90%
- [ ] one atomic commit per AGENTS.md rule 8 (`--no-gpg-sign`)
@@ -0,0 +1,22 @@
# Task 01 — the `CHIP_CAP` display cap in `renderChips`
**Phase:** `157_chat_chip_cap` · **Story:** n/a
## Objective
The shared chip component caps its displayed text at 90 characters with a clean `…` — a long opening question no longer stretches its chip to full container width — while the `title`, the `aria-label`, and the click payload keep the full text.
## Work
1. `frontend/assets/app.js`:
- `const CHIP_CAP = 90;` — the named constant at the module top, beside the other chip constants (the module's established placement — the `suggestions` section header, L1272, is the anchor; place it just above `renderChips`).
- `renderChips` (L1296–1315): the display line becomes the cap (the phase design's A1 shape): `btn.textContent = text.length > CHIP_CAP ? text.slice(0, CHIP_CAP).trimEnd() + "…" : text;` — the `title` line (`btn.title = text`) is UNCHANGED (the full text — the hover reveal, phase 104); the `aria-label` logic (L1314: the `scrollWidth > clientWidth` check → `aria-label = text`) is UNCHANGED; the click handler (`submitSuggestion(text)` + `onSelect(text, btn)`) is UNCHANGED (the full `text` — the load-bearing invariant).
- the phase-104 comment block (L1298–1301, "the single-line chip clips long questions — `title` is the hover reveal, `aria-label` the clipped-case accessible name") gains the cap sentence in the same voice (name the phase-157 cap as the PRIMARY guard, the CSS clip as the secondary — cite, don't restate the whole contract).
2. NO other file changes: the CSS clip rule stays (the secondary guard); the composer's phase-104 cap is a different surface (untouched); the "Maybe try" path reuses `renderChips` automatically (a single code path).
## Testing & Quality
- Unit: task 02's pins.
- Coverage: **>90%** on the modified JS (the pins).
## Completion Criteria
- [ ] the constant + the cap line land exactly as designed (the source pin)
- [ ] the `title` / `aria-label` / click-handler lines are byte-unchanged (a `git diff` scoped check)
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,23 @@
# Task 02 — unit pins: the 90-char boundary, the full-text invariant, the short-string guard
**Phase:** `157_chat_chip_cap` · **Story:** n/a
## Objective
The cap's three contracts are pinned at the source level: the 90-char boundary (display), the full-text invariant (title/aria/click), and the short-string byte-identity (the no-cap case).
## Work
1. `tests/unit/` — the `app.js` source-level pin suite (the phase-80/104 suites are the precedent — locate the chip pin module and extend it):
- **the boundary:** an 89-char item → `textContent` byte-identical to the input (no cap); a 90-char item → byte-identical (the cap is `>`, not `≥`); a 91-char item → `slice(0, 90).trimEnd() + "…"` — the EXACT string pinned (pin a 91-char fixture where char 90 is a letter AND one where it is a space — the `trimEnd` case is the artifact the cap avoids);
- **the full-text invariant:** for a >90-char item: `title` === the full text; the click handler's payload (the `onSelect` spy) === the full text; `submitSuggestion`'s argument === the full text (the load-bearing assertion — a capped chip sends the FULL question);
- **the short-string guard:** a <90-char item → `textContent`/`title` byte-identical to pre-phase + the `aria-label` logic unchanged (the `scrollWidth > clientWidth` check — the DOM-simulation idiom the phase-104 suite uses);
- **the Maybe-try path (the shared-component guard):** the cap applies identically through `appendMaybeTry` (a >90-char suggestion there renders capped — a single code path; a short one is byte-identical).
2. The `/api/suggestions` payload pin (if the suite pins it — the phase-80 idiom): UNCHANGED (no server change in this phase).
## Testing & Quality
- Unit: the pins above; every pre-existing pin stays green (additions only).
- Coverage: the pins exercise the changed branch.
## Completion Criteria
- [ ] `uv run pytest tests/unit -q` green — the new pins + every pre-existing pin
- [ ] the full-text invariant pin asserts the full string (not a length) in `title`, `onSelect`, and `submitSuggestion`
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,23 @@
# Task 03 — the dedicated Playwright suite: the capped chip, the full submit
**Phase:** `157_chat_chip_cap` · **Story:** n/a (a phase-specific suite — `tests/e2e/test_chat_chip_cap.py`)
## Objective
An isolated E2E proves the user-facing contract: a >90-char opening question renders a compact capped chip (display ≤ 91 chars ending in `…`), the `title` hover reveal + the clipped-case `aria-label` carry the full text, and clicking the capped chip submits the FULL question (the user bubble shows the full text).
## Work
1. `tests/e2e/test_chat_chip_cap.py` — the dedicated file (run in isolation per AGENTS.md rules 4/9; the fixture idiom from the phase-80 suite — read `test_suggestion_chips.py` (or its actual name in `tests/e2e/`) for the saved-chat-seeding idiom, mirror it, do not invent a new one):
- **the seed:** a saved chat whose opening question is a >90-char string (the suite builds it — a fixed >90-char question, e.g. 120 chars, so the cap engages deterministically; the saved-chat API the phase-80 suite uses is the seeding idiom) → the hero renders the 3-chip contract (phase 80/103) with the seeded question as a chip.
- **test_capped_display:** the chip's `textContent` length ≤ 91 AND ends with `…` AND equals the pinned `slice(0, 90).trimEnd() + "…"` of the seeded question (the exact-string pin — the E2E asserts the byte-for-byte display, not just a length);
- **test_full_text_reachable:** the chip's `title` attribute === the full seeded question; the chip's `aria-label` (the clipped case — the 91-char display overflows its single-line box at the E2E viewport) === the full seeded question (the phase-104 idiom);
- **test_click_submits_full (the load-bearing assertion):** the chip is clicked → the mock-LLM turn runs → the USER bubble's text === the FULL seeded question (not the capped display — the phase-80 contract: the chip sends the full question; the cap is display-only);
- **test_short_chip_unchanged (the regression guard):** a seeded <90-char opening question renders byte-identical to pre-phase (the chip's `textContent` === the question, no `…` appended) — the short-string guard on the live DOM.
2. The suite is DETERMINISTIC: the mock-LLM answer (no `E2E_REAL_LLM` dependency); the seeded questions are fixed strings (the pins, not generated lengths).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` in isolation (DB up: `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_chat_chip_cap.py -v --no-cov` green in isolation
- [ ] the full-text assertion is byte-for-byte (the `title`, the `aria-label`, and the user bubble all equal the FULL seeded question)
- [ ] the short-chip guard passes (a <90-char question is byte-identical to pre-phase)
@@ -0,0 +1,427 @@
# Phase 158 — Chronological thinking / tool-call timeline
**Source:** owner report (live interactive-browser session, 2026-10-04, shared page
`https://brain.reeseapps.com/shared/95dd7a2a-8cda-478b-b694-53936070b9fb` — a real
`turbo` turn with 6 tool calls): "the thinking block happens first but LLM sent a
normal response, not a thinking response, and then went back to thinking. This is
awkward. Then the tool calling happens, and the thinking block reads strangely
because the tool calls are shown below the thinking block. … show the thinking /
tool call / responses in order, rather than in single big blocks each."
**Story:** n/a (owner bug report — live site, 2026-10-04)
**Context:** `app/rag/agent.py::run_agent` (yields `thinking` / `content` /
`ToolCallPiece` / `ToolResultPiece` in TRUE chronological order per round —
thinking → tool call → thinking → … → answer; the SSE mapping in
`app/api/chat.py` preserves that order, so NO backend stream change is needed);
`frontend/assets/app.js` — `ensureThinkingBlock` (~L1053, ONE `details.thinking`
per message, idempotent; phase 109 D15 re-opens it on every later thinking frame),
`closeThinkingBlock` (~L1075), `closeToolCalls` (~L1086), `appendToolLine`
(~L1136, ONE `details.tool-calls-disclosure` per message — phase 117 D3
open-live/fold-at-rest), `armToolLineClock` (~L1479, the phase-87 clock queries
`.tool-calls .tool-call:last-child`), the turn handler in `runTurn`
(thinking ~L3148 / tool ~L3193 / delta ~L3300 / done ~L3313 branches;
`thinkingAcc` ~L3071, `toolAcc` ~L3080, `segRuns` — NEW — module scope beside
them; the done save point ~L3355, the `tool_result` branch ~L3389, the
stream-drop `finalizeFailedTurn` call ~L3417), `rememberBrainTurn` (L2589,
`meta` spread — the pagehide partial save reuses it), `finalizeFailedTurn`
(L2983); `renderStoredMessage` (~L2089: the `m.thinking` block + the `m.tools`
loop, both rendered collapsed); `frontend/assets/shared.js` — `addThinkingBlock`
(L126) + `addToolLines` (L175, the byte-parity copy, rendered closed);
`app/schemas.py` — `ChatMessage` (L1103, `extra="forbid"`, phase-83 value caps,
the wrap serializer that omits absent image keys), `ToolCall` (L1017),
`HistoryTurn` (L58 — the history wire, UNCHANGED); `tests/e2e/mock_llm.py`
(`_tool_flow` L1820, `_tool_call_stream` L3813, `_sse_stream` L3727,
`compose_thinking` L3589, the trigger-dispatch order in `chat_completions`
L3872 — the more-specific-phrase-wins convention); the E2E suites whose pins
this phase updates (task 05): `test_turn_progress_loader.py` (the phase-109 D15
re-open pin), plus the unit source pins in `test_frontend_feedback.py`,
`test_frontend_turn_loader.py`, `test_frontend_tool_states.py`,
`test_chat_persistence.py`, `test_shared_page.py`, `test_tool_call_compact.py`,
`test_thinking_scroll.py`, `test_big_read_progress.py`, `test_read_range_tool_lines.py`,
`test_read_truncation_frontend.py`, `test_search_tool_scope.py`,
`test_frontend_failed_turn.py` (each pins `ensureThinkingBlock` / `appendToolLine`
/ the clock in different depths — inspect each; ONLY the pins the redesign breaks
change, the rest stay green untouched).
## Bug basis (code-traced, 2026-10-04)
The server streams the turn in true chronological order (`run_agent` yields per
round: thinking → tool call → thinking → … → answer), but the client LUMPS:
`app.js` funnels every thinking frame of the turn into ONE `details.thinking`
block (`ensureThinkingBlock` is idempotent — the phase-109 D15 re-open re-opens
that same block for each later agent round) and every tool frame into ONE
`details.tool-calls-disclosure`. DOM order is therefore always
`[ALL thinking] → [ALL tool calls] → [answer]` regardless of the actual order —
the owner's "thought everything, then called all the tools, then answered"
reading. The "thinking → answer → back to thinking" awkwardness is the D15
re-open live: the model emits a short mid-round reply alongside a tool call
(React style), the first `delta` closes the one thinking block and the answer
starts, then the next round's thinking frame re-opens the SAME block above the
tool lines — the answer begins, then thinking resumes on top of it. The persisted
`bor.chat.v1` record only stores `thinking` (one concatenated string) + `tools`
(one array) — the per-run order is never captured, so the shared page
(`shared.js`) renders the same lumped layout, collapsed.
## Objective
Render each agent turn as a chronological timeline — each CONTIGUOUS run of
thinking frames in its own `details.thinking` block, each CONTIGUOUS run of tool
calls in its own `details.tool-calls-disclosure` (`Tool call(s) (N)` per run),
segments in arrival order — live, on restore, and on the shared page; persist the
run order in a new optional `segments` array on the brain record (additive;
records without it — all pre-phase — restore byte-identically through the
untouched legacy path).
## Owner decisions (chat, 2026-10-04 — recorded per AGENTS.md rule 3)
- **D1 — Per-run timeline.** Each contiguous thinking run = its own Thinking
block; each contiguous tool-call run = its own disclosure. Consecutive frames
of the same kind extend the current run; the first frame of a new kind closes
the previous run and opens the new one.
- **D2 — Option B mid-round placement (owner-chosen, shown as a live DOM demo in
the owner's browser 2026-10-04).** ALL segments render ABOVE the answer
bubble, in arrival order — including runs that start AFTER the first `delta`
(the mid-round reply case). The bubble stays ONE element; its content
accumulates in arrival order (the mid-round text and the final answer are one
streamed text). Rejected: Option A (post-answer segments below the bubble —
the bubble grows above its own scratchpad).
- **D3 — Plain `Thinking` label on every block (owner-chosen).** The phase-17
pinned summary literal stays byte-identical across runs (zero new literals;
position distinguishes the runs). The disclosure summary stays the phase-117
`Tool call(s) (N)` — now per run.
- **D4 — Additive persistence.** The brain record gains an OPTIONAL `segments`
array; `thinking` (full concatenated text — feeds the history wire + the
per-turn log) and `tools` (all call records) are KEPT (old-client graceful
degradation; the phase-113 lesson applied up front). `segments` is omitted when
absent (the `undefined`-drops-the-key house pattern) — a no-thinking,
no-tools record stays byte-identical to pre-phase.
- **D5 — Open live, fold at rest** (extends phase 117 D3 + the phase-17
open-while-thinking contract): the active run's block is OPEN while its frames
stream; a new run's first frame closes the previous run; the first `delta`,
`done`, and the stop/abort settle fold the open runs; the restore path and
the shared page render everything COLLAPSED.
- **D6 — Server: a validated `segments` field on `ChatMessage`.**
`extra="forbid"` means the new key would 422 every saved-chat write without a
schema field (the phase-113 lesson: a missing key silently UNSAVED every
grounded brain record). The field is a discriminated union (`kind` =
`"thinking"` | `"tool"`) with phase-83-style value caps, omitted-when-`None`
in the serializer (the phase-50 round-trip contract). No migration (the record
is opaque JSONB), no SSE contract change, no history-wire change.
## Design (shared by all tasks — the executor reads this, not the chat)
### The `segments` entry shape (D4)
```
segments: [
{ kind: "thinking", text: "<that run's full text>" },
{ kind: "tool", name: "…", argument?: "…", offset?: int, limit?: int,
path?: "…", truncated?: true, chars_shown?: int, chars_total?: int },
…
]
```
- One `thinking` entry per RUN (the run's accumulated text — the same text that
streams live into that run's block). One `tool` entry per TOOL CALL (the exact
`toolAcc` record shape — the phase-37 `{name, argument}` plus the phase-139
`offset/limit` and phase-146 `path` additive fields, and the phase-95
`truncated/chars_shown/chars_total` stamp when present).
- Built at save time from the run tracker: an EMPTY thinking run stores NOTHING
(a zero-char run — e.g. a `BOR_STREAM_THINKING`-suppressed turn — is absent);
a turn with no thinking and no tools stores no `segments` key at all.
- `thinking` (the full concatenation, for the history wire + the log) and `tools`
(every call record, for old clients) are UNCHANGED in the record.
### `app/schemas.py` (task 01)
- `ChatThinkingSegment`: `kind: Literal["thinking"]`, `text: str`
(`min_length=1`, `max_length=32_000` — `HistoryTurn`'s cap, phase 83).
- `ChatToolSegment`: `kind: Literal["tool"]` + the EXACT `ToolCall` field set
(name/argument/offset/limit/path/truncated/chars_shown/chars_total with their
existing caps) — one model, no duplication of the field definitions where
practical (the docstring cross-references `ToolCall`).
- `ChatSegment = Annotated[Union[ChatThinkingSegment, ChatToolSegment],
Field(discriminator="kind")]`.
- `ChatMessage.segments: list[ChatSegment] | None = Field(default=None,
max_length=50)` (the phase-83 cap style — far past any realistic turn: the
round cap is 10 → ≤ 20 entries; 50 matches the `tools` cap).
- The EXISTING wrap serializer (`_drop_image_when_absent`) gains the rule:
`segments is None → pop "segments"` (ABSENT, never `null` — the phase-50
contract; a record WITH segments keeps them). `HistoryTurn` and the
`history_to_messages` mapper are UNTOUCHED (segments never travel the wire —
the concatenated `thinking` travels exactly as today).
### `frontend/assets/app.js` — the live turn (task 02)
- Turn state: `segRuns = []` (module scope, reset in `runTurn` beside
`thinkingAcc`/`toolAcc` — an array of run objects
`{ kind, text, records }`), `currentRun = null` (turn-local).
- **Insertion anchor (D2):** every run element is inserted `before`
`body.querySelector(".bubble")` (get-or-create the bubble first, as today).
There is no other anchor — segments ALWAYS sit above the bubble, in arrival
order. `ensureThinkingBlock`'s old `.tool-calls-disclosure ?? .tool-calls ??
.bubble` anchor is REPLACED by this.
- **`thinking` frame (per frame):** `settleToolLine()` (unchanged);
`thinkingAcc += ev.text` + `sawThinking` + `clearTurnTimeout()` + `if (!wrap)
wrap = addMessage("brain","")` + `removeTyping()` (all unchanged). Then the
run logic: if `currentRun?.kind === "thinking"` → extend it (re-render
`renderMarkdown(run.text)` into THAT block's `.thinking-text` — the existing
pre-update pin-state / bottom-follow logic moves verbatim into this branch,
now per-block); else → close `currentRun` (if any, `details.open = false`),
create `details.thinking` (`open = true`, `<summary>Thinking</summary>` +
`.thinking-text`, inserted before the bubble), push `{kind:"thinking",text:"",
records:[]}` to `segRuns`, `currentRun` = it, extend.
- **`tool` frame (per frame):** the existing guards stay byte-identical
(the phase-139 `frameOffset`/`frameLimit`, the phase-146 `framePath`
well-formedness; `toolAcc.push(toolRecord)`; `clearTurnTimeout()`; the
`toolStatus` label logic; the phase-70 legacy-name mapping). Then the run
logic: if `currentRun?.kind === "tool"` → extend that disclosure; else → close
`currentRun` (if any), create `details.tool-calls-disclosure` (`open = true`,
`summary.tool-calls-summary` + the `div.tool-calls` list `role=list`
`aria-label="Tool calls"`, inserted before the bubble), push
`{kind:"tool",text:"",records:[]}` to `segRuns`, `currentRun` = it; extend =
build the line EXACTLY as today (`appendToolLine` — now with the container
passed in: `appendToolLine(container, name, argument, offset, limit, scope)`,
the line bytes + the four pinned `line.textContent` label literals + the
phase-139 range suffix + the phase-146 scope span UNCHANGED, the summary
count updated per append — `Tool call(s) (N)` per run), then
`armToolLineClock(wrap, lineEl)` (the NEW signature — task 02's clock fix).
- **`armToolLineClock(wrap, lineEl)` (the phase-87 multi-disclosure fix):**
store the `lineEl` reference (module scope, reset with the wrap); the interval
appends the `.tool-elapsed` suffix to `lineEl` instead of querying
`.tool-calls .tool-call:last-child` (that query breaks with multiple
disclosures — it would find the last line of the FIRST list). The visible
behavior is identical (the suffix on the newest line; `settleToolLine` still
removes every `.tool-elapsed` in the wrap + resets the reference).
- **`delta` handler:** `settleToolLine()` + `acc += ev.text` + the
`thinking→streaming` transition UNCHANGED; replace
`closeThinkingBlock(wrap)` + `closeToolCalls(wrap)` with `closeCurrentRun()`
(folds `currentRun` if open; idempotent; no-op without a run).
- **The other settle-pair sites — `finalizeFailedTurn`'s `if (wrap)` block
(L2990–L2991), the STOP path (L3477–L3478), and every remaining
`closeThinkingBlock(wrap); closeToolCalls(wrap);` pair (grep — the
redesign removes both functions):** `closeAllRuns(wrap)` (closes EVERY
open `details.thinking` + `details.tool-calls-disclosure` in the wrap —
defensive; by construction at most the last run is open, but a late
double-close must be a no-op). The rest of `done` is unchanged.
- **Save payloads (the FIVE sites):** `buildSegments(segRuns)` (a helper —
maps runs: thinking run → `{kind:"thinking",text}` when non-empty; tool run
→ one `{kind:"tool",…record}` per record, in order) — the `done` save point
(~L3355); the `finalizeFailedTurn` parameter + its `if (wrap)` saved record
(the two call sites — stream-drop ~L3431, network-error ~L3519 — pass
`buildSegments(segRuns)`); the STOP path's save meta (L3487–L3491,
`stopped: true`); the pagehide partial (L3642–L3650 — `segRuns` is module
scope, readable there). All include `segments` non-empty only. The
zero-frame fallback save needs nothing (zero frames → no runs → no key).
The `thinking`/`tools` keys are UNCHANGED at every site.
- The UI state machine, `#send-status` copy, the 300 s pre-token guard, the
stream-drop guard, and the `thinking-without-answer` fallback are ALL
UNCHANGED (§7.4 never-stale intact). `BOR_STREAM_THINKING=0` needs no client
change (no thinking frames arrive → no thinking runs).
### `frontend/assets/app.js` — the restore path (task 03)
- `renderStoredMessage`: after the bubble exists, IF
`Array.isArray(m.segments) && m.segments.length` → `restoreTimeline(wrap,
m.segments)` and SKIP the legacy `m.thinking` block + the `m.tools` loop
(segments take precedence — they carry the same data in order); ELSE the
legacy path is BYTE-IDENTICAL (a pre-phase record — or a `BOR_STREAM_THINKING=0`
turn saved with the old client — restores exactly as today).
- `restoreTimeline`: iterate `m.segments` in order; group CONSECUTIVE
`kind:"tool"` entries into ONE `details.tool-calls-disclosure` (created
CLOSED — D5; summary `Tool call(s) (N)`; each line through the SAME
`appendToolLine` container-signature path with the EXISTING tOffset/tLimit/tPath
guards + the existing `appendTruncatedNote` call for `t.truncated` records —
the current loop's guards move verbatim inside the grouping); a `kind:"thinking"`
entry → its OWN `details.thinking` (created CLOSED; `.thinking-text` =
`renderMarkdown(entry.text)` — the restore convention stays escape-first);
unknown `kind` values are SKIPPED (never render from storage — textContent
only, the house rule). All elements inserted before the bubble.
### `frontend/assets/shared.js` — the shared page (task 03)
- `addSegments(wrap, segments)` — the shared-page parity copy of
`restoreTimeline` (pure render, always COLLAPSED — D5): the same grouping,
the same line bytes (the four label literals + the phase-95 truncation note
+ the phase-139/146 suffixes, already parity-pinned in `addToolLines`), the
same summary count; the `addThinkingBlock`/`addToolLines` legacy path is
BYTE-IDENTICAL when the record has no `segments`. The record-dispatch site
(~L456): `if (Array.isArray(m.segments) && m.segments.length)
addSegments(wrap, m.segments); else { the existing addThinkingBlock +
addToolLines }`.
### `tests/e2e/mock_llm.py` (task 04)
- `THINK_TOOLS_TRIGGER = "think out loud and use your tools"` — checked in the
streaming branch BEFORE the `_tool_flow` dispatch (the more-specific-phrase-wins
convention — the phrase contains both `think out loud` and `use your tools`,
so it must not fall through to the plain answer path; no other existing
trigger phrase is a substring of it, and no existing E2E question carries it —
verify both before wiring).
- `_thinking_tools_flow(body)` — stateless, shaped like `_tool_flow` (gated on
the `tools` parameter + the `<tools>` section, like the other marker flows):
- request 1 (tools offered, no tool results yet): `thinking` =
`"Step 1: list the documents to find the reference server config."` →
`_tool_call_stream("ls", {}, "call_0", thinking=thinking)`;
- request 2 (a `tool`-role top-level source listing — the agent's `ls` tree
format, no file lines yet): `"Step 2: drill into the first source."` → the
`_drill_target` scoped `ls` (id `call_1`), with thinking;
- request 3 (a `tool`-role folder listing with file lines — parse the FIRST
file line's labeled `source:` / `path:` fields, the phase-63/70 format):
`"Step 3: read the file line's document."` → `read` on the JOINED
combined `source/path` (id `call_2`), with thinking;
- request 4 (a `tool`-role read result — content starting with the agent's
`"Document <source/path>:"` header): `"Step 4: compose the answer from the
read document."` → the phase-37 answer convention: `Read <source/path>.
<first 80 chars of the read document's content>` (content stream, with
thinking ahead of it).
- `_tool_call_stream(name, arguments, call_id, thinking="")` — the optional
`thinking` param: when non-empty, yield `reasoning_content` chunks FIRST
(the 12-char split — the `_sse_stream` convention), then the existing
`tool_calls` frames + `finish_reason: "tool_calls"` (unchanged). Default
`""` → every pre-phase 3-arg caller streams BYTE-IDENTICAL.
- Docstring: the new trigger entry (the phase-158 flow) — the mock stays
mock-only by design (`E2E_REAL_LLM` note unchanged).
### `tests/e2e/test_thinking_timeline.py` (task 04 — dedicated suite, run in isolation)
The conftest `app_server`/`page`/`seeded_kb` fixtures + the phase-37 admin
login + the trigger question (the KB must be seeded so request 4's read result
exists — mirror `test_agent_document_tools.py`). Six tests:
1. **`test_timeline_order_after_completed_turn`** — after `done`: the brain
wrap's `.msg-body` children before the bubble are EXACTLY
`[details.thinking, details.tool-calls-disclosure, details.thinking,
details.tool-calls-disclosure, details.thinking, details.tool-calls-disclosure,
details.thinking]` (7 elements); the four `.thinking-text`s carry DISTINCT
`Step 1`…`Step 4` markers (distinctness = the runs are separate blocks, not
one lump); each disclosure summary is `Tool call (1)`; the answer bubble
carries the `Read <path>` echo.
2. **`test_runs_fold_at_rest`** — after `done`: NO `.thinking` /
`.tool-calls-disclosure` has the `open` attribute; the summaries are
visible (D5).
3. **`test_live_runs_open_while_streaming`** — during the turn (an
`expect.poll` predicate over the mock's 0.1 s pacing window): at least one
`.thinking` or `.tool-calls-disclosure` is `open` while the answer bubble is
still empty (the D5 live-open contract — the live feedback the owner keeps).
4. **`test_restored_timeline_and_record`** — RELOAD (same context): the
restored message renders the same 7-element timeline, all closed; the
localStorage `bor.chat.v1` record for that brain turn carries a `segments`
array of EXACTLY 7 entries in the pattern
`thinking, tool, thinking, tool, thinking, tool, thinking` (4 thinking runs
+ 3 tool entries).
5. **`test_shared_page_timeline`** — the phase-51 save + share flow, a FRESH
context on `/shared/<token>`: the same 7-element timeline renders, all
collapsed (the `shared.js` parity — `addSegments`).
6. **`test_no_thinking_turn_single_disclosure`** — a plain `use your tools`
question (NO thinking trigger): the turn renders EXACTLY ONE
`.tool-calls-disclosure` (3 lines, `Tool calls (3)`) and ZERO `.thinking`
(the run-grouping contract: no thinking frames → one tool run — the
phase-117 behavior for no-thinking turns is preserved).
## Tasks
1. `01_schema_segments.md` — the discriminated `ChatSegment` models + the
`ChatMessage.segments` field + the serializer omission + the integration
contract (accept / caps / byte-identical round-trip / shared read).
2. `02_live_timeline.md` — the per-run live rendering (run tracker, D2 anchor,
the clock re-target, the settle sites, the save payloads) + the JS source
unit pins.
3. `03_restore_and_shared.md` — the `restoreTimeline` dispatch in
`renderStoredMessage` + the `shared.js` `addSegments` parity + the unit pins.
4. `04_mock_and_e2e.md` — the mock `THINK_TOOLS_TRIGGER` flow + the dedicated
`test_thinking_timeline.py` story suite, run in isolation.
5. `05_regression_update_and_commit.md` — the phase-109 suite pin update + the
remaining unit-pin updates, the full gate, the documentation pass, the
atomic commit, the phase move.
## Testing & Quality
- Unit (new): `tests/unit/test_thinking_timeline.py` — the source-level pins:
the per-run creation (a NEW `details.thinking` when the run kind changes,
insert-before-bubble anchor — D2), the plain `Thinking` literal (D3), the
per-run disclosure + the `Tool call(s) (N)` per-run count, the
`armToolLineClock` reference (the `:last-child` query is GONE), the
`segRuns` → `buildSegments` mapping (empty-run skip, the tool-record shape),
the `delta` `closeCurrentRun` + the settle-pair sites → `closeAllRuns`, the
FIVE save-payload sites (done / finalizeFailedTurn + its 2 call sites /
stop path / pagehide), the restore dispatch (segments vs legacy — the
legacy path byte-identical), the `shared.js` `addSegments` parity, and
`innerHTML` ABSENT from the restore/shared builders (the house rule).
- Integration: `tests/integration/test_chats_api.py` (extended) — a record WITH
segments (both kinds, every additive tool field) is accepted and round-trips
through create/get; `segments` > 50 → 422; a thinking-segment text > 32 000 →
422; an unknown segment `kind` → 422 (`extra="forbid"` intact — a stray
`<b>`-ish key still 422s); a record WITHOUT `segments` round-trips with the
key ABSENT (never `null` — the phase-50 byte-identical contract, pinned);
the shared read (`/api/chats/<id>/shared` or the public read path) carries
segments (the public shape).
- Existing unit suites MUST stay green: every pin NOT broken by the redesign
(the tool-line label literals, the phase-117 deboxed CSS, the phase-87
`.tool-elapsed` behavior, the phase-139/146 suffix pins, the emoji guard,
the `test_tool_call_compact.py` pins — a no-thinking turn is one run, so the
phase-117 E2E contract is UNCHANGED).
- E2E: `tests/e2e/test_thinking_timeline.py` (new — the six tests above), green
**in isolation** (`uv run pytest tests/e2e/test_thinking_timeline.py -v
--no-cov`), mock-only.
- Regression E2E (task 05, in isolation): `test_thinking_display.py` (a
no-tools thinking turn is ONE run — the suite's contract is unchanged),
`test_tool_call_compact.py` (unchanged — no-thinking single-run),
`test_turn_progress_loader.py` (UPDATED in task 05 — the D15 pin → the
per-run contract), `test_big_read_progress.py` (the clock's visible behavior
is identical — must stay green UNCHANGED), `test_agent_document_tools.py`,
`test_share_chat.py`, `test_chat_persistence.py`, `test_smoke.py`.
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` **>90%** on
`app/` (task 01's schema code is unit/integration-covered; the rest is held
by the suite).
- Lint/types: `uv run ruff check . && uv run pyright` clean.
- The phase is "Complete" only when the **entire** project test suite passes
100% (every test in the project, not just this phase's).
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the
project as it stands after this phase — fixing every inconsistency found,
**including pre-existing ones not caused by the phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are
user-visible (the timeline is a visible UI change — name it where the chat
thinking/tool-call display is described, if anywhere).
## Completion Criteria
- [ ] A completed tool+thinking turn renders `Thinking → Tool call (1) →
Thinking → Tool call (1) → … → answer` — the E2E's 7-element order pin
(task 04 test 1); the runs fold at rest (test 2) and stay open while
streaming (test 3).
- [ ] A reload restores the same timeline + the record's `segments` array
(test 4); a fresh context on `/shared/<token>` renders the same timeline
(test 5).
- [ ] A no-thinking tool turn is ONE disclosure, 3 lines (test 6) — the
phase-117 contract is preserved; `test_tool_call_compact.py` +
`test_thinking_display.py` + `test_big_read_progress.py` green in
isolation UNCHANGED.
- [ ] A pre-phase saved record (no `segments`) restores byte-identically
(the integration pin + the E2E regression suites).
- [ ] `uv run pytest` green (100% of the project suite);
`uv run pytest --cov=app --cov-report=term-missing` > 90%;
`tests/e2e/test_thinking_timeline.py` green in isolation;
`uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`,
`frontend/assets/shared.js`, `tests/`, and the phase files.
- [ ] README.md and AGENTS.md are accurate and consistent with the whole
project (verified in the final pass).
- [ ] ONE owner-directed exception to "no behavior change in completed phases":
the phase-109 D15 re-open contract is REPLACED by the per-run blocks
(owner direction 2026-10-04 — recorded above); `test_turn_progress_loader.py`'
pin is updated in task 05 to the per-run contract (two thinking blocks,
each carrying its own round, open in its live window, folded at rest).
- [ ] One atomic `--no-gpg-sign` commit (below); the phase dir moved to
`.agents/phases/complete/`.
## Commit
```bash
git add -A .agents/ app/schemas.py frontend/ tests/ README.md && git commit --no-gpg-sign -m "feat(ui): render thinking and tool runs as a chronological timeline (per-run blocks, segments persistence)"
```
## Locked decisions
- **The SSE contract is UNCHANGED (A15 intact).** The frames already arrive in
true chronological order (`run_agent`'s yield order — verified in code); this
phase is client-grouping + record persistence only. `app/rag/` is untouched.
- **D1–D6 (owner, 2026-10-04)** — the per-run timeline, Option B placement,
the plain `Thinking` label, the additive `segments` record, open-live/
fold-at-rest, and the validated `ChatMessage.segments` field (the
`extra="forbid"` boundary preserved with phase-83-style caps).
- **A10 history wire UNCHANGED** — segments never travel to the LLM (the
concatenated `thinking` travels as `reasoning_content` exactly as today;
`HistoryTurn` + the mapper are untouched).
- **A13 UNCHANGED** — no migration (the record is opaque JSONB; the schema
field is validation at the boundary only).
- **A16 honoured** — one new story E2E suite + the task-05 regression updates;
mock-only by design. **A17 honoured** — one atomic `--no-gpg-sign` commit.
- **The phase-87 clock re-target** (the `:last-child` query → the tracked
reference) is a bug fix in service of D1; its visible behavior (the suffix on
the newest line) is byte-identical — `test_big_read_progress.py` stays green
unchanged. No other anchor changed.
@@ -0,0 +1,69 @@
# Task 01 — the validated `ChatMessage.segments` field (server)
**Phase:** `158_thinking_timeline` · **Story:** n/a
## Objective
Saved-chat writes accept the new optional `segments` array (a discriminated
`thinking`/`tool` union with phase-83-style caps), omit it when absent so
pre-phase records round-trip byte-identically, and reject malformed values at
the `extra="forbid"` boundary.
## Work
1. `app/schemas.py`:
- `ChatThinkingSegment` (a `BaseModel`, beside `ToolCall`): `kind:
Literal["thinking"]`, `text: str = Field(min_length=1, max_length=32_000)`
(`HistoryTurn`'s cap, phase 83 — docstring cross-reference).
- `ChatToolSegment` (a `BaseModel`): `kind: Literal["tool"]` + the EXACT
`ToolCall` field set (name / argument / offset / limit / path / truncated /
chars_shown / chars_total with their existing types and caps — mirror, do
not weaken; docstring cross-references `ToolCall` as the shape source).
- `ChatSegment = Annotated[Union[ChatThinkingSegment, ChatToolSegment],
Field(discriminator="kind")]`.
- `ChatMessage.segments: list[ChatSegment] | None = Field(default=None,
max_length=50)` — with the phase-83 voice comment (the cap sizing: far
past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches
the `tools` cap).
- The EXISTING wrap serializer (`_drop_image_when_absent`) gains the
omission rule: after the two image pops, `if self.segments is None:
data.pop("segments", None)` — ABSENT, never `null` (the phase-50
round-trip contract; the phase-122 `SourceRef.image_url` omission
precedent, the phase-113 lesson applied up front: the field EXISTS so
the boundary does NOT 422 the done-time auto-save carrying it).
- The `ChatMessage` docstring gains the phase-158 paragraph (the shape,
the omission rule, the `extra="forbid"` boundary) in the established
voice; `HistoryTurn` and everything in `app/rag/` is UNTOUCHED (the
history wire stays `{who, text, thinking}`).
2. NO migration, NO router change (`app/api/chats.py` validates through
`ChatMessage` — `m.model_dump()` + `ChatMessage.model_validate(m)` already
carry the new field end to end, including the public shared read
`SharedChatOut.messages`).
## Testing & Quality
- Integration (extend `tests/integration/test_chats_api.py` — the file already
pins the brain-record contract, phase 83/113/120/123/124):
- a record WITH `segments` (one `thinking` entry + two `tool` entries
carrying every additive field: `offset/limit`, `path`,
`truncated/chars_shown/chars_total`) is accepted by `POST /api/chats` and
round-trips through `GET` (the shape preserved, entry order preserved);
- `segments` with 51 entries → 422; a thinking-segment `text` of 32 001
chars → 422; a segment with `kind: "bogus"` → 422; a record with a stray
top-level key (e.g. `"foo": 1`) still → 422 (the `extra="forbid"`
boundary intact);
- a record WITHOUT `segments` (a pre-phase brain record, the full phase-113
shape) round-trips with the key ABSENT from the served JSON (never
`null` — the byte-identity pin, the phase-50 contract);
- the shared read path (the public read endpoint's `messages`) carries the
segments of a saved record with them (the public shape, the phase-51
contract).
- Coverage: **>90%** on this task's new/modified code (the schema code is
exercised by the integration tests).
## Completion Criteria
- [ ] the three models + the field + the serializer rule land exactly as
designed (a source check)
- [ ] the new integration tests are green; the EXISTING
`test_chats_api.py` tests stay green unchanged
- [ ] `uv run pytest tests/integration -q` green; `uv run ruff check . && uv
run pyright` clean
- [ ] no change in `app/rag/`, no migration, no router change (a `git diff
--stat` scoped check)
@@ -0,0 +1,195 @@
# Task 02 — the per-run live timeline in `app.js`
**Phase:** `158_thinking_timeline` · **Story:** n/a
## Objective
Replace the single-block/single-disclosure turn rendering with the per-run
timeline: each contiguous thinking run streams into its own `details.thinking`
block, each contiguous tool-call run into its own
`details.tool-calls-disclosure`, all segments ABOVE the answer bubble in
arrival order (D2); the runs open live and fold at rest (D5); the phase-87
clock is re-targeted to a tracked line reference; the save payloads persist
the `segments` array (D4).
## Work
1. `frontend/assets/app.js` — turn state (the `runTurn` locals / module scope,
~L3065–L3080):
- `segRuns = []` — MODULE scope (beside `thinkingAcc`, which the pagehide
partial reads), reset to `[]` in `runTurn` with the other turn locals.
Entries: `{ kind: "thinking", text: "", records: [] }` /
`{ kind: "tool", text: "", records: [] }` (the tool entry's `records`
holds the `toolAcc` record objects — the SAME objects `toolAcc` holds,
so the phase-95 `tool_result` stamp (`t.truncated = true; t.chars_shown
= …; t.chars_total = …`, ~L3398–L3404) flows into both automatically —
NO new wiring).
- `currentRun = null` — turn-local (the active run's element + kind).
2. `frontend/assets/app.js` — the run builders (replace `ensureThinkingBlock`
+ the single-container logic in `appendToolLine`):
- `ensureThinkingRun(wrap)` — returns the CURRENT thinking run's
`details.thinking` if `currentRun?.kind === "thinking"`; else closes
`currentRun` (if any — `open = false`), creates a NEW
`details.thinking` (`open = true`, `<summary>Thinking</summary>` — the
phase-17 literal, D3 — built via the existing `innerHTML` template that
`ensureThinkingBlock` uses, OR createElement/textContent — either is fine,
but the RESTORE path's builder stays createElement/textContent (the house
rule pins `innerHTML` ABSENT from the restore/shared builders — see task
03)), inserts it `before body.querySelector(".bubble")` (the bubble
get-or-created first), pushes a fresh `{kind:"thinking",…}` to `segRuns`,
sets `currentRun`.
- `ensureToolRun(wrap)` — the same, for
`details.tool-calls-disclosure` (`open = true`,
`summary.tool-calls-summary` + the `div.tool-calls` list `role=list`
`aria-label="Tool calls"` — the phase-117 structure, now per run).
- `appendToolLine` — the signature becomes
`appendToolLine(container, name, argument, offset = null, limit = null,
scope = null)` (the CALLER passes the run's disclosure — no more
`body.querySelector(".tool-calls-disclosure")` single-container
derivation). EVERYTHING else in the function is BYTE-IDENTICAL: the
`.tool-calls` list get-or-create, the line bytes, the four pinned
`line.textContent` label literals, the phase-139 range suffix, the
phase-146 scope span, the `Tool call(s) (N)` count.
- `closeCurrentRun()` — folds `currentRun` (if any, `open = false`);
idempotent; no-op without a run.
- `closeAllRuns(wrap)` — closes EVERY open `details.thinking` +
`details.tool-calls-disclosure` in the wrap (defensive; the `done` +
stop/abort settle sites).
3. `frontend/assets/app.js` — the turn-handler branches:
- the `thinking` branch (~L3148): keep `settleToolLine()`,
`thinkingAcc += ev.text || ""`, `sawThinking`, `clearTurnTimeout()`,
`if (!wrap) wrap = addMessage("brain", "")`, `removeTyping()` — all
unchanged. REPLACE the `ensureThinkingBlock` + `block.open = true`
(phase-109 D15 re-open — SUPERSEDED by D1) with
`const block = ensureThinkingRun(wrap);` + the run's text extension:
the run's `text` accumulates `ev.text`; the pre-update pin-state /
bottom-follow logic (the `isThinkingNearBottom` capture +
`scrollTop = scrollHeight`, ~L3166–L3189) moves VERBATIM into this
branch, now operating on THIS block (per-block — the existing comment
block stays, updated to name the per-run placement).
- the `tool` branch (~L3193): keep the phase-139/146 guards,
`toolAcc.push(toolRecord)`, `clearTurnTimeout()`, the wrap
get-or-create, and the `toolStatus` label logic UNCHANGED. REPLACE
`appendToolLine(wrap, …)` + `armToolLineClock(wrap)` with:
`const disc = ensureToolRun(wrap); const line =
appendToolLine(disc, name, argument, frameOffset, frameLimit, framePath);`
(the function returns the appended `line` element — a one-line change,
the function already has it in scope) +
`armToolLineClock(wrap, line)`. The run record is the SAME `toolRecord`
object pushed to `toolAcc` (push it into the run's `records` too — or
derive the run's `records` from `segRuns` at save time via the call
order — the executor picks ONE, but the phase-95 stamp must reach the
`segments` payload: the simplest is the run's `records.push(toolRecord)`
alongside the `toolAcc.push(toolRecord)`).
- the `delta` branch (~L3300): `settleToolLine()` + `acc += …` + the
`thinking→streaming` transition UNCHANGED; replace
`closeThinkingBlock(wrap)` + `closeToolCalls(wrap)` with
`closeCurrentRun()`.
- the THREE remaining settle-pair sites — `finalizeFailedTurn`'s
`if (wrap)` block (L2990–L2991), the STOP path (L3477–L3478), and any
other `closeThinkingBlock(wrap); closeToolCalls(wrap);` pair (grep for
the pair — the redesign removes both functions): replace the pair with
`closeAllRuns(wrap);`.
- `removeTyping()` / the typing-indicator / the `#send-status` copy / the
UI state machine / the 300 s guard / the stream-drop guard / the
`thinking-without-answer` fallback: ALL UNCHANGED (§7.4 intact).
4. `frontend/assets/app.js` — the clock (the phase-87 re-target, ~L1479):
- `armToolLineClock(wrap, lineEl)` — the NEW second param: store `lineEl`
in a module-scope `toolLineEl` (reset to `null` with `toolLineWrap` in
`stopToolLineClock` and on re-arm); the interval body uses `toolLineEl`
(null-guarded — the wrap-reset no-op stays) instead of
`toolLineWrap?.querySelector?.(".tool-calls .tool-call:last-child")`.
The suffix element + the `(Ns)` text + the `TOOL_LINE_ELAPSED_AFTER_MS`
gate are UNCHANGED.
- `settleToolLine()` — the `querySelectorAll(".tool-elapsed")` removal stays
(it clears every line in the wrap — correct with multiple disclosures);
add `toolLineEl = null` alongside.
- the comment block updated to name the re-target (the multi-disclosure
fix, phase 158 D1).
5. `frontend/assets/app.js` — the save payloads (D4) — the FIVE sites:
- `buildSegments(segRuns)` — a helper (beside the turn state): maps the
runs IN ORDER — a thinking run → `{ kind: "thinking", text: run.text }`
ONLY when `run.text` is non-empty (an empty run stores nothing); a tool
run → one `{ kind: "tool", …record }` per record in order (spread the
record object — the additive `offset/limit/path` + the phase-95 stamp
fields ride through; NO re-validation on the client — the server's
`ChatMessage` boundary validates).
- the `done` save point (~L3355, the `rememberBrainTurn` meta):
`segments: buildSegments(segRuns).length ? buildSegments(segRuns) :
undefined` (the `undefined`-drops-the-key house pattern, D4) — the
`thinking: thinkingAcc || undefined` + `tools: toolAcc.length ? toolAcc :
undefined` lines are UNCHANGED.
- `finalizeFailedTurn` (L2983): the parameter object gains `segments` (the
TWO call sites — the stream-drop site ~L3431 and the network-error
site ~L3519 — pass `buildSegments(segRuns)`); the function's `if
(wrap)` saved record includes `segments` ONLY when the passed value is
a non-empty array (`segments: segments?.length ? segments :
undefined`); the existing `acc`/`thinking`/`tools` fields are
UNCHANGED.
- the STOP path's save point (L3487–L3491, the `stopped: true` meta):
gains the same `segments` field (non-empty only) — a stopped partial
restores its timeline.
- the pagehide partial (L3642–L3650, the module-scope read — `segRuns`
is module scope, so it is readable there): gains the same `segments`
field (non-empty only). The zero-frame fallback save (L3456–L3462)
needs NOTHING (zero frames → no runs → no key).
- the `thinking`/`tools` keys are UNCHANGED at every site.
6. `tests/unit/test_thinking_timeline.py` (NEW — the source-level house
pattern, the `js[fn : js.find("\n}\n", fn)]` body-extraction the existing
pins use):
- `ensureThinkingRun` body: creates `details` + `className = "thinking"` +
`open = true` + the plain `Thinking` summary literal (D3 — the phase-17
literal, NOT a numbered variant) + inserts before `.bubble` (D2 — the
anchor is the bubble, NOT the old `.tool-calls-disclosure ?? …` chain);
`ensureToolRun` body: `className = "tool-calls-disclosure"` + `open =
true` + the summary/list structure.
- `appendToolLine` signature: the first param is the container (no
`body.querySelector(".tool-calls-disclosure")` in the body — the
single-container derivation is GONE); the four label literals STILL
present (mirror the guard); returns the line.
- `armToolLineClock`: the `:last-child` query is GONE; a `toolLineEl`
reference is used; `settleToolLine` resets it.
- `closeCurrentRun` / `closeAllRuns` defined; the `delta` branch calls
`closeCurrentRun` (pin: appears in the delta-branch region); `done` +
the stop/abort settle call `closeAllRuns`.
- `buildSegments` present; the `done` save + the `finalizeFailedTurn`
call site + the pagehide partial all reference it (≥ 3 call sites).
- `innerHTML` NOT in `ensureThinkingRun`… — pin per the builder choice
made in Work item 2 (if the live builder reuses the phase-17 innerHTML
template, pin that; the RESTORE/SHARED builders (task 03) are pinned
innerHTML-free there).
- `segRuns` is module scope + reset in `runTurn`.
- Builder style: MIRROR the function it replaces (the live thinking
builder keeps the phase-17 static-structure convention — a static
template with NO data — the data flows through `renderMarkdown` /
`textContent` exactly as today; the live tool builder keeps the
phase-117 createElement/textContent style — `innerHTML` ABSENT from
`appendToolLine`, as pinned today). Pin the STRUCTURE + the no-data-
through-innerHTML rule, not the mechanism.
## Testing & Quality
- Unit: `tests/unit/test_thinking_timeline.py` (this task's pins) + UPDATE the
existing source pins THIS task's change breaks (inspect, in order:
`test_frontend_feedback.py`, `test_frontend_turn_loader.py` — the phase-109
D15 re-open pin becomes the per-run pin (a NEW block, not a re-open),
`test_frontend_tool_states.py` — the `appendToolLine` single-container pin,
`test_big_read_progress.py` — the clock-query pin, `test_frontend_failed_turn.py`
— the `finalizeFailedTurn` payload pin, `test_thinking_scroll.py`,
`test_chat_persistence.py` — ONLY the pins this task touches; the
restore-path pins are task 03's, the shared-page pins are task 03's). Every
pin this task does NOT break (the tool-line label literals, the phase-117
deboxed CSS, the phase-139/146 suffix pins, the emoji guard) stays green
UNCHANGED.
- Coverage: **>90%** on this task's modified code (the pins).
## Completion Criteria
- [ ] a live turn renders per-run blocks in arrival order (the code paths
land exactly as designed — the unit pins)
- [ ] the clock targets the tracked line reference (the `:last-child` query
is gone — the unit pin)
- [ ] the `done` / failed / pagehide save payloads carry `segments`
(non-empty only) — the unit pins
- [ ] `uv run pytest tests/unit -q` green (every pin this task breaks is
updated; every pin it doesn't touch stays green); `uv run ruff check .
&& uv run pyright` clean
- [ ] `git diff --stat` limited to `frontend/assets/app.js`,
`tests/unit/test_thinking_timeline.py`, and the unit pin files updated
in this task
@@ -0,0 +1,82 @@
# Task 03 — the restore path + the shared-page timeline
**Phase:** `158_thinking_timeline` · **Story:** n/a
## Objective
A restored brain record renders its `segments` as the collapsed timeline (the
grouping mirrors the live run structure); a record WITHOUT `segments` — every
pre-phase record — restores byte-identically through the untouched legacy
path; the shared page renders the same timeline from the same record shape.
## Work
1. `frontend/assets/app.js` — `renderStoredMessage` (~L2089):
- AFTER the bubble exists: `if (Array.isArray(m.segments) &&
m.segments.length) { restoreTimeline(wrap, m.segments); } else { THE
EXISTING `m.thinking` BLOCK + THE EXISTING `m.tools` LOOP,
BYTE-IDENTICAL (including the `closeToolCalls(wrap)` call — RENAME it
to `closeAllRuns(wrap)` — task 02 removed `closeToolCalls` — the
restore's folded-at-rest result is the same) }`.
- `restoreTimeline(wrap, segments)` (a new function, the house style —
createElement/textContent, `renderMarkdown` for the thinking text,
`innerHTML` ABSENT for the structure): iterate `segments` in order:
- a `kind: "thinking"` entry (guard: `typeof entry.text === "string"`)
→ its OWN `details.thinking` (created CLOSED — D5;
`<summary>Thinking</summary>` — the phase-17 literal, D3;
`.thinking-text` = `renderMarkdown(entry.text)` — the existing
restore convention, escape-first), inserted before the bubble;
- CONSECUTIVE `kind: "tool"` entries → ONE
`details.tool-calls-disclosure` (created CLOSED — D5;
`summary.tool-calls-summary` + the `div.tool-calls` list
`role=list` `aria-label="Tool calls"` — the phase-117 structure),
each entry's line through `appendToolLine` (task 02's container
signature) with the EXISTING tOffset/tLimit/tPath guards
(`Number.isInteger(x) && x >= 1` / the string guard — byte-identical,
the phase-139/146 restore precedent) + the EXISTING
`appendTruncatedNote(wrap, arg, …)` call for `entry.truncated`
records (the phase-95 marker re-render — moved verbatim inside the
grouping); the summary count `Tool call(s) (N)` set once per
disclosure after its entries;
- an entry with an unknown `kind` (or a malformed shape) → SKIPPED
(never render from storage — textContent only, the house rule;
a guard comment naming the skip).
2. `frontend/assets/shared.js` — `addSegments(wrap, segments)` (new, the
shared-page parity copy of `restoreTimeline` — pure render, always
COLLAPSED — D5): the same grouping, the same line bytes (the four label
literals + the phase-95 truncation note + the phase-139/146 suffixes —
the parity `addToolLines` already carries; mirror it), the same summary
count, the same unknown-`kind` skip. The record-dispatch site (~L456):
`if (Array.isArray(m.segments) && m.segments.length) {
addSegments(wrap, m.segments); } else { THE EXISTING addThinkingBlock +
addToolLines, BYTE-IDENTICAL }`. The `addThinkingBlock` / `addToolLines`
functions themselves are UNCHANGED (the legacy path is intact).
3. `tests/unit/test_thinking_timeline.py` — ADD the pins:
- the `renderStoredMessage` dispatch (segments take precedence; the
legacy path intact when `segments` is absent — pin the
`Array.isArray(m.segments) && m.segments.length` guard);
- `restoreTimeline` body: `document.createElement("details")` +
`className = "thinking"` (NO `open = true` — collapsed, D5) + the plain
`Thinking` literal + the consecutive-tool grouping (one disclosure per
run — pin the grouping loop) + the `appendTruncatedNote` call + the
unknown-`kind` skip; `innerHTML` ABSENT (structure built via
createElement/textContent).
- the `shared.js` `addSegments` body: the same structure pins (collapsed,
the parity) + `innerHTML` ABSENT; the dispatch guard in `shared.js`.
## Testing & Quality
- Unit: `tests/unit/test_thinking_timeline.py` (this task's pins) + UPDATE the
existing source pins THIS task's change breaks (inspect:
`test_chat_persistence.py` — the restore-path pins, `test_shared_page.py` —
the `addToolLines`/dispatch pins; every pin this task does NOT break stays
green UNCHANGED).
- Coverage: **>90%** on this task's modified code (the pins).
## Completion Criteria
- [ ] the `restoreTimeline` + `addSegments` builders land exactly as designed
(the unit pins)
- [ ] the legacy restore / shared paths are byte-unchanged (a `git diff`
scoped check — only the dispatch guard is added)
- [ ] `uv run pytest tests/unit -q` green; `uv run ruff check . && uv run
pyright` clean
- [ ] `git diff --stat` limited to `frontend/assets/app.js`,
`frontend/assets/shared.js`, `tests/unit/test_thinking_timeline.py`,
and the unit pin files updated in this task
@@ -0,0 +1,115 @@
# Task 04 — the mock interleaved flow + the dedicated E2E suite
**Phase:** `158_thinking_timeline` · **Story:** n/a
## Objective
A deterministic mock trigger that interleaves a DISTINCT reasoning run before
every tool call and before the final answer (the phase-158 flow), and the
dedicated `test_thinking_timeline.py` Playwright suite that pins the
timeline contract (order / fold-at-rest / live-open / restore + record /
shared-page parity / the no-thinking single-run grouping).
## Work
1. `tests/e2e/mock_llm.py`:
- `THINK_TOOLS_TRIGGER = "think out loud and use your tools"` (beside
`THINKING_TRIGGER`, L992) + the docstring entry for the flow (the
four-request stateless shape, the `Step N` markers, the phase-37
answer convention). BEFORE wiring, VERIFY (and note in the docstring):
no existing trigger phrase is a substring of it, and NO existing E2E
question contains it (grep `tests/e2e/*.py` for the phrase — the
convention that keeps the other suites byte-unaffected).
- `_thinking_tools_flow(body) -> tuple[str, ...] | None` (shaped like
`_tool_flow`, L1820 — the same stateless discrimination, the same
`<tools>` + `tools`-parameter gating, reusing `_drill_target` /
`_first_file_line` / the `read`-result header check):
- request 1 (tools offered, no tool results yet):
`thinking = "Step 1: list the documents to find the reference server
config."` → `_tool_call_stream("ls", {}, "call_0", thinking)`;
- request 2 (a `tool`-role top-level source listing — no file lines
yet): `thinking = "Step 2: drill into the first source."` → the
`_drill_target` scoped `ls` (id `call_1`), with thinking;
- request 3 (a `tool`-role folder listing WITH file lines — the
phase-63/70 labeled `source:` / `path:` format): `thinking = "Step 3:
read the file line's document."` → `read` on the JOINED combined
`source/path` (id `call_2`), with thinking;
- request 4 (a `tool`-role read result — content starting with the
agent's `"Document <source/path>:"` header): `thinking = "Step 4:
compose the answer from the read document."` → the phase-37 answer
convention: `Read <source/path>. <first 80 chars of the read
document's content>` (the `_sse_stream` content path — the
`thinking=` param of `_sse_stream` already exists — pass it; the
final answer's thinking streams ahead of the content, exactly like
the `think out loud` answer path).
- `_tool_call_stream(name, arguments, call_id, thinking="")` (L3813):
the optional fourth param — when non-empty, yield
`reasoning_content` chunks FIRST (the 12-char split — the
`_sse_stream` convention), then the EXISTING `tool_calls` frames +
`finish_reason: "tool_calls"` UNCHANGED. Default `""` → every
pre-phase 3-arg caller (the ~10 call sites) streams BYTE-IDENTICAL
(verify with the existing tool-flow E2E suites in the regression
pass). The docstring updated (the param, the phase-158 flow).
- the `chat_completions` streaming dispatch: check
`_thinking_tools_flow(body)` BEFORE the `_tool_flow(body)` dispatch
(L4453 region) — the more-specific-phrase-wins convention (the phrase
contains `use your tools`, so the plain `_tool_flow` must NOT see
it); on a hit, route each flow stage through the same
`StreamingResponse` wrapper the `_tool_flow` stages use.
2. `tests/e2e/test_thinking_timeline.py` (NEW — the six tests, the phase
design's contract):
- fixtures: the conftest `app_server` / `page` / `seeded_kb` (mirror
`test_agent_document_tools.py` — the admin login, the seeded KB so
request 4's read result exists);
- 1. `test_timeline_order_after_completed_turn` — the trigger
question; after the `done` state: the brain wrap's `.msg-body`
children before the bubble are EXACTLY `[details.thinking,
details.tool-calls-disclosure, details.thinking,
details.tool-calls-disclosure, details.thinking,
details.tool-calls-disclosure, details.thinking]` (assert the
tag+class sequence — 7 elements); the four `.thinking-text`s carry
DISTINCT `Step 1`…`Step 4` markers (a substring assert each —
distinctness proves the runs are separate blocks, not one lump); each
disclosure summary is `Tool call (1)`; the answer bubble carries the
`Read ` echo.
- 2. `test_runs_fold_at_rest` — after `done`: NO `.thinking` /
`.tool-calls-disclosure` has the `open` attribute (a `count` assert
on `[open]`); the summaries are visible.
- 3. `test_live_runs_open_while_streaming` — an `expect.poll` predicate
over the mock's pacing window (the tool frames' 0.1 s + the thinking
stream): a `.thinking` or `.tool-calls-disclosure` is `open` while
the answer bubble is still EMPTY (the D5 live-open contract).
- 4. `test_restored_timeline_and_record` — RELOAD (same context): the
restored message renders the same 7-element sequence, all closed; the
localStorage record (read via `page.evaluate` on the `bor.chat.v1`
storage key — the phase-14 key) for that brain turn carries a
`segments` array of EXACTLY 7 entries in the pattern
`thinking, tool, thinking, tool, thinking, tool, thinking` (assert the
`kind` sequence + the `Step N` text on the thinking entries + the
`name` on the tool entries); the `thinking` (full) + `tools` (3
entries) keys are STILL present (D4 — the legacy fallback fields).
- 5. `test_shared_page_timeline` — the phase-51 save + share flow (the
`test_share_chat.py` pattern), a FRESH context on `/shared/<token>`:
the same 7-element sequence renders, all collapsed, the `Step N`
markers visible when expanded (expand ONE — the click on a summary —
then assert), the answer bubble carries the `Read ` echo.
- 6. `test_no_thinking_turn_single_disclosure` — a plain `use your
tools` question (NO thinking trigger): EXACTLY ONE
`.tool-calls-disclosure` (3 `.tool-call` lines, summary `Tool calls
(3)`) and ZERO `.thinking` (the run-grouping contract — no thinking
frames → one tool run — the phase-117 behavior preserved).
## Testing & Quality
- The mock change is exercised by the suite (the trigger flow) + the
existing tool-flow suites stay byte-unaffected (the `thinking=""` default
— verified in the regression pass, task 05).
- E2E: `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov`
green **in isolation** (prereq: `podman compose up -d db`), mock-only.
- Coverage: the mock is test code (no `app/` change — the coverage floor is
held by the untouched suite; task 01's schema code is covered there).
## Completion Criteria
- [ ] the mock trigger + flow + the `_tool_call_stream` param land exactly
as designed; the pre-phase callers are byte-unchanged (a `git diff`
scoped check)
- [ ] all six tests pass in isolation
- [ ] `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` green
in isolation; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,85 @@
# Task 05 — the phase-109 suite pin update, the full gate, docs pass, commit
**Phase:** `158_thinking_timeline` · **Story:** n/a
## Objective
Update `test_turn_progress_loader.py` from the phase-109 D15 re-open pin to
the per-run contract (the ONE owner-directed behavior change — recorded in
the phase design), run the full completion gate (unit + integration,
coverage >90%, the phase E2E + the regression E2E suites in isolation,
lint + types), run the documentation & consistency pass, and cut the single
atomic commit.
## Work
1. `tests/e2e/test_turn_progress_loader.py` — the D15 pin → the per-run
contract (the mock's turn-progress flow is `delta → tool → thinking →
delta → thinking → done` — the first thinking run starts AFTER the
first delta):
- the re-open pin (`test_thinking_block_reopens_after_delta_with_visible_loader`
— "the thinking block re-opens after the answer has started … the
scratchpad carries both thinking rounds") becomes the PER-RUN pin:
after the turn, the brain wrap holds TWO `.thinking` blocks (the
`count` assert), EACH carrying its OWN round's text (the flow's two
distinct reasoning texts — assert a marker on each, the way the mock's
two reasoning streams are already distinct); both are FOLDED at rest
(no `[open]`); the visible-during-streaming half of the pin (the
loader visible, the thinking text non-empty and open while its run
streams) is kept — a run is open in its live window (D5 replaces D15's
re-open: a new block instead of a re-opened one — the user sees the
new thinking either way; the D15 symptom — "the frozen chat" — stays
fixed). The module docstring updated (the phase-158 supersession,
owner direction 2026-10-04, recorded).
- the other tests in the file (the loader visibility pins) stay green
UNCHANGED — verify.
2. The full gate (the phase's Testing & Quality section):
- `uv run pytest` — the ENTIRE project suite, 100% green;
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on
`app/`;
- `uv run pytest tests/e2e/test_thinking_timeline.py -v --no-cov` — the
phase E2E, green in isolation (prereq: `podman compose up -d db`);
- the regression E2E suites, each green in isolation
(`-v --no-cov`): `test_thinking_display.py`,
`test_tool_call_compact.py`, `test_turn_progress_loader.py` (the
UPDATED pins), `test_big_read_progress.py` (UNCHANGED — the clock's
visible behavior is identical), `test_agent_document_tools.py`,
`test_share_chat.py`, `test_chat_persistence.py`, `test_smoke.py`;
- `uv run ruff check . && uv run pyright` clean.
- If a regression suite fails on a pin the redesign broke that tasks
02–04 missed: update THAT pin to the per-run contract (the same
supersession logic as item 1 — note it in the commit body), and
re-run the suite in isolation.
3. The documentation & consistency pass (the phase's Documentation &
Consistency section):
- `README.md` + `AGENTS.md` reviewed in full against the project as it
stands after this phase; every inconsistency fixed, **including
pre-existing ones** (the README's chat thinking/tool-call display
description, if any, gains the timeline sentence).
- the `app/schemas.py` docstring voice (task 01) + the
`frontend/assets/app.js` / `shared.js` comment blocks (tasks 02–03)
read in the established phase-citation voice (spot-check).
4. The commit + the phase move:
- `git add -A .agents/ app/schemas.py frontend/ tests/ README.md && git
commit --no-gpg-sign -m "feat(ui): render thinking and tool runs as a
chronological timeline (per-run blocks, segments persistence)"`
(the AGENTS.md rule 8 + A17; the LLM-Generated trailer per the
git-commit skill's convention for this repo's commits — check the
last few commits' bodies for the trailer form and match);
- `mv .agents/phases/todo/158_thinking_timeline
.agents/phases/complete/158_thinking_timeline`.
## Testing & Quality
- The gate above IS this task's testing; no new code.
- Coverage: the >90% floor (the gate).
## Completion Criteria
- [ ] `test_turn_progress_loader.py` updated to the per-run contract +
green in isolation (item 1)
- [ ] the FULL gate green (item 2 — every line)
- [ ] README.md + AGENTS.md accurate and consistent with the whole project
(item 3 — the final pass)
- [ ] `git diff --stat` of the phase's total footprint limited to
`app/schemas.py`, `frontend/assets/app.js`,
`frontend/assets/shared.js`, `tests/`, the docs, and the phase files
(the phase design's scope check)
- [ ] one atomic `--no-gpg-sign` commit; the phase dir moved to
`.agents/phases/complete/`
@@ -0,0 +1,58 @@
# Phase 159 — Edit the last question and re-ask it
**Source:** `TODO.md` L3 — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-10-05)
**Context:** `frontend/assets/app.js` (4,007 lines) — `getActionsRow(wrap)` (L498, the `.msg-meta.msg-actions` `role=list` row every bubble action joins as a `listitem`), `appendTuneButton` (L517) + `openTuneForm` (L866, the inline-form precedent: `form.tune-form` + labelled `textarea[maxlength]` + `.tune-form-actions` Save/Cancel + a `role=alert` status line + "one form at a time" via `document.querySelectorAll(".tune-form").forEach(f => f.remove())`), `appendRetryButton` (L617) + `markLastRetryable()` (L650, the single-target discipline: remove every `.retry-btn`, re-append on exactly one bubble, gated on the record invariant), `addMessage(who, html, scroll, image)` (L969, the ONE bubble renderer — `.msg.<who> > .avatar + .msg-body > .bubble`), `renderStoredMessage(m)` (L2254; the user branch L2262 renders `renderMarkdown(m.text)` + the optional `m.image`), `restoreConversation()` (L2410; ends with `markLastRetryable()`), `MAX_QUESTION_CHARS = 4000` (L1689, mirrors `ChatRequest.message max_length=4000`) + `updateCharCount()` (L1702, the `len/4000 — character limit` copy), `conversation` + `trimToBudget` (L2128) + `saveConversation()` (L2143, localStorage `bor.chat.v1`) + `persistConversation()` (L2532, the phase-55 auto-save), `retryLastTurn(wrap)` (L2984, the redo-in-place precedent: pop the last brain record, keep the question as the last entry, `runTurn(text, { reask: true })`), `runTurn(text, { reask, image })` (L3260; `reask` skips the user append + save point 1, `history = conversation.slice(0, -1)`), `finalizeFailedTurn` (L3209, phase 120), `UI_STATE` + `setUiState` (the thinking/streaming in-flight gate every action obeys), `lastBrainWrap`.
Server side (read-only for this phase): `app/schemas.py` `ChatRequest.message` (`max_length=4000`, L77) and `ChatMessage.text` (`max_length=32_000`, L1264, `extra="forbid"`), `app/api/chats.py` (the raw `bor.chat.v1` `messages` JSONB round-trip — logic untouched), `app/api/chat.py` L705–733 (the phase-114 terminal `error` frame `detail="Question too long — trim it and re-ask."` + its `hint` — deterministic, no retry frames: the exact turn this phase gives a remedy for).
CSS: `frontend/assets/styles.css` `.msg-meta.msg-actions` (L795), `.tune-btn` (L980), `.retry-btn` (L1010, the pill family), `.tune-form` + `label` + `textarea` + `.tune-form-actions` (L1116–1141), the mobile `min-height: 44px` action rules (L5708–5709).
## Objective
The last question the user sent carries an **Edit** action: click it, the bubble becomes a textarea prefilled with the question, trim it, Save — the answer is re-asked from the edited text. A "Question too long — trim it and re-ask" turn (or any mistyped / half-formed question) is fixed by editing, never by retyping.
## Dependencies
- `158_thinking_timeline` (complete) — pipeline predecessor (execution order) only.
- Code dependencies (all complete): phase 49/53 redo-in-place retry (`retryLastTurn` + `runTurn({reask:true})`), phase 120 failed-turn records (`appendFailedNote`, `finalizeFailedTurn` — the driver case leaves a failed brain record after the question), phase 14 persistence (`conversation` / `saveConversation`), phase 55 auto-save (`persistConversation`), phase 130 `.msg-actions` row, phase 114 the "question too long" error frame, phase 123 the user-bubble image (`m.image`).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Target set (LOCKED, owner 2026-10-05): exactly ONE editable bubble** — the most recent user message. `markLastEditable()` mirrors `markLastRetryable()`: remove every `.edit-btn` first, then append to the last user bubble, but only when the last **user** record exists in `conversation` (scan newest-first for `who === "user"`; the records after it are the answers it produced — that index is `editIdx`). The bubble itself is tracked in a new `lastUserWrap` module variable (the exact mirror of `lastBrainWrap`, assigned in `runTurn`'s fresh-send `addMessage("user", …)` and in `renderStoredMessage`'s user branch, cleared by `startNewChat()`). Earlier bubbles are never editable, so no mid-session branch exists and the "what happens to the turns after it" question never arises for an old message.
- **Editing happens IN the bubble (LOCKED, owner 2026-10-05)** — `openEditor(wrap, idx)` is the `openTuneForm` family: `form.edit-form` in the bubble's `.msg-body`, `<label for="edit-input-N">`, `textarea[maxlength="4000"]` prefilled from `conversation[idx].text`, Save (submit) + Cancel, a `role=alert` `.edit-error` line. The rendered `.bubble` is hidden (`.is-editing`) while the form is open; Cancel / `Escape` restores it and returns focus to the toggle button; `Ctrl/Cmd+Enter` submits; only one editor open at a time.
- **Save = replace + truncate + re-ask (LOCKED)** — the edited text replaces `conversation[editIdx].text` in place, **every record after it is dropped** (the answers to the old text are stale by construction), `saveConversation()` + `persistConversation()` persist the new shape, the dropped bubbles are removed from the DOM, `lastBrainWrap = null`, `markLastRetryable()` re-targets, then `runTurn(newText, { reask: true })`. Because the edited question ends up as the LAST record, `runTurn`'s `history = conversation.slice(0, -1)` invariant and `reask` (skip the user append + save point 1) hold **unchanged** — this phase adds no API, no schema field, no migration (**A10 stateless holds**).
- **Empty guard:** a whitespace-only save is rejected in-form (the `.edit-error` line, the form stays open) — `ChatRequest.message` / `ChatMessage.text` both require `min_length=1`; the >4000 case is unreachable because the textarea carries `maxlength`.
- **The image survives, the re-ask is text-only** (the phase-49/123 A7 rule): the record keeps its `image` key and its bubble keeps the attachment; `runTurn(newText, { reask: true })` sends no `image`, exactly like `retryLastTurn`.
- **In-flight discipline:** `markLastEditable()` no-ops while `uiState` is `thinking`/`streaming` (the same guard `retryLastTurn` opens with), and `saveEditor` re-checks it — a save can never start a second turn.
- **NOT touched:** `runTurn` (its `reask` path is consumed as-is), `retryLastTurn`, `handleSend`, `rememberBrainTurn`, `trimToBudget`/`persistConversation` (called, not changed), `app/` server code, the SSE contract (A15), the shared-chat page.
## Tasks
1. `01_edit_affordance_on_user_bubbles.md` — the Edit button (`appendEditButton` / `markLastEditable`) + the inline editor (`openEditor`, Save updates the record + the bubble text + localStorage only).
2. `02_resend_in_place.md` — Save also truncates the records after the edited question (DOM + `conversation`) and re-asks through `runTurn({reask:true})`; the in-flight and empty guards.
3. `03_editable_error_turn.md` — the driver case: the question stays editable when its turn failed (the "Question too long" frame, the phase-120 failed record, a stopped answer) and after a refresh.
4. `04_edit_resend_tests.md` — unit source-pins + the `test_chats_api` edited-record round-trip + the isolated `tests/e2e/test_edit_and_resend.py`.
## Testing & Quality
- Unit (house style = source pins on `frontend/assets/app.js`, e.g. `tests/unit/test_frontend_failed_turn.py`): the `.edit-btn` join-the-actions-row rule, `markLastEditable`'s remove-then-append single target, the inline form's labelled textarea + `maxlength=4000` + Cancel/`Escape`/`Ctrl+Enter` paths, the replace-and-truncate record edit, the `runTurn(text, { reask: true })` call, the in-flight no-op, and `retryLastTurn`/`runTurn` byte-unchanged.
- Integration: `tests/integration/test_chats_api.py` (extend) — a `PUT /api/chats/<id>` whose last user record carries the edited text and whose later records were dropped round-trips byte-identically (no schema change is required; this pins the claim).
- Coverage: **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
- E2E (mandatory, A16): `tests/e2e/test_edit_and_resend.py`, run **in isolation** (`uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov`).
## Completion Criteria
- [ ] The most recent user bubble carries Edit; opening it shows a textarea prefilled with the question; Cancel/`Escape` restores the original bubble text.
- [ ] Saving trimmed text updates the bubble + the `bor.chat.v1` record in place, drops every turn after the question (DOM + storage + the saved chat), and streams a fresh answer from the edited text with no duplicated question.
- [ ] A refresh after an edit shows the edited question and the new answer (no stale tail resurrected).
- [ ] The "Question too long — trim it and re-ask" turn's question is editable, so trimming + Save re-asks successfully.
- [ ] Retry, Stop, the failed-turn path, and every successful turn behave byte-identically to before; earlier user bubbles show no Edit button.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation (DB up); regressions green in isolation: `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
## Locked decisions
- **Owner-locked (2026-10-05, roadmap confirmation): last question only** — one editable bubble, managed by the `markLastRetryable()` single-target discipline. Mid-conversation branching is explicitly out of scope.
- **Owner-locked (2026-10-05): inline edit in the bubble** — the `.tune-form` family, not the composer (the composer keeps its own draft untouched by editing).
- **Save truncates the tail and re-asks** (the phase-49 redo-in-place rule, one level up: the question is replaced instead of kept verbatim) — no message versioning, no "edited" history trail, one new API-free flow.
- **A10 untouched** — `POST /api/chat` stays stateless; the edited question is just the client's next `message`. No migration, no `ChatMessage` field (the ≤4000 / ≤32 000 caps already cover it).
- **Edit is available to every visitor who can chat** — no admin gate, the phase-49 Retry precedent.
- **A16/A17 honoured** — one E2E file, one atomic commit.
## Commit
```bash
git add -A .agents/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(chat): edit the last question and re-ask it — fixes trim-and-re-ask with no way to trim"
```
@@ -0,0 +1,37 @@
# Task 01 — Edit button + inline editor on the last question
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
**Story:** n/a
## Objective
The most recent user bubble carries an **Edit** action that swaps the bubble for a textarea prefilled with the question; Save writes the new text into the bubble and the `bor.chat.v1` record, Cancel restores the original. (The re-ask itself lands in task 02 — this task ships shippable: an editable question that persists.)
## Work
1. `frontend/assets/app.js` — add `EDIT_ICON` beside `RETRY_ICON` (L606) and `TUNE_ICON` (L475): an inline `aria-hidden` pencil SVG in the same stroke family (`fill="none" stroke="currentColor" stroke-width="1.7"`, 24×24 viewBox) — no icon font, no CDN (A11).
2. `frontend/assets/app.js` — `appendEditButton(wrap)`, directly after `appendRetryButton` (L617) and shaped exactly like it: `getActionsRow(wrap)` → bail if the row is missing → one-per-bubble guard on `.edit-btn` → `btn.type = "button"`, `btn.className = "edit-btn"`, `role="listitem"` when the row reports `role="list"`, `btn.innerHTML = EDIT_ICON + "<span>Edit</span>"`, click → `openEditor(wrap, btn)`. A visible "Edit" label carries the accessible name (no ellipsis clipping needed at this width).
3. `frontend/assets/app.js` — `markLastEditable()`, the sibling of `markLastRetryable()` (L650): `messagesEl.querySelectorAll(".edit-btn").forEach(b => b.remove())` FIRST (an older bubble's button is stale the moment a newer question lands), then bail when `uiState === UI_STATE.thinking || uiState === UI_STATE.streaming` (the `retryLastTurn` in-flight guard, L2985), then scan `conversation` newest-first for `who === "user"` through a small helper `lastUserRecordIndex()` (returns `-1` when the conversation holds no user record); bail on `-1`; append the button on `lastUserWrap`. Task 02 reuses `lastUserRecordIndex()` for the truncation.
- ASSUMPTION: the last-user-record index comes from that newest-first `conversation` scan (the `markLastRetryable` last-brain-index rule, one `who` over), NOT from a DOM↔record index map — the DOM and the record list are parallel today but re-deriving the index from the record keeps the editor correct after a task-02 truncation.
- The wrap is tracked in a new module variable `lastUserWrap`, the exact mirror of `lastBrainWrap`: assigned where a user bubble is created — `runTurn`'s fresh-send `addMessage("user", …)` (L3270) and `renderStoredMessage`'s user branch (L2262, which is what makes the restore path work for free) — cleared by `startNewChat()` (L2886). `markLastEditable()` appends only when `lastUserWrap` exists AND its record really is the last user record (the `editIdx` scan above), so a stale wrap can never carry the button. Task 02 reuses the same variable.
- Call sites — pair every existing `markLastRetryable()` call (there are exactly six: `restoreConversation()` L2415, `restoreSavedChatFromUrl()` L2482, `finalizeFailedTurn()` L3247, the `done` settle L3649, the zero-frame fallback L3734, the stop finalize L3769): call `markLastEditable()` immediately after each. `startNewChat()` (L2886) needs no call: resetting `messagesEl` removes the buttons with the list (the phase-49 reasoning). Task 03 verifies the error-path pairs (L3247 / L3734 / L3769) end-to-end.
4. `frontend/assets/app.js` — `openEditor(wrap, toggleBtn)`, the `openTuneForm` (L866) family, module-level `editSeq` for unique ids (the `tuneSeq` precedent):
- `document.querySelectorAll(".edit-form").forEach(f => f.remove())` — one editor at a time.
- `form.className = "edit-form"`, `form.noValidate = true`; `<label for="edit-input-N">Edit your question</label>`; a `textarea` with `id="edit-input-N"`, `rows="3"`, `maxlength="${MAX_QUESTION_CHARS}"` (L1689 — the 4000 cap stays single-sourced), `value = conversation[editIdx].text` assigned with `.value` (never an HTML string — the house no-injection rule), `placeholder` reused from the composer if one exists.
- `.edit-form-actions` with a `submit` `.edit-save` ("Save") and a `type="button"` `.edit-cancel` ("Cancel"); a `<p class="edit-error" role="alert" hidden>` for the empty-text rejection.
- While open: `wrap.querySelector(".bubble")` gets `.is-editing` (CSS-hides the rendered question so the question appears ONCE); on close the class is removed.
- Keys: `Escape` in the textarea → cancel; `Ctrl/Cmd+Enter` → submit; cancel restores the bubble, removes the form, returns focus to `toggleBtn` (the `openTuneForm` cancel rule).
- Submit (`e.preventDefault()`): re-check the in-flight guard (no-op), `const text = textarea.value.trim()`; empty → set the `.edit-error` copy ("Ask something first — the question can't be empty."), unhide, keep the form open, return; otherwise this task's scope: `conversation[editIdx].text = text`, `saveConversation()`, re-render the bubble text through `renderMarkdown(text)` (the `addMessage` renderer, so markdown rules stay in one place), close the editor, announce through the existing `#steering-announcer`-style pattern only if a generic announcer already exists — otherwise the visible re-render is the feedback (AGENTS.md rule 10 is satisfied by the bubble change; do NOT invent a new live region).
- Leave a clearly-marked `// task 02: truncate the tail + re-ask` seam where the close happens, so the next task adds the resend without restructuring.
5. `frontend/assets/styles.css` — `.edit-btn` in the `.retry-btn` family (L1010: same pill size/spacing, the global `:focus-visible` ring applies — verify it is not scoped to `.tune-btn`/`.retry-btn` only and extend the selector list if it is), `.edit-btn:hover` in the same tint; `.msg.user .bubble.is-editing { display: none; }`; the `.edit-form` card, label, textarea and actions row cloned from the `.tune-form` block (L1116–1141) and `.edit-error` cloned from `.tune-error`; add `.edit-btn { min-height: 44px; }` beside `.tune-btn`/`.retry-btn` in the mobile block (L5708–5709) and honor `prefers-reduced-motion` if the family transitions anything.
6. `frontend/index.html` — no change (the chat shell, the `.messages` live region and the composer already exist). Confirm no new `<script>`/`<link>` is introduced (AGENTS.md rule 6, the no-CDN test).
## Testing & Quality
- Unit: `tests/unit/test_frontend_edit_message.py` (new) — house-style source assertions: `.edit-btn` is appended through `getActionsRow` and joins `role=list` as `listitem`; `markLastEditable` removes before appending and bails on `UI_STATE.thinking`/`streaming`; `openEditor` renders a `<label for>` + `textarea[maxlength]` bound to `MAX_QUESTION_CHARS` and prefills from `conversation`; `Escape`/Cancel restore, `Ctrl/Cmd+Enter` submits; an empty trim shows `.edit-error` and keeps the form; the bubble text is re-rendered with `renderMarkdown`; `retryLastTurn` and `runTurn` are byte-unchanged (read the file and pin the two function bodies' current text).
- Coverage: **>90%** on `app/` — unchanged (frontend-only task).
- Manual check (visible browser or `page.goto` in a scratch Playwright run): send a question, Edit → Cancel leaves the original text; Edit → Save updates the bubble and survives a reload.
## Completion Criteria
- [ ] `grep -n "edit-btn\|edit-form\|markLastEditable" frontend/assets/app.js frontend/assets/styles.css` shows the button, the form and the CSS family.
- [ ] Exactly one bubble (the newest question) shows Edit at rest; after a new send the button moved to the new bubble.
- [ ] Save updates the bubble text and the `bor.chat.v1` localStorage record; a reload shows the edited question.
- [ ] Cancel / `Escape` leave the question byte-identical; an empty trim is refused in-form.
- [ ] `uv run pytest` green; coverage TOTAL unchanged (>90%); `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,35 @@
# Task 02 — Save re-asks: replace the question, drop the stale tail, stream the new answer
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
**Story:** n/a
## Objective
Saving an edited question **resends** it: the answers that followed the old text are removed (DOM + `conversation` + the saved chat), the edited question becomes the last record, and a fresh answer streams from it — the phase-49 redo-in-place semantics one level up (the question is replaced instead of kept verbatim).
## Work
1. `frontend/assets/app.js` — in `openEditor`'s submit handler (task 01's seam), after the record text is written and BEFORE the editor closes:
- `const editIdx = lastUserRecordIndex()` (the newest-first `who === "user"` scan from task 01) and `conversation.splice(editIdx + 1)` — every record the OLD question produced is dropped outright (never rewritten, the phase-114 whole-turn rule).
- Remove the matching DOM bubbles: every `.msg` after `lastUserWrap` in `messagesEl` (`let n = lastUserWrap.nextSibling; while (n) { const next = n.nextSibling; n.remove(); n = next; }`).
- `lastBrainWrap = null` (the removed answer is no longer a retry target) and clear `leavePartialIndex` / the phase-73 pagehide partial pointer if it points past `editIdx` (a partial saved from the dropped turn must not be re-persisted into the new shape — read `runTurn`'s `leavePartialIndex` usage before deciding the exact reset; `= -1` is the safe value).
- `saveConversation()` (localStorage, `trimToBudget` rides it unchanged) + `persistConversation()` (the phase-55 auto-save pushes the truncated shape to `PUT /api/chats/<id>` — fire-and-forget, never blocks the turn).
- `markLastRetryable()` — the Retry button re-targets the (now removed) tail: after the drop there is no brain record, so the call simply clears every `.retry-btn`.
2. `frontend/assets/app.js` — same handler, then `closeEditor()` and `return runTurn(text, { reask: true })`:
- `reask: true` is the whole point — `runTurn` (L3260) skips the user-bubble append and save point 1, so the question is never duplicated; `history = conversation.slice(0, -1)` (L3340) is exactly the prior turns because the edited question is the LAST record.
- Return the promise (the `retryLastTurn` L2984/`regenerateStaleChat` convention — `runTurn` never rejects; a failure lands in the phase-120 failed-turn path).
- No `image` is passed: a re-ask sends text only (the phase-49/123 locked A7 rule); the record keeps its `image` key and the bubble keeps its attachment.
3. `frontend/assets/app.js` — guards, in this order at the top of the submit handler: in-flight (`uiState === UI_STATE.thinking || UI_STATE.streaming` → no-op, a save can never start a second turn), then the empty trim (`Ask something first — the question can't be empty.` in `.edit-error`, form stays open). Re-check the in-flight state a second time immediately before `runTurn` (the same double-check `retryLastTurn` gets by being a single synchronous stretch — keep the handler synchronous from the guard to the `runTurn` call except for nothing: no `await` between the guard and the call).
4. `frontend/assets/app.js` — keep the composer out of it: editing never touches `input.value`, `autoGrow()` or `updateCharCount()` (the user's unsent draft survives an edit), and `MAX_QUESTION_CHARS` stays the textarea's `maxlength` so a >4000 trim is unreachable server-side (A10: the server's `ChatRequest.message max_length=4000` remains the real bound).
5. No server change: `app/api/chat.py` sees an ordinary `POST /api/chat` with the edited `message` and a shorter `history`; `app/api/chats.py` stores the shorter `messages` array through the existing `SavedChatUpdate` path (no new field, no migration).
## Testing & Quality
- Unit: `tests/unit/test_frontend_edit_message.py` (extend) — the splice is `conversation.splice(editIdx + 1)` (the whole tail, nothing before it), the DOM removal starts at `lastUserWrap.nextSibling`, `lastBrainWrap = null` precedes the re-ask, `saveConversation()` + `persistConversation()` both run before `runTurn`, and the call is exactly `runTurn(text, { reask: true })` with no `image`; the in-flight guard precedes the record edit; `retryLastTurn` stays byte-unchanged.
- Integration: `tests/integration/test_chats_api.py` (extend) — a `PUT /api/chats/<id>` carrying the edited last user record and a truncated `messages` list round-trips byte-identically (pins "no schema change is needed"; a shortened array is already legal).
- Coverage: **>90%** on `app/` (no `app/` code changes — the gate must hold).
- Manual: ask a question, get an answer, Edit → change one word → Save: the old answer disappears, the new answer streams in its place, the question appears once.
## Completion Criteria
- [ ] Save removes every turn after the edited question (DOM + `bor.chat.v1` + the saved-chat row) and streams exactly one fresh answer from the edited text.
- [ ] The question is never duplicated in the DOM, in localStorage, or in the request `history`.
- [ ] Retry on the new answer, Stop during the new turn, and "New chat" all behave as before.
- [ ] Editing while a turn is in flight is impossible (no Edit button, and a queued Save no-ops).
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 03 — The driver case: a failed or rejected question stays editable
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
**Story:** n/a
## Objective
The exact turn the owner hit — the terminal `error` frame `Question too long — trim it and re-ask.` — leaves its question **editable**, so trimming + Save re-asks successfully; the same holds for every phase-120 failed turn, a stopped partial, and after a page refresh.
## Work
1. `frontend/assets/app.js` — verify and, where missing, add the `markLastEditable()` pairing on the paths that settle WITHOUT a `done` frame, so the question bubble always ends up with its Edit button once the turn is terminal:
- the `runTurn` non-OK-HTTP path (L3359: a 422 from `ChatRequest` — e.g. `message` over 4000 — throws with the server `detail` and lands in the error catch) and the SSE `error` frame path (the phase-114 "Question too long" frame arrives as an `error` event, L3675/L3806) — both settle through the error catch → `finalizeFailedTurn` (L3209), which already calls `markLastRetryable()` at L3247: `markLastEditable()` must be the very next statement;
- the zero-frame fallback (L3734) and the phase-48 stop finalize (L3769) — same pairing;
- the 300s guard abort (`armTurnTimeout`, L3283 → `setUiState(UI_STATE.error, …)`) — confirm the catch's `aborted` branch reaches a `markLast*()` call; if it settles without one, call `markLastRetryable()` + `markLastEditable()` there too (a timed-out turn must not strand the question).
2. `frontend/assets/app.js` — the error state must not lock the editor: `markLastEditable()`'s guard bails only on `UI_STATE.thinking` / `UI_STATE.streaming`, never on `UI_STATE.error` / `UI_STATE.idle` (a failed turn IS idle-or-error at rest — read `setUiState` and pin the two states the guard checks). `saveEditor`'s in-flight guard mirrors it exactly.
3. `frontend/assets/app.js` — the failed answer record is part of the stale tail: after an edit-save the phase-120 `{ who: "brain", failed: true, error }` record is dropped by task 02's `conversation.splice(editIdx + 1)` like any other answer, and its bubble (with `.failed-note`) is removed with it. Confirm `lastBrainWrap = null` happens on that path so no stale Retry target survives, and that `rememberBrainTurn`'s internal `trimToBudget`-driven index shift (its `brainIdx -= trimDrop` bookkeeping, L2861) can never be confused by the truncation (the truncation happens before the new turn starts, so the new turn's `rememberBrainTurn` sees the post-splice array — add the one-line comment saying so).
4. `frontend/assets/app.js` — the restore paths: a reload after a failed turn renders the failed bubble (phase 120) and the question above it; `renderStoredMessage`'s user branch sets `lastUserWrap` (task 01) and the paired `markLastEditable()` after `markLastRetryable()` (L2415 / L2482) puts Edit back on that question. Verify the shared-chat page (`?chat=` admin restore) does NOT get the button for a read-only view — ASSUMPTION: the `?chat=` restore is an editable working copy exactly like the local restore (it already is Retry-able, so Edit follows); if the code shows a genuinely read-only render path, gate on it and record the finding here.
5. `frontend/assets/app.js` — the banner: `showErrorBanner(detail, opts)` (L2918) is NOT changed (the phase-120 rule — its `opts.retryable && lastBrainWrap` condition stays byte-identical). Edit is an in-bubble affordance beside Retry, never a banner control.
## Testing & Quality
- Unit: `tests/unit/test_frontend_edit_message.py` (extend) — `markLastEditable()` follows every `markLastRetryable()` call site (assert the six pairings by counting both calls), the guard names only `UI_STATE.thinking`/`UI_STATE.streaming` (no `UI_STATE.error`), `showErrorBanner` is byte-unchanged, and the failed-turn splice claim (a `failed: true` record after the edited question is dropped, not kept).
- Integration: none new (the phase-120 failed-record round-trip in `tests/integration/test_chats_api.py` already covers the shape this task reads).
- Coverage: **>90%** on `app/` (unchanged — no `app/` change).
- Manual: with the mock LLM's embed-failure trigger (`tests/e2e/mock_llm.py` `EMBED_FAIL_TRIGGER`, L1321) or a >4000-char question, confirm the error frame's question is editable and a trimmed Save answers successfully.
## Completion Criteria
- [ ] After the "Question too long — trim it and re-ask." error: the question bubble shows Edit, the failed bubble shows Retry, trimming + Save produces a real answer.
- [ ] The same holds for a network-error turn, a mid-stream drop, a stopped answer, and a 300s timeout.
- [ ] A page refresh after any of the above restores the question with a working Edit.
- [ ] `showErrorBanner` and the phase-120 failed-record shape are byte-unchanged.
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,38 @@
# Task 04 — Tests for edit-and-resend (unit + integration + isolated E2E)
**Phase:** `159_edit_and_resend_message` · **Source:** `TODO.md:3` — "Need to be able to edit and resend a message. Got 'FailedQuestion too long — trim it and re-ask' but I can't trim it because I can't edit it."
**Story:** n/a
## Objective
The phase's contract is pinned: one editable bubble, the inline editor, replace-and-truncate-and-re-ask, the editable failed turn, and byte-identical Retry/Stop/persistence behavior — with the phase's own Playwright file run in isolation (AGENTS.md rule 9).
## Work
1. `tests/unit/test_frontend_edit_message.py` (new) — house-style source pins against `frontend/assets/app.js` (+ `styles.css`), one focused assertion cluster per behavior:
- `EDIT_ICON` exists and is an `aria-hidden` inline SVG; `.edit-btn` is created in `appendEditButton` through `getActionsRow` with the one-per-bubble guard and the `role=list` → `listitem` rule;
- `markLastEditable()` removes `.edit-btn` before appending, bails on `UI_STATE.thinking`/`UI_STATE.streaming` (and NOT on `UI_STATE.error`), uses `lastUserRecordIndex()` + `lastUserWrap`, and is called after all six `markLastRetryable()` call sites;
- `openEditor`: `<label for>` + `textarea` + `maxlength` bound to `MAX_QUESTION_CHARS`, prefilled from `conversation`, `.edit-form` uniqueness, `Escape`/Cancel restore, `Ctrl/Cmd+Enter` submit, the empty-trim `.edit-error` branch;
- save path order: guard → `conversation[editIdx].text = text` → `conversation.splice(editIdx + 1)` → DOM removal from `lastUserWrap.nextSibling` → `lastBrainWrap = null` → `saveConversation()` → `persistConversation()` → `runTurn(text, { reask: true })` (no `image` key);
- regressions pinned as byte-unchanged: `retryLastTurn`, `runTurn`, `showErrorBanner`, `handleSend`, `rememberBrainTurn`;
- `styles.css` carries `.edit-btn`, `.edit-form`, `.edit-error`, `.is-editing` and the 44px mobile rule.
2. `tests/integration/test_chats_api.py` (extend) — two cases: (a) a `PUT /api/chats/<id>` whose `messages` array is the post-edit shape (edited last user record, later records dropped) round-trips byte-identically on `GET`; (b) the same shape on a shared chat still serves through `SharedChatOut.messages` — pins "the edited/truncated conversation needs no schema change".
3. `tests/e2e/test_edit_and_resend.py` (new) — mock-LLM suite on the `app_server`/`page`/`db_ready` fixtures, the phase-120 file as the template (TRUNCATE-then-seed `saved_chats` in an autouse fixture; `auth_helpers.login` for the authed chat; seed `tests/fixtures/docs/homelab/kubernetes.md` as `docs`/`homelab/kubernetes.md` so retrieval grounds and the mock answers with `MOCK_ANSWER_MARKER`):
1. `test_edit_last_question_reasks_in_place` — send "How is my Kubernetes cluster set up?" → grounded answer → the user bubble has Edit and the OLD brain bubble count is 1 → click Edit → the textarea is prefilled with the exact question → append " in one line" → Save: exactly ONE user bubble (the edited text), the old answer bubble is gone, the new answer streams with the mock marker, `bor.chat.v1` holds `[…user(edited), brain]` with no duplicate question, and `saved_chats.messages` matches (poll the row).
2. `test_only_the_last_question_is_editable` — two turns: exactly one `.edit-btn` exists and it is on the SECOND user bubble; the first user bubble has none.
3. `test_edit_cancel_is_a_no_op` — open the editor, press `Escape`: the bubble text, the record, and the answer are unchanged (byte-identical `localStorage` payload before/after); the button returns focus (assert `document.activeElement` is the Edit toggle).
4. `test_edit_drops_a_stale_tail_and_survives_refresh` — after an edit-save, `page.reload()`: the edited question + the new answer restore, the dropped answer never reappears, and Edit is back on the question.
5. `test_failed_turn_question_is_editable_and_trim_reasks` — the driver case: send a question carrying the mock's terminal-failure trigger (`ALWAYS_FAIL_TRIGGER`, "always fail") so the turn settles with an `error` frame and a failed bubble (the phase-114 "question too long" frame takes the identical settle path — see the docstring note: its exact detail string is real-endpoint-only), assert Edit exists on the question beside the failed bubble's Retry, then Edit → Save with the trigger stripped out and watch the grounded answer stream.
6. `test_edit_is_inert_while_the_turn_runs` — start a long answer (`LONG_ANSWER_TRIGGER`, "write a long answer"), assert no `.edit-btn` while streaming, wait for `done`, assert Edit lands.
7. negative: `test_composer_draft_survives_an_edit` — type an unsent draft in the composer, edit + save the last question, the draft text is still in `#composer`'s input.
- MOCK-ONLY: the suite sets no `E2E_REAL_LLM`; the file's docstring states the trigger names, the seed, and the per-test contract mapping (the phase-120 file's shape).
4. `tests/e2e/test_edit_resend.py` is NOT created (one file per phase, A16) — regression runs stay in the phase's final gate: `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`, `test_stop_generation.py`, `test_smoke.py`, each in isolation.
## Testing & Quality
- Run in this order: `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` → `uv run pytest` (full unit + integration) → `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%) → `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` (DB up) → the regression files above, isolated.
- No `app/` code changes in this phase: coverage must not move; if a `tests/`-only change drops the TOTAL below 90%, the gate failure is a phase bug (investigate, do not add filler tests).
## Completion Criteria
- [ ] `uv run pytest` green (unit + integration) with TOTAL coverage >90%.
- [ ] `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation.
- [ ] Every regression E2E file listed above green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] The phase dir moves to `.agents/phases/complete/` with the single `--no-gpg-sign` commit (`00_phase.md` §Commit).
@@ -0,0 +1,64 @@
# Phase 160 — The agent knows its tool-call budget (and never dumps tool calls into the answer)
**Source:** `TODO.md` L4 — "The LLM need to know how many tool calls it has left. During long research sessions it will sometimes end the conversation with tool calls that just get dumped into chat because it ran out of turns. Every tool call should also return 'you have x tool calls remaining' or some other helpful message and then on the last one attempted tool calls should be rejected rather than spit out to chat and the LLM should be encouraged to come up with a reply or tell the user to continue the session by allowing more tool calls (or something similar)."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-10-05)
**Context:** `app/rag/agent.py` (2,322 lines) — `run_agent` (L1995): `max_rounds = settings.agent_max_rounds` (L2103), `tools = agent_tools(settings) if max_rounds > 0 else None` (L2104), `rounds = 0` (L2106), the round loop (L2107) whose pieces are all yielded verbatim (L2136-2142: `if isinstance(piece, ToolCallPiece): calls.append(piece)` … `yield piece`), `call = calls[0]` (L2224 — "a stream can carry several calls; run the first"), `result = _execute_tool(tool_db, call, seed_docs, holder, settings)` (L2227, a plain string), `rounds += 1` (L2229), the `agent tool=%s args=%s round=%d/%d` INFO line (L2230), `messages.append({role: assistant, tool_calls: […]})` + `messages.append({"role": "tool", "tool_call_id": …, "content": result})` (L2237-2251), the phase-95 `ToolResultPiece` yields (L2258), and the **round-cap block** (L2270-2322): the `agent round cap reached (rounds=%d) — forcing a final no-tools answer` warning, the forced `chat_stream_retried(llm, messages, tools=None, …, scaffolding=final_filter)` whose loop yields EVERY piece (`ToolCallPiece` included), and the phase-71 terminal `MalformedReplyError` when the forced final carried no visible content but stripped scaffolding.
Refusal/teaching constants (the house pattern for teaching the model in-band): `ALREADY_IN_CONTEXT` (L537), `UNKNOWN_TOOL` (L541), `READ_TRUNCATION_NOTICE` (L578), `REREAD_PARTIAL` (L628), `CORRECTION_INSTRUCTION` (L732, folded into the single system message as `system_prompt + "\n" + CORRECTION_INSTRUCTION` at L2178 / L1044 in `app/api/chat.py`). `AgentHolder` (L1482: `read_docs`, `tool_calls`, `scaffold_stripped`, `read_truncations`, `shown_images`, `read_coverage` — the API layer reads it after the stream).
`app/config.py` `agent_max_rounds: int = 10` (+ its loud validator, ~L229-240 / L694 pattern), `.env.example` L69 (`# BOR_AGENT_MAX_ROUNDS=10 … (0 = no tools)`).
`app/api/chat.py` — the grounded vs deflected dispatch (L840-900), `tools=None` for the deflected turn (L870) and the kill switch, `_pump(pieces)` (L893, the ONE piece→SSE loop: `ToolCallPiece` → `tool` frame at L909, `ToolResultPiece` → `tool_result`, `thinking`/`delta`), the per-turn log line (L1196-1219: `… thinking_chars=%d tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d`), the `ChatDoneEvent` yield (~L1265).
`app/schemas.py` — `ChatDoneEvent` (L316: `type`/`deflected`/`sources`/`related`/`suggestions`/`answer_image` + the `@model_serializer(mode="wrap")` omission rule that drops `answer_image` when `None` — "ABSENT, never null"), `ChatErrorEvent.hint` (the phase-114 additive precedent), `ChatMessage` (L1264ff, `extra="forbid"`, the optional `failed`/`error`/`stopped` markers).
`app/rag/scaffolding.py` — `SCAFFOLD_PATTERNS` (the observed-forms registry: `<|tool_call_start|>…<|tool_call_end|>`, `<|tool_calls|>`, `<|tool_call|>`) + `ScaffoldingFilter` (content only, never thinking).
Tests: `tests/unit/test_agent.py` (the always-calling fake LLM: `_run(llm, holder, _settings(agent_max_rounds=N))`, the round-cap cluster L4561-4944, the forced-final scaffolding case L5241-5255 asserting `MalformedReplyError`, `llm.requests[i][1] is None` as the "no tools offered" pin), `tests/integration/test_chat_api.py`, `tests/e2e/mock_llm.py` (`TOOLS_TRIGGER` L1114 "use your tools", `MULTI_READ_TRIGGER` L1123 "read two documents", `SCAFFOLD_TRIGGER` L1375 "emit raw tool markup", `LONG_ANSWER_TRIGGER` L1035), `tests/e2e/test_agent_unlimited_tools.py` (the multi-read flow + the TRUNCATE-then-seed KB fixture).
PLAN §4 (SSE contract), §9 (the required per-turn line), A15 (event types `thinking, delta, tool, retry, done, error, tool_result`), A10 (`POST /api/chat` stateless), A16/A17.
## Objective
Every tool result tells the model how many tool calls it has left; when the budget is gone a further attempted call is **rejected instead of reaching the chat** (no tool lines, no raw markup, never an empty answer); the forced final is explicitly coached to answer with what it has or tell the user it can keep going; and the user gets a one-click **Continue researching** action that runs a fresh turn with a whole new budget. `BOR_AGENT_MAX_ROUNDS` stays 10 (owner-locked).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Telemetry is model-facing (LOCKED):** the remaining count rides the tool-result string the loop already appends as `{"role": "tool", …}` — no new SSE event type, A15 untouched. Trailer constants live with the other teaching strings in `app/rag/agent.py`:
`TOOL_BUDGET_REMAINING = "\n\n[tool calls remaining: {n}]"` for `n >= 2`, `… 1 call remaining` wording at `n == 1`, and `TOOL_BUDGET_SPENT = "\n\n[No tool calls remain — answer now from what you have, or tell the user to say \"continue\" to keep researching.]"` at `n == 0`. Appended in `run_agent` (NOT inside `_execute_tool`, which stays a pure `(db, call, seed_docs, holder, settings) → str` seam and keeps its existing unit tests byte-identical), after `rounds += 1`, so a truncated read's `ToolResultPiece` and the `read_coverage` bookkeeping are unaffected.
- **`agent_max_rounds = 0` (the kill switch) is untouched:** no tools are offered, so no tool result and no trailer exist — the single `tools=None` request stays byte-identical (the phase-45 pin, `tests/unit/test_agent.py::test_zero_max_rounds_is_one_request_without_tools`).
- **The bug class this phase kills:** a request made WITHOUT tools can still come back with structured `tool_calls` deltas. Today those `ToolCallPiece`s become SSE `tool` frames (`app/api/chat.py::_pump`, L909) and render as tool lines in the chat — a round-capped or deflected turn "ending with tool calls dumped into chat", exactly the owner's report. New invariant: **a request with `tools=None` never produces a `tool` frame, and never leaves the answer empty.**
- **The cap path (task 02):** the forced final's loop stops yielding `ToolCallPiece`s (they are REJECTED: not run, not shown, not counted in `holder.tool_calls` — the refusal style: "counts in nothing"), counts them in a new `AgentHolder.rejected_calls`, logs one warning (`agent: %d tool call(s) attempted after the round cap — rejected, not shown`), and — when the forced final streamed no visible content — yields the deterministic `ROUND_CAP_FALLBACK_ANSWER` content piece (an honest "I've used up this turn's research budget — here's where I got to; say continue to dig further" line) instead of dying. The model is coached (next bullet) to produce the "say continue to keep going" line itself; `ROUND_CAP_FALLBACK_ANSWER` is the deterministic backstop for the case where it emits only calls or only scaffolding.
- **Coaching the forced final (task 02):** `ROUND_CAP_WRAPUP_INSTRUCTION` (a module constant beside `CORRECTION_INSTRUCTION`) is folded into the single system message for the forced final — the exact L2178 pattern (`system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION`) — spelling out: no tools remain, NEVER emit tool-call markup or a tool call, answer from the results you already have, and if the answer is incomplete say so and tell the user they can say "continue" to keep researching.
- **The same rule for the other tool-less requests (task 02):** `app/api/chat.py::_pump` drops `ToolCallPiece`s while the request had no tools — the deflected turn (A8 `DEFLECT_MODE`, L870) and the kill-switch turn — so the phase-71 incident class is closed at the transport too. The `ScaffoldingFilter` keeps stripping textual markup forms; this task only stops STRUCTURED calls from becoming UI lines.
- **User-visible signal (tasks 03/04):** one additive `done` key `budget_exhausted: bool` (serialized ABSENT when false, the `answer_image` omission precedent — every pre-phase frame stays byte-identical), driven by `AgentHolder.cap_reached`, plus the per-turn log fields `rounds=%d` and `cap_hit=%d`. The frontend answers it with a `Continue researching` button that sends a plain next turn (`runTurn("continue")`): the budget is per-turn, so a new turn has `agent_max_rounds` again — **no client-supplied budget in the request body** (a client-tunable round cap is a DoS surface and would break the server-side kill switch, A10-adjacent).
- **NOT touched:** `BOR_AGENT_MAX_ROUNDS`'s default and validator, the `tool` SSE event shape (A15), `done.sources`/`related`/`suggestions`/`answer_image`, the retry primitive (A2/phase 67), `ScaffoldingFilter`'s pattern registry (a new form gets an observed capture, not a guess), the phase-71 recovery path's `MalformedReplyError` (non-cap: a deflected/round turn whose SECOND reply is scaffolding-only still errors).
## Tasks
1. `01_remaining_in_tool_results.md` — the budget trailer on every executed tool result (model-facing), + the docstring/env-comment updates.
2. `02_reject_calls_at_the_cap.md` — no `tool` frames from a `tools=None` request (the forced final, the deflected turn, the kill switch), `AgentHolder.rejected_calls`, the cap-path fallback answer, and `ROUND_CAP_WRAPUP_INSTRUCTION` on the forced final.
3. `03_budget_exhausted_flag_and_log.md` — `AgentHolder.rounds` / `cap_reached`, the additive `done.budget_exhausted` key, the `rounds=` / `cap_hit=` log fields, and the one PLAN.md §9/§4 revision note.
4. `04_continue_researching_button.md` — the UI: the button, the persisted record marker, the restore path, CSS.
5. `05_budget_tests.md` — the mock's round-cap marker + unit + integration + the isolated `tests/e2e/test_tool_budget_telemetry.py`.
## Testing & Quality
- Unit: `tests/unit/test_agent.py` (extend) — the trailer on every tool result (the fake LLM's recorded `messages`: `[n] remaining` decreasing 9, 8, … 1, then the spent line at 0; `agent_max_rounds=1` → the spent line on the only result; `agent_max_rounds=0` → NO tool message at all), the forced final's system message carries `ROUND_CAP_WRAPUP_INSTRUCTION`, a `ToolCallPiece` on the forced final is not yielded and does not reach the API layer (`holder.rejected_calls` counts it, `holder.tool_calls` does not), the cap path with a call-only / scaffolding-only forced final yields `ROUND_CAP_FALLBACK_ANSWER` instead of raising, and the existing cap cluster (`llm.requests[i][1] is None`, the `round cap reached (rounds=N)` warning) still passes.
- Integration: `tests/integration/test_chat_api.py` (extend) — a cap-hitting mock turn streams NO `tool` frame after the cap and ends with `done` carrying `budget_exhausted: true`; a normal grounded turn's `done` frame is byte-identical to today (the key is absent, not false); a deflected turn emits no `tool` frame even when the mock returns structured calls.
- Coverage: **>90%** on `app/` (AGENTS.md rule 9) — the new branches (cap-path rejection, the fallback answer, the serializer omission) each need a test.
- E2E (mandatory, A16): `tests/e2e/test_tool_budget_telemetry.py`, run **in isolation** (`uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov`).
## Completion Criteria
- [ ] A grounded turn's tool messages carry a decreasing `[tool calls remaining: N]` trailer and the last one says the budget is spent (visible in a live turn's server log or a captured request).
- [ ] A turn that hits the cap ends with a readable answer — zero tool lines after the cap, zero raw tool markup in the answer, no empty answer, no `MalformedReplyError` on the cap path.
- [ ] `done.budget_exhausted` is `true` exactly on cap-reached turns and ABSENT (never `false`) everywhere else.
- [ ] The exhausted answer offers **Continue researching**; clicking it sends one new turn and streams a new answer with a fresh tool budget; earlier answers' buttons are gone.
- [ ] A refresh keeps the Continue button on a flagged answer; a shared chat never shows it.
- [ ] `agent_max_rounds=0` still sends exactly one `tools=None` request and no trailer; deflected turns are unchanged; `BOR_AGENT_MAX_ROUNDS` default is still 10.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` green in isolation (DB up); regressions green in isolation: `test_agent_unlimited_tools.py`, `test_agent_document_tools.py`, `test_chat_rag.py`, `test_harness_aligned_tools.py`, `test_grep_regex_teaching.py`, `test_smoke.py`.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/`.
## Locked decisions
- **Owner-locked (2026-10-05): `BOR_AGENT_MAX_ROUNDS` stays 10** — this phase teaches the model and the user about the budget; it does not raise it.
- **Owner-locked (2026-10-05): Continue lives on the answer** — a button, not a live counter in the tool lines (the "you have x remaining" half of the TODO is model-facing).
- **A15 holds — no new SSE event type:** the count rides the tool result text; the user-facing signal is one additive `done` key (the `related`/`answer_image` precedent, PLAN §4's additive rule).
- **The boundary call still runs:** the call the model emitted while budget remained is executed and its result carries `[tool calls remaining: 0]`; only calls attempted AFTER the budget is spent are rejected (never executed, never shown) — no in-flight research is thrown away and the OpenAI wire stays valid.
- **Recorded revision of phase 71's cap-path terminal (AGENTS.md rule 3, owner-confirmed 2026-10-05):** a round-capped forced final that carries no visible content no longer raises `MalformedReplyError` — it yields `ROUND_CAP_FALLBACK_ANSWER`, because an honest "budget spent" line beats an error frame for the exact symptom this TODO reports. The non-cap recovery path (a round or deflected reply that is pure scaffolding twice in a row) keeps `MalformedReplyError` unchanged. The PLAN.md revision note in task 03 records it (the phase-45 pattern: a recorded revision, never a silent deviation).
- **A10 holds** — Continue is an ordinary stateless `POST /api/chat` with the client's history; no session state, no client-controlled budget, no migration.
- **A16/A17 honoured** — one E2E file, one atomic commit.
## Commit
```bash
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(agent): teach the tool-call budget and reject calls past the cap instead of dumping them into chat"
```
@@ -0,0 +1,32 @@
# Task 01 — Every tool result tells the model how many calls remain
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "The LLM need to know how many tool calls it has left… Every tool call should also return 'you have x tool calls remaining' or some other helpful message."
**Story:** n/a
## Objective
The model sees its remaining round budget in every tool result it receives — so it plans its last reads and wraps up on its own instead of hitting the wall mid-research.
## Work
1. `app/rag/agent.py` — add the trailer constants beside the other in-band teaching strings (the `READ_TRUNCATION_NOTICE` L578 / `REREAD_PARTIAL` L628 family, before `CORRECTION_INSTRUCTION` L732):
- `TOOL_BUDGET_REMAINING: str = "\n\n[tool calls remaining: {n}]"` — the `.format(n=…)` call site is the only place that builds it;
- `TOOL_BUDGET_LAST: str = "\n\n[1 tool call remaining — make it count, then answer]"`;
- `TOOL_BUDGET_SPENT: str = "\n\n[No tool calls remain — answer now from what you have, or tell the user to say \"continue\" to keep researching.]"`;
- one small pure helper `def _budget_note(remaining: int) -> str` (n ≥ 2 → the formatted remaining line, 1 → `TOOL_BUDGET_LAST`, ≤ 0 → `TOOL_BUDGET_SPENT`) so the wording lives in exactly one place and the unit tests hit a pure function.
2. `app/rag/agent.py::run_agent` — after `result = _execute_tool(...)` (L2227) and `rounds += 1` (L2229), append the note before the message goes on the wire: `messages.append({"role": "tool", "tool_call_id": call.id, "content": result + _budget_note(max_rounds - rounds)})` (L2251). Keep `_execute_tool` itself unchanged — its return value is the raw tool result and its existing unit tests stay byte-identical.
- The note rides EVERY executed call, including a **rejected** call's refusal text (`UNKNOWN_TOOL`, `ALREADY_IN_CONTEXT`, …) — a refusal consumes a round, so the budget line is exactly as true there.
- A truncated read is unaffected: the phase-95 `ToolResultPiece` yields (L2258) read `holder.read_truncations`, not the result string, so the SSE `tool_result` frame and the UI marker stay byte-identical.
- `agent_max_rounds == 0` never reaches this code (no tools offered, `tools=None` at L2104) — the kill-switch request stays byte-identical.
3. `app/rag/agent.py` — extend the module docstring's loop contract (the L31-60 block, and the phase-45 round-cap paragraph at L214-235): one sentence naming the trailer, its three forms, and the rule "the count is what the model has LEFT after the call it is reading" (so the first result of a 10-round budget reads 9).
4. `.env.example` — extend the `BOR_AGENT_MAX_ROUNDS` comment (L69) to mention the telemetry: `# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools); every tool result carries the model's remaining count (phase 160)`. No new setting, no default change (LOCKED).
## Testing & Quality
- Unit (`tests/unit/test_agent.py`, extend the round-cap cluster at L4561): a 3-round always-calling fake LLM produces three tool messages whose contents end with `[tool calls remaining: 2]`, `[1 tool call remaining…`, `[No tool calls remain…` respectively (assert on the recorded `llm.requests[i][0]` message list — the fake already records the exact messages sent); `agent_max_rounds=1` → the single tool message ends with `TOOL_BUDGET_SPENT`; `agent_max_rounds=0` → the request carries no `role: "tool"` message at all (the existing kill-switch pin, extended with one assert); a rejected call (unknown tool) still ends with its refusal + the note; `_budget_note` boundary table (`2`, `1`, `0`, `-1`).
- The existing cap/retry/scaffolding tests must keep passing unchanged (the trailer is appended content, not a new message).
- Coverage: **>90%** on `app/` — `_budget_note` + the `run_agent` append line fully covered.
## Completion Criteria
- [ ] A live grounded turn's second model request contains a `role: "tool"` message ending in `[tool calls remaining: N]` with `N == agent_max_rounds - rounds_consumed`.
- [ ] The final tool result of a capped turn carries the "no tool calls remain … say \"continue\"" line.
- [ ] No SSE frame, `done` shape, `tool_calls=` log field, `read_coverage` span or `tool_result` frame changed (`uv run pytest tests/integration/test_chat_api.py -v --no-cov` green).
- [ ] `agent_max_rounds=0` still issues exactly one `tools=None` request with no trailer anywhere.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,37 @@
# Task 02 — Past the cap: reject the call, coach the wrap-up, never dump into chat
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…on the last one attempted tool calls should be rejected rather than spit out to chat and the LLM should be encouraged to come up with a reply or tell the user to continue the session by allowing more tool calls (or something similar)."
**Story:** n/a
## Objective
A tool call attempted after the budget is spent is rejected — it never runs, never becomes a `tool` frame, never renders in the chat — the forced final is explicitly coached to answer or hand off, and a cap-path turn always ends with readable text.
## Background (the actual defect — read before coding)
The round-cap path already forces one `tools=None` request (L2270-2322), but that request's pieces are yielded **verbatim**, and `app/api/chat.py::_pump` (L893) turns every `ToolCallPiece` into an SSE `tool` frame (L909). A model that answers a tool-less request with structured `tool_calls` anyway — the documented behavior of the hosted `lite` family, same family as the phase-71 scaffolding incident — therefore lands its tool calls in the user's chat as tool lines with no answer behind them. That is the reported symptom. The same holds for the deflected turn (A8, `tools=None` at L870) and the `agent_max_rounds=0` kill switch.
## Work
1. `app/rag/agent.py` — new module constants next to `CORRECTION_INSTRUCTION` (L732):
- `ROUND_CAP_WRAPUP_INSTRUCTION: str` — the forced final's coaching, in the CORRECTION_INSTRUCTION voice, covering exactly four points: no tool calls remain; do NOT emit a tool call, tool-call JSON, or any `<|tool_call…|>`-style markup (a tool call now has no effect at all); answer from the results already in the conversation, stating plainly what is still unchecked; if the answer is incomplete, end by telling the user they can say **"continue"** to let you keep researching.
- `ROUND_CAP_FALLBACK_ANSWER: str` — the deterministic backstop line (one short sentence in the brand voice: the research budget for this turn ran out before an answer came together, what was gathered still stands above, say **"continue"** to keep digging). Keep it free of markdown decoration (the UI renders it through the normal markdown path).
2. `app/rag/agent.py` — `AgentHolder` (L1482): add `rejected_calls: int = 0` (+ a docstring paragraph in the `tool_calls` style: calls the model attempted after the round cap — never executed, never shown, counted here only; the refusal style, "counts in nothing").
3. `app/rag/agent.py::run_agent`, the forced-final block (L2270-2322):
- build the coached message list the way the phase-71 recovery does it (L2172-2180): `messages_capped = [{"role": "system", "content": system_prompt + "\n" + ROUND_CAP_WRAPUP_INSTRUCTION}, *messages[1:]]` and pass THAT to `chat_stream_retried(..., tools=None, …)` (the retry/restart semantics of locked A2 are unchanged);
- inside the consumption loop, intercept before yielding: `if isinstance(piece, ToolCallPiece): holder.rejected_calls += len(…); continue` — one counter increment per attempted call, nothing yielded (no `tool` frame, no visible markup);
- keep the existing `final_content` tally for `StreamPiece(kind="content")` and the existing `holder.scaffold_stripped` / per-span warning block;
- after the stream closes, when `final_content == 0`: log one warning (`agent: round cap hit with no visible reply (rejected_calls=%d, scaffold_stripped=%d) — answering with the budget fallback`) and `yield StreamPiece(kind="content", text=ROUND_CAP_FALLBACK_ANSWER)`; **delete the `MalformedReplyError` raise on this path only** (the recorded revision, `00_phase.md` §Locked decisions — an honest "budget spent" line beats an error frame for this symptom), while every non-cap `MalformedReplyError` (the round recovery, the deflected recovery, `app/api/chat.py` L1060ff) stays exactly as it is;
- keep the `agent round cap reached (rounds=%d) — forcing a final no-tools answer` warning (its existing unit pin at `tests/unit/test_agent.py` L4944 must not change), and add the attempted-calls warning beside it: `agent: %d tool call(s) attempted after the round cap — rejected, not shown`.
- Retries still obey locked A2: the coached request goes through `chat_stream_retried` with the same `settings.llm_retries` / `llm_retry_delay` (the phase-67 pin at `tests/unit/test_agent.py` L4800 keeps passing — update its expectation only if it asserts the exact system text).
4. `app/api/chat.py::_pump` (L893) — the transport half of the same invariant: add a keyword `tools_offered: bool = True` to the piece loop and drop `ToolCallPiece`s (counting them into a local so one debug line can name them) when `tools_offered is False`. Pass `tools_offered=False` for the deflected/kill-switch request (the `chat_stream(..., tools=None)` call at L860-875 and its recovery at L1040-1060) and `True` for the grounded first pass — `run_agent` already suppresses the cap-path calls internally, so the grounded loop keeps its current behavior. A15 is untouched: the event TYPE list in PLAN §4 does not change, a frame is simply no longer emitted for a call that cannot run.
5. `app/rag/agent.py` — the `run_agent` docstring's round-cap paragraph (L214-235) gains the rejection rule + the fallback-answer rule in two sentences, and `app/api/chat.py`'s module docstring gains one line for the no-`tool`-frame-without-tools invariant.
## Testing & Quality
- Unit (`tests/unit/test_agent.py`, extend the phase-71 cluster at L5241 and the cap cluster at L4561): a fake LLM whose forced final (`llm.requests[i][1] is None`) streams `ToolCallPiece`s only → the run yields NO `ToolCallPiece`, yields the `ROUND_CAP_FALLBACK_ANSWER` content piece, `holder.rejected_calls == N`, `holder.tool_calls` unchanged, no exception; the same shape with tool calls + real content → the content streams and no fallback is appended; a scaffolding-only forced final → the fallback replaces the old `MalformedReplyError` (update that existing test and name the revision in its docstring); the forced final's recorded system message contains `ROUND_CAP_WRAPUP_INSTRUCTION` while `messages[1:]` (user + tool history) is unchanged; the `round cap reached (rounds=N)` and the new `attempted after the round cap` warnings both appear in `caplog`.
- Integration (`tests/integration/test_chat_api.py`, extend — the scripted `tool_script` fake at L84/L125 drives this without touching `tests/e2e/mock_llm.py`): a grounded turn with `agent_max_rounds=1` whose forced final returns a `ToolCallPiece` streams `tool` ×1 (the executed round) and then NOTHING of the sort — no second `tool` frame, no `error` frame — and settles with `done` whose delta text is the answer or the fallback; a deflected turn whose mock returns a `ToolCallPiece` emits no `tool` frame at all.
- Coverage: **>90%** on `app/` — every new branch (intercept, fallback, `tools_offered=False`, `final_content == 0`) has a named test.
## Completion Criteria
- [ ] A cap-hitting turn produces ZERO `tool` frames after the cap and ZERO raw tool markup in the answer.
- [ ] That turn always ends with readable content (`done`, not `error`) — the fallback line appears when the model gave nothing but calls/scaffolding.
- [ ] `holder.rejected_calls` counts the attempts; `tool_calls=N` in the per-turn log still counts executed calls only.
- [ ] Non-cap `MalformedReplyError` behavior is byte-identical (deflected twice-scaffolding integration test still green).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 03 — Surface it: `done.budget_exhausted` + the `rounds=` / `cap_hit=` log fields
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…tell the user to continue the session by allowing more tool calls…" (the server-side half: the turn must REPORT that it ran out, so the UI can offer it and the log can prove it).
**Story:** n/a
## Objective
A cap-reached turn reports itself: one additive `done` key (absent on every other turn) plus two per-turn log fields — so the frontend can act on it (task 04) and the owner can grep how often the cap actually bites.
## Work
1. `app/rag/agent.py` — `AgentHolder` (L1482): add `rounds: int = 0` and `cap_reached: bool = False`, documented in the existing field-doc style — `rounds` is the number of rounds CONSUMED (an executed call, a refusal, or a rejected attempt's round; today it moves only with `rounds += 1`), `cap_reached` is set exactly where the forced no-tools final runs (L2270's block).
- Mirror the loop-local `rounds` into `holder.rounds` at the same statement (`holder.rounds = rounds` beside `rounds += 1`) AND once before the forced final, so a turn that ends at the cap reports the full count; do not re-derive it in the API layer.
2. `app/schemas.py` — `ChatDoneEvent` (L316): add `budget_exhausted: bool = False` with a docstring paragraph in the `answer_image` style (phase 160, TODO L4 — true ONLY on a turn that hit `BOR_AGENT_MAX_ROUNDS`; the UI's Continue affordance keys on it) and EXTEND the existing `@model_serializer(mode="wrap")` rule so the key is popped when `False`: frames of every non-cap turn serialize byte-identically to today ("ABSENT, never null" — the `answer_image` / `SourceRef.image_url` / phase-123 `image` precedent, and the byte-identical-off rule in PLAN §13).
3. `app/api/chat.py` — the `ChatDoneEvent(...)` construction (~L1265): pass `budget_exhausted=holder.cap_reached` (the grounded turn only; the deflected path builds its own done frame and never sets it — a deflected turn runs no agent, so the default `False`/absent is correct by construction).
4. `app/api/chat.py` — the per-turn log line (L1196-1219): insert `rounds=%d cap_hit=%d` immediately after `tool_calls=%d` (PLAN §9's field order is `… tool_calls=N …`, the agent fields stay grouped), with `holder.rounds` and `int(holder.cap_reached)`. Keep every existing field, its order and its format byte-identical (several tests grep this line).
5. `.agents/PLAN.md` — the ONLY plan edit of this phase (the phase-45 pattern, owner permission = the 2026-10-05 roadmap confirmation for this phase; AGENTS.md rule 3 — a recorded revision, never a silent deviation):
- §4's SSE line: `done` `{deflected, sources[], suggestions[], budget_exhausted?}` + one sentence: a `tools=None` request never emits a `tool` frame, and a round-capped turn whose final reply carries no visible content answers with the budget fallback instead of erroring (revising the phase-71 cap-path note);
- §9's required line: add `rounds=N cap_hit=0|1` after `tool_calls=N`;
- §12 is left alone (the roadmap table lists shipped phases by directory).
6. No migration, no new table, no new setting — `agent_max_rounds` keeps its default of 10 (LOCKED).
## Testing & Quality
- Unit: `tests/unit/test_schemas*.py` (extend the `ChatDoneEvent` serialization pins — the phase-124 answer-image tests are the template): `budget_exhausted=False` → the key is ABSENT (assert `"budget_exhausted" not in payload`), `True` → present and `true`; a full done frame with sources + related + suggestions + no flag serializes byte-identically to the pre-phase expectation (the existing golden payload test must pass untouched). `tests/unit/test_agent.py` — `holder.rounds` counts refusals as rounds today (an unknown-tool call bumps `rounds` but not `tool_calls`), and `cap_reached` is `False` on a turn that answered inside the cap.
- Integration (`tests/integration/test_chat_api.py`, extend): a cap-hitting scripted turn ends with `data: …"budget_exhausted": true…` and the captured `caplog` line contains `rounds=N cap_hit=1`; an ordinary grounded turn's `done` frame has no `budget_exhausted` key and its log line reads `cap_hit=0`; the deflected turn's line reads `rounds=0 cap_hit=0`.
- Coverage: **>90%** on `app/` (the serializer branch + both holder fields covered).
## Completion Criteria
- [ ] `done.budget_exhausted` is `true` exactly on cap-reached turns and absent (never `false`) on every other turn — including all deflected turns.
- [ ] The per-turn line reads `… tool_calls=10 rounds=10 cap_hit=1 …` on a capped turn; every pre-existing field is unchanged in value and order.
- [ ] `.agents/PLAN.md` §4/§9 carry the additive revision note and nothing else changed in the plan.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,35 @@
# Task 04 — "Continue researching" on an answer that ran out of tool calls
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "…tell the user to continue the session by allowing more tool calls (or something similar)."
**Story:** n/a
## Objective
When a turn ends because its tool budget was spent, the answer bubble offers **Continue researching**; one click sends the next turn (fresh history + a whole new round budget) so a long research session keeps going without typing.
## Work
1. `frontend/assets/app.js` — `CONTINUE_ICON` beside `TUNE_ICON` (L475) / `RETRY_ICON` (L606): an inline `aria-hidden` SVG in the same stroke family (a forward/arrow-on form, 24×24, `fill="none" stroke="currentColor" stroke-width="1.7"`) — no icon font, no CDN (A11).
2. `frontend/assets/app.js` — `const CONTINUE_QUESTION = "continue";` and `appendContinueButton(wrap)` directly after `appendRetryButton` (L617), same shape: `getActionsRow(wrap)` (L498), one-per-bubble guard on `.continue-btn`, `role="listitem"` when the row reports `role="list"`, `btn.innerHTML = CONTINUE_ICON + "<span>Continue researching</span>"`, click → `continueTurn()`.
3. `frontend/assets/app.js` — `continueTurn()`, the `retryLastTurn` (L2984) discipline:
- no-op while `uiState === UI_STATE.thinking || uiState === UI_STATE.streaming` (one turn at a time);
- clear the button from every bubble first (`messagesEl.querySelectorAll(".continue-btn").forEach(b => b.remove())`) so it can never be clicked twice into two turns;
- `return runTurn(CONTINUE_QUESTION)` — a plain send (`reask` defaults to `false`): the user bubble appears, `conversation.push({who:"user", text:"continue"})` + `saveConversation()` + `persistConversation()` are the ordinary save points, and the request carries the full `history`, so the model resumes with everything it already read and gets a fresh `agent_max_rounds`. A10 holds: nothing server-side knows or cares that this is a continuation.
- ASSUMPTION: the continuation question is the literal word `continue` (what the model is taught to ask the user for in `ROUND_CAP_WRAPUP_INSTRUCTION` / `TOOL_BUDGET_SPENT`, so the visible bubble, the typed equivalent and the model's own suggestion all agree).
4. `frontend/assets/app.js` — the `done` settle (L3600-3652): add `budget_exhausted: ev.budget_exhausted || undefined` to the `rememberBrainTurn(…)` meta object (the `related` / `answer_image` optional-key precedent — `undefined` drops the key, so every non-cap record stays byte-identical), and after the existing `markLastRetryable()` (L3649) add `if (ev.budget_exhausted) appendContinueButton(wrap);` (so the row order is Tune → Retry → Continue → Save-as-doc, the insertion order phase 130 relies on).
5. `frontend/assets/app.js` — the restore: `renderStoredMessage`'s brain branch (after L2281) gets `if (m.budget_exhausted) appendContinueButton(wrap);` — a reload keeps the affordance (a restored `history` is exactly what the continuation turn needs). `startNewChat()` (L2886) needs nothing (resetting `messagesEl` drops the buttons with the list).
6. `app/schemas.py` — `ChatMessage` (L1264ff, `extra="forbid"`): add the optional marker `budget_exhausted: bool | None = None` documented in the phase-120 `failed` / phase-48 `stopped` style (the persisted answer that ran out of tool calls — the client's restore hint, never a server behavior). No migration: `saved_chats.messages` is raw JSONB (phase 50), and `app/api/chats.py`'s logic is unchanged.
7. `frontend/assets/styles.css` — `.continue-btn` in the `.retry-btn` family (L1010: identical pill size/spacing/color, the shared `:focus-visible` ring — extend the selector list if the ring rule is per-class) + `min-height: 44px` beside `.retry-btn` in the mobile block (L5708-5709) + any `prefers-reduced-motion` parity the family needs (PLAN §7 feedback rules).
8. Verify the shared page (`/shared/<token>`, rendered server-side, not by `app.js`) shows no Continue control for a flagged record — the key is ignored there by construction; record the check in the task's commit note (no code change expected).
## Testing & Quality
- Unit (house-style source pins; new `tests/unit/test_frontend_tool_budget.py`): `appendContinueButton` joins the actions row with the one-per-bubble guard and `listitem` role; `continueTurn` bails on `thinking`/`streaming`, removes every `.continue-btn` before sending, and calls `runTurn(CONTINUE_QUESTION)` with no `reask`; the `done` handler passes `budget_exhausted` through `rememberBrainTurn` and appends the button only when flagged; the restore branch keys on `m.budget_exhausted`; `retryLastTurn`, `markLastRetryable` and `rememberBrainTurn` bodies are byte-unchanged.
- Integration (`tests/integration/test_chats_api.py`, extend): a saved chat whose brain record carries `budget_exhausted: true` round-trips byte-identically on `GET /api/chats/<id>` (and on the shared endpoint) — pins the additive `ChatMessage` field.
- Coverage: **>90%** on `app/` (the only `app/` change is the schema field — covered by the round-trip test).
- Manual: with `BOR_AGENT_MAX_ROUNDS=1` (or the task-05 mock marker) confirm the answer offers Continue, one click runs a fresh turn, and a reload restores the button.
## Completion Criteria
- [ ] An answer from a cap-reached turn carries a working **Continue researching** button; clicking it sends exactly one new turn whose request carries the prior `history`, and its answer streams normally.
- [ ] A non-cap answer shows no such button, and neither does any earlier answer once a newer turn lands.
- [ ] The button is inert while a turn is in flight; no double-send is possible.
- [ ] A page refresh keeps the button on a flagged answer; the shared page never shows it.
- [ ] `bor.chat.v1` records of non-cap turns are byte-identical to pre-phase (no `budget_exhausted` key).
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,38 @@
# Task 05 — Budget tests: the mock's round-cap flow, unit/integration pins, isolated E2E
**Phase:** `160_tool_budget_telemetry` · **Source:** `TODO.md:4` — "The LLM need to know how many tool calls it has left. During long research sessions it will sometimes end the conversation with tool calls that just get dumped into chat because it ran out of turns…"
**Story:** n/a
## Objective
The whole contract is pinned end to end: a deterministic mock flow that runs a turn to the cap and then tries to dump tool calls into chat, the unit/integration assertions for the trailer / rejection / flag, and the phase's own Playwright file run in isolation (AGENTS.md rule 9).
## Work
1. `tests/e2e/mock_llm.py` — a new deterministic flow, documented in the module docstring's marker table:
- `ROUND_CAP_TRIGGER = "run out of tool calls"` — detected on the LAST user message only (the mock's existing invariant: the last `user` message is the current question, the earlier non-system messages are prior turns — see the `last_user` scan around L3186);
- while tools are offered (`"tools"` present on the request) and no tool result answers the last emitted call: stream ONLY `tool_calls` deltas — `ls` with no arguments, incrementing id `call_cap_1`, `call_cap_2`, … , `finish_reason: "tool_calls"`, no content — i.e. the model never stops asking, so the turn is guaranteed to hit `BOR_AGENT_MAX_ROUNDS`;
- on the request WITHOUT tools (the forced final) in this flow: stream a `tool_calls` delta anyway (`read` on the seed path, id `call_cap_dump`) and NO content — the exact dumped-call shape the phase kills, and the shape `SCAFFOLD_TRIGGER` (L1375) does not cover (that one emits textual markup, this one is structured);
- when the last user message is exactly `continue` and an earlier message carries the trigger: stream a stable answer `ROUND_CAP_CONTINUE_ANSWER` (a fixed sentence naming the seed document, so the E2E asserts a real continuation answer rather than a tool call).
- The new branch must not disturb any existing trigger (the phase-45 multi-read flow, `TOOLS_TRIGGER`, `SCAFFOLD_TRIGGER`, the failure markers) — the existing mock-only suites are the regression gate.
2. `tests/e2e/test_tool_budget_telemetry.py` (new, MOCK-ONLY — no `E2E_REAL_LLM` support, the phase-45/120 files are the template: `app_server` / `page` / `db_ready` fixtures, `auth_helpers.login`, the TRUNCATE-then-seed KB pattern over `tests/fixtures/docs/homelab/kubernetes.md`, the autouse `saved_chats` truncate):
1. `test_tool_results_carry_the_remaining_count` — server-side pin at the wire level: run the cap flow and read the mock's recorded requests (the mock exposes its request log; otherwise assert through the app's captured log lines) that each successive tool message ends with `[tool calls remaining: N]` counting down and the last says no tool calls remain.
2. `test_capped_turn_never_shows_tool_lines_after_the_cap` — with `BOR_AGENT_MAX_ROUNDS=3` (the `app_server` env override the existing suites already use for tuning): the chat shows at most 3 tool lines total, the dump call produces NO tool line, the answer bubble holds non-empty readable text (the fallback or the coached answer), the turn settles with `done` (never the error banner), and the SSE capture has no `error` frame.
3. `test_done_flags_budget_exhausted` — the captured SSE stream's `done` frame carries `budget_exhausted: true`; a normal grounded turn's `done` carries no such key (byte-identical control case in the same file).
4. `test_continue_button_runs_a_fresh_turn` — the capped answer bubble shows **Continue researching**; clicking it appends exactly one user bubble `continue`, the mock answers with `ROUND_CAP_CONTINUE_ANSWER`, the earlier tool lines do not duplicate, and `bor.chat.v1` holds `[…brain(flagged), user("continue"), brain]`.
5. `test_continue_is_inert_while_streaming_and_single_shot` — clicking Continue twice fast (or during the streamed turn) starts ONE turn (count the `/api/chat` requests via `page.on("request")`).
6. `test_flag_survives_a_refresh` — `page.reload()` after a capped turn: the flagged answer restores WITH the Continue button and clicking it still works.
7. negative `test_deflected_turn_gets_no_continue_button` — a deflected question (A8 LOW mode) settles deflected with no `budget_exhausted` key and no Continue button.
- The file's docstring states: the markers, the `BOR_AGENT_MAX_ROUNDS` override used, the seed, and this numbered test → contract mapping (the phase-120 file's shape).
3. `tests/unit/test_tool_budget_note.py` (new, if `tests/unit/test_agent.py` gets too long for the reviewer's taste — otherwise extend that file's round-cap cluster L4561): `_budget_note` boundaries, the trailer's presence on executed AND refused calls, `agent_max_rounds=0` producing no tool message at all.
4. `tests/unit/test_frontend_tool_budget.py` — already written in task 04; extend it with anything task 04 left unpinned rather than duplicating it here.
5. Regression gate (each file in isolation, `--no-cov`): `test_agent_unlimited_tools.py` (the multi-read flow must still stream its 3 tool lines and answer), `test_agent_document_tools.py`, `test_harness_aligned_tools.py`, `test_grep_regex_teaching.py`, `test_big_read_progress.py` + `test_read_range_tool_lines.py` (the `tool_result` frame path the trailer rides next to), `test_chat_rag.py`, `test_scaffolding_guardrails.py` (or its current name in `tests/e2e/`), `test_smoke.py`.
## Testing & Quality
- Order: `uv run pytest tests/unit -v --no-cov` → `uv run pytest` (unit + integration) → `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%) → `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` (DB up) → the regression list above, isolated.
- Every new branch of tasks 01-04 must be attributable to a named test in the coverage report (`--cov-report=term-missing` shows no new red lines in `app/rag/agent.py`, `app/api/chat.py`, `app/schemas.py`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_tool_budget_telemetry.py -v --no-cov` green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90% with no uncovered new lines in the touched modules.
- [ ] Every regression E2E file listed above green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agents/PLAN.md` shows exactly the task-03 revision note; the phase dir moves to `.agents/phases/complete/` with the single `--no-gpg-sign` commit (`00_phase.md` §Commit).

Some files were not shown because too many files have changed in this diff Show More