Compare commits
4
Commits
bdeea71f5d
...
6e654df387
| Author | SHA1 | Date | |
|---|---|---|---|
 ducoterra
|
6e654df387 | ||
|
ducoterra
|
f50d4c8bc3 | ||
|
ducoterra
|
07fbc4c450 | ||
|
ducoterra
|
0c8018971f |
@@ -0,0 +1,38 @@
|
||||
# Phase 127 — Incremental folder summaries: sync rewrites only the directories its changed files touch
|
||||
|
||||
**Source:** `TODO.md` L3 — "On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/api/sync.py` — `_run_sync` step 5 (the changed-KB full-regeneration call to `generate_folder_summaries`, the unchanged-walk gap probe, the `_summary_hook` progress closure, the `SyncStatus` phase machine); `app/rag/folder_summaries.py` — `folder_of`, `group_by_folder` (the recursive-subtree concept), `_candidates`, `_stored_keys`, `missing_folder_summaries`, `generate_folder_summaries` (full / `only_missing` modes, manual-row skip, fail-soft, prune pass, `on_progress` hook, "only flushes — the caller commits"); `app/rag/importer.py` — `ImportSummary` (L212, counts only — no path tracking), the `summary.added += 1` / `summary.updated += 1` increment sites (grep them: the text path ~L794, the image path ~L1068, the PDF path ~L1328, ~L1730, ~L1994) and `_prune` (L2015, deletes vanished docs); `scripts/import_docs.py` — the CLI caller (passes no changed set — full regeneration stays its contract); `tests/unit/test_folder_summaries.py`, `tests/unit/test_importer.py`, `tests/unit/test_sync_button.py` — the existing suites to extend; `tests/e2e/test_sync_summary_visibility.py` — the sync E2E pattern (local `kind=local` source fixture, autouse truncate cleanup, mock-only).
|
||||
|
||||
## Objective
|
||||
A KB-changing sync regenerates folder summaries for ONLY the folders affected by the documents that actually changed (each changed document's folder chain up to its source root) — one changed file rewrites a handful of rows, not the whole table. Unchanged folders keep their stored rows byte-identical (text AND `updated_at`). Full regeneration remains available and stays the exact behavior for callers that pass no change set (the CLI, any future full-regen need); the unchanged-walk gap probe is untouched.
|
||||
|
||||
## Dependencies
|
||||
- `126_pdf_support` (complete) — last completed phase; this phase builds on the shipped sync pipeline (phases 32/64/98), the folder-summary generator (phases 94/96/97/98) and the importer.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Change tracking (task 01):** `ImportSummary` gains `changed_paths: list[tuple[str, str]] = field(default_factory=list)` — one `(source, path)` entry per document that was **added, updated, or pruned** in the run, in the order encountered (added/updated in walk order, pruned in `_prune` order). Recorded at EVERY `summary.added += 1` / `summary.updated += 1` site with the site's `(source, rel)` / `(doc.source, doc.path)`, and in `_prune` for each deleted doc (record before `db.delete`). No dedupe needed: a document is added-or-updated at most once per run and pruned docs are gone; the consumer builds a set anyway. `unchanged` / `summary_backfilled` / `dates_updated` NEVER record (their contract is no-regeneration, phase 106/118). The `log()` line is unchanged (the count is derivable: `len(changed_paths) == added + updated + pruned`).
|
||||
- **Affected keys (task 02):** `app/rag/folder_summaries.py` gains a pure helper `affected_folder_keys(changed: Iterable[tuple[str, str]]) -> set[tuple[str, str]]` — for each `(source, path)`: the source-root key `(source, "")` plus every folder prefix obtained by walking `folder_of` from the document's nearest folder up to `""` (the SAME one concept `group_by_folder` pass 1 uses — a doc under `a/b/` affects `a/b`, `a`, and the root). `generate_folder_summaries` gains `changed_docs: Sequence[tuple[str, str]] | None = None`:
|
||||
- `None` (default) → today's full regeneration, byte-identical (the CLI path and any explicit full-regen call).
|
||||
- given → `keys = sorted(set(candidates) & affected_folder_keys(changed_docs))`; the generation loop, the manual-row skip (`kept_manual`), the per-folder fail-soft, and the `on_progress` hook (whose `total` becomes the affected count — the sync status's `summaries_total` then reports the affected count through the existing hook) all run over exactly those keys. An affected folder with NO stored row (a gap in an affected folder) is generated like any other candidate. Missing rows in UNaffected folders stay the unchanged-walk gap probe's job (untouched).
|
||||
- the prune pass runs in BOTH modes, unchanged (vanished folders' rows are dropped there, manual rows exempt — the phase-97 rule).
|
||||
- `only_missing=True` (the gap-fill) is untouched and orthogonal.
|
||||
- **Sync wiring (task 03):** `app/api/sync.py` `_run_sync` step 5, the changed-KB branch only: `generate_folder_summaries(fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook)`; the unchanged-walk gap probe branch is byte-identical. The module docstring's step-5 description and the `SyncStatus` docstring get updated to the incremental contract. The success log line names the change count (`sync: folder_summaries changed=N stats=…` — the stats dict stays log-only, the detail shape untouched, phase-98 convention).
|
||||
|
||||
## Tasks
|
||||
1. `01_importer_changed_paths.md` — `ImportSummary.changed_paths` recorded at every added/updated/prune site.
|
||||
2. `02_folder_summaries_affected_mode.md` — `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)`.
|
||||
3. `03_sync_wiring.md` — the sync's changed-KB branch passes the change set; docstrings + logging.
|
||||
4. `04_e2e_incremental_sync.md` — dedicated Playwright E2E, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: every behavior above — the change-set contents/order (all three doc kinds + prune), the affected-keys math (nested folders, roots, multi-source, `#page=`-fragment paths), the generator's changed mode (only-affected upserts, untouched rows byte-identical text AND `updated_at`, manual skip, fail-soft, prune pass, hook total), the CLI no-change path (byte-identical full regen), the sync status's `summaries_total` = affected count.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
- E2E: one dedicated Playwright file, run in isolation (`uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov`), mock-only.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a one-file change in a folder rewrites exactly that folder's chain (folder + ancestors + source root) — sibling and other-source rows keep their `updated_at` and text
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%
|
||||
- [ ] `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (CLI full regen, gap probe, manual rows, phase machine, keep-final-counts)
|
||||
- [ ] one atomic Conventional Commit with `--no-gpg-sign` per the AGENTS.md git protocol
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# Task 01 — `ImportSummary.changed_paths`: the importer records which documents it changed
|
||||
|
||||
**Phase:** `127_sync_incremental_folder_summaries` · **Source:** `TODO.md:3` — "On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time."
|
||||
|
||||
## Objective
|
||||
`ImportSummary` carries the identity of every document the run added, updated, or pruned — `(source, path)` tuples — so the sync path can scope the folder-summary regeneration to exactly the folders those documents touch.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — `ImportSummary` (L212): add the field with a docstring comment matching the house style:
|
||||
```python
|
||||
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
|
||||
#: tuples in encounter order (added/updated in walk order, pruned in
|
||||
#: _prune order). Drives the sync's incremental folder-summary
|
||||
#: regeneration (phase 127): len(changed_paths) == added + updated +
|
||||
#: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
|
||||
changed_paths: list[tuple[str, str]] = field(default_factory=list)
|
||||
```
|
||||
2. `app/rag/importer.py` — record at every increment site. Grep `summary.added +=` and `summary.updated +=` (currently the text path ~L794/796, the image path ~L1068/1070, the PDF path ~L1328/1330, ~L1730, ~L1994/1996 — the exact set may have shifted; the rule is: **every** site). At each, immediately beside the counter increment, append the site's identity: `summary.changed_paths.append((source, rel))` (the variables each site already has — `source` + `rel` for the index paths; where a site only has the `doc` row, use `(doc.source, doc.path)`).
|
||||
3. `app/rag/importer.py` — `_prune` (L2015): in the delete loop, for each document that is actually deleted (i.e. NOT skipped by the images-off guard and NOT skipped by the phase-126 embedded-image parent rule), append `(doc.source, doc.path)` to `summary.changed_paths` BEFORE `session.delete(...)` — `_prune` needs the summary passed in (add a `summary: ImportSummary | None = None` parameter, recorded only when given; the existing callers — `import_sources`'s prune call — pass it; any other caller keeping `None` stays byte-identical). Check every existing `_prune` call site and update only the `import_sources` one.
|
||||
4. `app/rag/importer.py` — do NOT touch `ImportSummary.log()` (the new count is derivable) and do NOT touch the sync's `detail` dict shape (task 03 keeps it log-only).
|
||||
|
||||
ASSUMPTION: one entry per changed document is enough (no per-file verb) — the consumer only needs folder keys, and the verb is already in the counters.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_importer.py` (extend) or a focused new file —
|
||||
- a walk with 2 added + 1 updated + 1 pruned doc (text files, a temp source dir) → `changed_paths` holds exactly those 4 `(source, path)` pairs, added/updated in walk order, the pruned pair last (or in `_prune` order);
|
||||
- an images-on run with a changed image file records its `(source, rel)` (the image increment site);
|
||||
- a PDF run with a changed PDF records its pair (the PDF increment site);
|
||||
- an unchanged re-run (all `unchanged`) → `changed_paths` empty;
|
||||
- a backfill/date-only run (phase 118/106 shapes, if covered by existing fixtures) → still empty.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ImportSummary()` default construction gives `changed_paths == []` (no migration of existing construction sites)
|
||||
- [ ] every `added`/`updated`/`pruned` increment in the importer has a matching `changed_paths` append (a grep audit passes: same count of sites)
|
||||
- [ ] `uv run pytest tests/unit -q` green; full suite green; no behavior change in completed work
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
# Task 02 — `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)`: regenerate only the folders a change touches
|
||||
|
||||
**Phase:** `127_sync_incremental_folder_summaries` · **Source:** `TODO.md:3` — "On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time."
|
||||
|
||||
## Objective
|
||||
The folder-summary generator can be told the set of changed documents and then regenerates ONLY the candidate folders whose recursive subtree contains at least one of them — the change's folder chain per document, plus each source root — with every existing guarantee (manual skip, fail-soft, prune pass, progress hook) intact.
|
||||
|
||||
## Work
|
||||
1. `app/rag/folder_summaries.py` — new pure helper (place it next to `folder_of` / `group_by_folder`, with a docstring tying it to the recursive-subtree concept):
|
||||
```python
|
||||
def affected_folder_keys(
|
||||
changed: Iterable[tuple[str, str]],
|
||||
) -> set[tuple[str, str]]:
|
||||
"""The candidate folder keys a changed-document set invalidates.
|
||||
|
||||
For each ``(source, path)``: the source-root key ``(source, "")``
|
||||
plus every folder prefix of *path* — ``folder_of`` walked from the
|
||||
document's nearest folder up to ``""`` — the SAME prefix walk
|
||||
``group_by_folder`` pass 1 uses (one concept): a doc under
|
||||
``a/b/`` affects ``a/b``, ``a``, and the source root. A
|
||||
root-level file (``folder_of(path) == ""``) affects ONLY the
|
||||
source root. Embedded-image fragment paths (``x.pdf#page=…#img=…``,
|
||||
phase 126) fold into their parent's folder chain with no special
|
||||
case — ``folder_of`` never sees a ``/`` inside the fragment.
|
||||
Returns a set; the generator sorts it (deterministic order).
|
||||
"""
|
||||
```
|
||||
Empty input → empty set.
|
||||
2. `app/rag/folder_summaries.py` — `generate_folder_summaries`: add the kwarg `changed_docs: Sequence[tuple[str, str]] | None = None` (after `only_missing`, before `on_progress` — keyword-only section). Inside, after `candidates = _candidates(_catalog_rows(db))` and the `existing` fetch:
|
||||
- `keys = sorted(candidates)` stays the full-mode default;
|
||||
- when `changed_docs is not None`: `keys = sorted(set(candidates) & affected_folder_keys(changed_docs))` — generation is restricted to affected CANDIDATES (a folder that exists in the catalogue but is unaffected keeps its row; an affected folder with no stored row — a gap — is a candidate and gets generated);
|
||||
- `only_missing=True` keeps its exact current semantics (restricts to missing candidates; the two kwargs are never combined by any caller — document that in the docstring);
|
||||
- the generation loop, `kept_manual` skip, per-folder `LLMError` fail-soft, and the `on_progress(i + 1, len(keys), source, folder_path)` hook are untouched code — `total` simply becomes the affected count when restricted;
|
||||
- the prune pass is untouched (runs in both modes, deletes rows for folders that dropped below the minimum, manual rows exempt).
|
||||
Update the function docstring: the new mode, the one-line contract ("a document's change invalidates exactly its folder chain + source root — the folders whose subtree summary text can change"), and that `changed_docs=None` is the byte-identical full regeneration (the `scripts/import_docs.py` CLI).
|
||||
3. `scripts/import_docs.py` — verify (do not modify) that the CLI call passes no `changed_docs` — it must stay a full regeneration.
|
||||
|
||||
ASSUMPTION: the intersection (`candidates & affected`) is the correct restriction — an affected folder that no longer has any document is handled by the prune pass, not by generation; an unaffected folder's summary cannot change because its subtree's documents and their first-summary-lines are untouched.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_folder_summaries.py` (extend) —
|
||||
- `affected_folder_keys`: a root-level file → `{(src, "")}`; `a/b/c.md` → `{(src, ""), (src, "a"), (src, "a/b")}`; mixed multi-source input; an embedded-image fragment path (`img.pdf#page=2#img=0` under `dir/`) → the `dir` chain; empty input → empty.
|
||||
- `changed_docs=` mode over a seeded catalogue (3 sources, nested folders, a manual row, a failing folder): exactly the affected keys are upserted (fake LLM counts calls — one per non-manual affected key); unaffected rows keep their stored `summary` AND `updated_at` (assert both); manual affected row → `kept_manual`, untouched; a failing affected folder → `failed`, its previous row kept, others still land; the hook fires `len(affected)` times with `total == len(affected)`; the prune pass still drops a vanished folder's row;
|
||||
- `changed_docs=None` over the same catalogue → the pre-task full-regeneration behavior (all candidates, byte-identical stats shape).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `affected_folder_keys` is a pure function unit-tested on nested / root / multi-source / fragment paths
|
||||
- [ ] `generate_folder_summaries(db, llm)` (no new kwarg) behaves exactly as before (existing suite green, unchanged)
|
||||
- [ ] restricted mode leaves every unaffected row byte-identical (text AND `updated_at`)
|
||||
- [ ] `uv run pytest tests/unit -q` green; full suite green
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 03 — Sync wiring: the changed-KB branch regenerates only affected folders
|
||||
|
||||
**Phase:** `127_sync_incremental_folder_summaries` · **Source:** `TODO.md:3` — "On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time."
|
||||
|
||||
## Objective
|
||||
`POST /api/sync` passes the import's change set into the folder-summary generator, so a one-file sync burns `lite` calls only for that file's folder chain. The status phase machine, progress hook, gap probe, and terminal conventions are unchanged in behavior (the hook's `total` now naturally reports the affected count).
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py` — `_run_sync`, step 5, the CHANGED-KB branch (the `if summary.added + summary.updated > 0:` arm that currently calls `generate_folder_summaries(fs_db, llm, on_progress=_summary_hook)`): pass `changed_docs=summary.changed_paths`. The else arm (unchanged walk → `missing_folder_summaries` gap probe → `only_missing=True`) stays byte-identical, including the no-gap skip's "stays `import`" phase behavior.
|
||||
2. `app/api/sync.py` — logging: the changed-KB branch's existing `logger.info("sync: folder_summaries stats=%s", folder_stats)` becomes `logger.info("sync: folder_summaries changed=%d stats=%s", len(summary.changed_paths), folder_stats)` (PLAN §9 ample logging — the owner can see how many folders a sync actually rewrote). The gap-fill log line is unchanged.
|
||||
3. `app/api/sync.py` — docstrings: the module docstring's step-5 paragraph (the "full regeneration" sentence) and the `SyncStatus` docstring's `summaries_done`/`summaries_total` note — update to the incremental contract: on a changed KB the phase runs over the CHANGED set's folder keys (the hook's `total` is the affected count, not the table's candidate count); on an unchanged walk the gap-fill total is the missing count (as before); terminal states keep the final counts (the phase-64 convention, unchanged).
|
||||
4. Verify: the `detail` dict shape is untouched (no new key — the stats stay log-only, phase-98 convention); `GET /api/sync/status` shape is untouched (the chunk-free status fields are already there; no new fields in this phase).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: extend the sync unit suite (`tests/unit/test_sync_button.py` and/or the sync-pipeline test file that fakes `import_sources` + `generate_folder_summaries` — inspect the existing fakes first):
|
||||
- a changed-KB sync (fake import returning 1 updated doc) → `generate_folder_summaries` receives `changed_docs=` equal to the fake's change set (assert on the fake's captured kwargs) and the success detail is unchanged in shape;
|
||||
- an unchanged walk with a gap → still `only_missing=True`, no `changed_docs` (the gap probe is orthogonal);
|
||||
- the status during the summaries phase reports `summaries_total == len(affected keys)` (fake generator whose hook total is its key count — follow the existing phase-machine test pattern).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a changed-KB sync calls the generator with `changed_docs=summary.changed_paths`; an unchanged walk calls it exactly as before
|
||||
- [ ] `GET /api/sync/status` and the success `detail` shapes are byte-identical to pre-phase
|
||||
- [ ] docstrings + log lines reflect the incremental contract
|
||||
- [ ] `uv run pytest tests/unit -q` green; full suite green
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
# Task 04 — E2E: a one-file change rewrites only that folder's summary chain
|
||||
|
||||
**Phase:** `127_sync_incremental_folder_summaries` · **Source:** `TODO.md:3` — "On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time."
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) proving the owner-visible fix end to end: after a first full sync, changing ONE file in ONE folder and re-syncing re-stamps exactly that folder's chain (the folder + its ancestors + the source root) and leaves every other stored row byte-identical.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_incremental_folder_summaries.py` (new) — follow `tests/e2e/test_sync_summary_visibility.py`'s house pattern exactly (module-level app fixture over the shared E2E Postgres, host temp-dir `kind=local` source registered via the admin git-sources API, autouse cleanup that truncates `documents`/`chunks`/`query_log`/`steering_notes`/`kb_overview`/`git_sources`/`folder_summaries` and waits for no running sync, mock-only — `E2E_REAL_LLM=1` unsupported). No slow-LLM proxy needed: the assertions are on POST-terminal DB state + the terminal status (the keep-final-counts convention), not on in-flight sampling.
|
||||
KB fixture — one local source `incsum/` (temp dir) with FOUR folders `alpha/`, `bravo/`, `charlie/`, `delta/`, two markdown docs each, plus one root-level doc: the generator's candidate set is exactly 5 rows (source root + 4 folders — the recursive-subtree ≥ 1 rule).
|
||||
2. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_first_sync_generates_all_rows` — admin sync #1 settles `success`; the `folder_summaries` table holds exactly the 5 rows (the mock's deterministic `FOLDER_SUMMARY_MODE` canned text names each folder — `tests/e2e/mock_llm.py`'s existing canned reply), each `updated_at` non-null.
|
||||
- `test_one_file_change_restamps_only_that_folders_chain` — after test 1's state is re-seeded (autouse truncates, so seed again): re-run sync #1, capture each row's `(summary, updated_at)`; then modify ONE doc inside `alpha/` (change its content), sync #2 settles `success`; assert: `alpha` + the source root rows have a NEW `updated_at` (strictly after the captured ones) and the canned text; `bravo`/`charlie`/`delta` rows are byte-identical (same `summary` AND same `updated_at`); the terminal `GET /api/sync/status` keeps `summaries_done == summaries_total == 2` (the affected count — alpha + root, the keep-final-counts convention) and `detail.added/updated/unchanged/pruned` show the single-file change.
|
||||
- `test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_folder` — seed + full sync; capture; delete `bravo/`'s LAST doc (the folder vanishes) and modify one doc in `charlie/`; sync settles; assert: the `bravo` row is PRUNED (gone — the prune pass), the source root + `charlie` rows re-stamped, `alpha`/`delta` byte-identical.
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov`), the mock-only gate, the fixture layout, and the test→observable table (the house format).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's unit/integration coverage bar (>90% on `app/`) is already carried by tasks 01–03; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the three tests above map 1:1 to the TODO's observable: one changed file → a handful of rewrites, not 700+
|
||||
- [ ] no behavior change in completed phases (the full-regen first sync, the gap probe, the phase machine)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Phase 128 — The sync label shows on mobile, count always visible
|
||||
|
||||
**Source:** `TODO.md` L4 — "The `Syncing… name/of/file/here (2/3)` doesn't show up on mobile, it needs to" (the item is truncated in the TODO; ASSUMPTION A1 — read as: the running label must be fully visible on mobile, with the `(n/m)` position never clipped off-screen).
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `frontend/index.html` L449–451 — the `#sync-btn` / `#sync-label` markup (one span holds the whole label string); `frontend/assets/sources.js` — `fmtSyncLabel(kind, currentFile, done, total)` (~L650) and `fmtSyncPhaseLabel(status)` (~L669) build the running labels as ONE string, the poll loop applies them to `#sync-label` + the button `title` + `#sync-result` (the aria-live announcer — the untruncated text rides both, A4 convention); `frontend/assets/styles.css` — `.sync-label` (~L2067: `inline-block`, `max-width: min(16rem, 40vw)`, ellipsis, `nowrap`) and the mobile media query (~L5323: `.sync-btn { width: 100%; }`, `.sync-label { max-width: none; min-width: 0; }` — with `max-width: none` the inline-block sizes to its FULL content, so the label's tail — where `(n/m)` lives — runs off-screen: the reported bug); `tests/unit/test_frontend_sync_upload.py` — the pinned frontend label tests to update.
|
||||
|
||||
## Objective
|
||||
On a phone, the running sync label is fully visible: the file path ellipsizes to fit, and the `(n/m)` position sits in its own non-shrinking span that is NEVER clipped — in the import phase (`Syncing… <file> (2/3)`) and the summaries phase (`Summarizing folders… <folder> (n/m)`). Desktop renders the same content with its existing visual layout (the cap, the ellipsis, the title/aria-live untruncated text all preserved).
|
||||
|
||||
## Dependencies
|
||||
- `127_sync_incremental_folder_summaries` (todo) — pipeline order only; code dependencies are the completed phase-64/98 label contract.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Markup (task 01):** `#sync-label` gains two child spans — `<span class="sync-label-main"></span><span class="sync-label-count" hidden></span>` (static markup in `index.html`). The JS builders no longer return one string: a single `setSyncLabel(mainText, countText)` helper writes `textContent` into the parts (the createElement/textContent house rule — no HTML strings) and toggles the count span's `hidden` (absent count → hidden, empty). The FULL untruncated label (`mainText + (countText ? " " + countText : "")`) is still set on the button `title` and `#sync-result` exactly as today (A4 — screen readers hear the whole thing).
|
||||
- **Label mapping (task 01):** main/count per state — prelude (null phase, no file): main `"Syncing…"`, no count; import: main `"Syncing… <file>"`, count `"(n/m)"` while `total > 0`; overview: main `"Writing KB overview…"`, no count; summaries: main `"Summarizing folders… <folder>"` (no folder part while `current_summary` is null), count `"(n/m)"`; upload: main `"Importing"`, no count (phase-90 contract — the upload status carries no file or counts); idle/settle labels (`"Sync sources"`, `"Synced HH:MM"`): main only. `fmtSyncPhaseLabel`'s decision tree is preserved 1:1 — only the OUTPUT shape changes (string → parts).
|
||||
- **CSS (task 02):** `.sync-label` becomes a flex row (`display: inline-flex; align-items: baseline; gap: 0.35rem; min-width: 0; max-width: min(16rem, 40vw);`); `.sync-label-main { min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }`; `.sync-label-count { flex: 0 0 auto; white-space: nowrap; }`. Mobile media query (the existing block at ~L5323): `.sync-label { max-width: none; flex: 1 1 auto; min-width: 0; }` — the label fills the full-width pill, the main part ellipsizes against the remaining width, the count span stays visible at the right edge. The `#sync-result` mobile clip (visually-hidden) is untouched.
|
||||
|
||||
## Tasks
|
||||
1. `01_label_parts.md` — markup + `setSyncLabel` part builders in `sources.js` (all label states, A4 untruncated text preserved).
|
||||
2. `02_mobile_css.md` — the flex-part CSS + the mobile media query.
|
||||
3. `03_e2e_mobile_label.md` — dedicated Playwright E2E, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_frontend_sync_upload.py` — the pinned label outputs updated to the part shape (main/count per state, the byte-identical label strings where they must hold: prelude, upload, settle); the `title`/`#sync-result` untruncated text pins stay.
|
||||
- E2E: one dedicated Playwright file, run in isolation, mobile viewport, slow-LLM pattern.
|
||||
- Coverage: **>90%** on new/modified code (frontend: the JS is covered by the pinned unit tests + the E2E; `app/` untouched this phase).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] at a phone viewport (390×844) the running label is fully in the viewport — file path ellipsized, `(n/m)` visible — in BOTH the import and summaries phases
|
||||
- [ ] desktop label visually unchanged (same strings, same cap/ellipsis, same `title`/`#sync-result` untruncated text)
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_label_mobile.py -v --no-cov` green in isolation
|
||||
- [ ] `uv run pytest` (unit + integration) green; no behavior change in completed phases
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 01 — `#sync-label` becomes parts: the file path ellipsizes, the `(n/m)` count is pinned
|
||||
|
||||
**Phase:** `128_sync_label_mobile` · **Source:** `TODO.md:4` — "The `Syncing… name/of/file/here (2/3)` doesn't show up on mobile, it needs to" (A1: fully visible on mobile, count never clipped)
|
||||
|
||||
## Objective
|
||||
The label's string is split into a shrinks-to-fit main part (prefix + file/folder path) and a pinned count part (`(n/m)`), so CSS can ellipsize the path while the position always stays on-screen. Every existing label state keeps its exact rendered text.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` (L449–451) — inside `#sync-label`, replace the bare text with the two part spans (the initial text lives in the main span so the pre-JS idle label renders):
|
||||
```html
|
||||
<span class="sync-label" id="sync-label"><span class="sync-label-main">Sync sources</span><span class="sync-label-count" hidden></span></span>
|
||||
```
|
||||
2. `frontend/assets/sources.js` — one new helper where `fmtSyncLabel`/`fmtSyncPhaseLabel` live (~L650): `setSyncLabel(mainText, countText = "")` — sets `syncLabelMain.textContent = mainText`; if `countText`: `syncLabelCount.textContent = countText`, `syncLabelCount.hidden = false`; else `syncLabelCount.textContent = ""`, `hidden = true`; then sets the UNTRUNCATED full label — `mainText + (countText ? " " + countText : "")` — on `syncBtn.title` and `syncResult.textContent` (the A4 convention: title + aria-live always carry the whole thing). Get the part elements once at `mount` scope (the `querySelector` house pattern; null-guard like `syncLabel`).
|
||||
- `fmtSyncLabel(kind, currentFile, done, total)` → return the PARTS: `main = currentFile ? `${prefix} ${currentFile}` : prefix` (prefix `"Syncing…"` / `"Importing"` as today), `count = total > 0 ? `(${done}/${total})` : ""`; call `setSyncLabel(main, count)`. The byte-identical full strings hold: `"Syncing…"` (no file), `"Syncing… <file>"` (no total), `"Syncing… <file> (n/m)"`, `"Importing"`.
|
||||
- `fmtSyncPhaseLabel(status)` → same decision tree (overview / summaries / fall-through), emitting parts: overview → `("Writing KB overview…", "")`; summaries → `(`Summarizing folders…${folder}`, `(${status.summaries_done}/${status.summaries_total})`)` with the folder part omitted while `current_summary` is null (as today) — the count is ALWAYS emitted in this phase (the hook always carries a total); fall-through → the `fmtSyncLabel` parts. The full-string shape is byte-identical to the pre-task strings.
|
||||
- Every OTHER call site that writes `#sync-label` (idle `"Sync sources"`, settle `"Synced HH:MM"`, the 202/409 click fallback, the re-attach `initSyncButton` path) goes through `setSyncLabel(main)` — count hidden.
|
||||
- The upload job keeps the bare `"Importing"` (no file, no counts — phase-90 contract).
|
||||
3. `tests/unit/test_frontend_sync_upload.py` — update the pinned label tests to the part shape: pin the main/count pair per state (prelude, import with/without total, overview, summaries with/without folder, upload, idle, settle) AND the `title` / `#sync-result` untruncated concatenation (the A4 pins).
|
||||
|
||||
ASSUMPTION (A1 restated): the truncated TODO item is satisfied by "label fully visible on mobile + count never clipped" — no wrapping, no progress bar.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the updated pins above green; no HTML strings anywhere in the label path (textContent only).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] every running/idle/settle label renders the SAME full string as pre-task on desktop (the part concatenation is byte-identical)
|
||||
- [ ] the count span is `hidden` whenever there is no `(n/m)` to show
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,46 @@
|
||||
# Task 02 — CSS: the label parts flex — path ellipsizes, count pinned, mobile fully visible
|
||||
|
||||
**Phase:** `128_sync_label_mobile` · **Source:** `TODO.md:4` — "The `Syncing… name/of/file/here (2/3)` doesn't show up on mobile, it needs to" (A1: fully visible on mobile, count never clipped)
|
||||
|
||||
## Objective
|
||||
The label's two parts are laid out so the path can ellipsize against whatever width is available — and on a phone, where the pill is full-width, the path ellipsizes and the `(n/m)` count sits pinned at the right edge, always on-screen.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/styles.css` — replace the `.sync-label` rule (~L2067, the phase-64 comment block above it stays, updated to name the parts):
|
||||
```css
|
||||
.sync-label {
|
||||
display: inline-flex;
|
||||
align-items: baseline;
|
||||
gap: 0.35rem;
|
||||
min-width: 0;
|
||||
max-width: min(16rem, 40vw);
|
||||
}
|
||||
.sync-label-main {
|
||||
min-width: 0; /* the flex item that shrinks — what engages the ellipsis */
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.sync-label-count {
|
||||
flex: 0 0 auto; /* NEVER clipped — the (n/m) position is the point */
|
||||
white-space: nowrap;
|
||||
font-family: var(--mono); /* the .sync-result mono language (position reads as data) */
|
||||
}
|
||||
```
|
||||
(The old `display: inline-block; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; vertical-align: bottom;` moves to `.sync-label-main` — `vertical-align` is gone: the flex row aligns it.)
|
||||
2. `frontend/assets/styles.css` — the mobile media query block (~L5323, the "full-width LABELED pills" comment): update the two sync rules:
|
||||
```css
|
||||
.sync-btn { width: 100%; }
|
||||
.sync-label { max-width: none; flex: 1 1 auto; min-width: 0; }
|
||||
```
|
||||
— the label now fills the full-width pill; `.sync-label-main` (min-width: 0 inside a flex row that can shrink) ellipsizes against the width left by the pinned count; the count stays visible at the right edge. Do NOT touch the `#sync-result` visually-hidden clip, the `#sync-btn` aria-label, or the other mobile rules in that block.
|
||||
3. Audit: grep for every `.sync-label` reference in `styles.css` — the only rules are the base + this mobile block (the phase-64/98 history is in comments). Confirm no other stylesheet (theme override, page-scoped) styles the label; the built-in theme must stay byte-identical in the no-row case (B4 — no `ui_settings` dependency here, this is static CSS).
|
||||
|
||||
## Testing & Quality
|
||||
- The visual gate is the E2E (task 03) — desktop vs mobile bounding-box + text assertions; no new Python surface.
|
||||
- Coverage: **>90%** on new/modified code (CSS has no direct coverage — the E2E + unit pins carry it).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] desktop: long paths still ellipsize inside the `min(16rem, 40vw)` cap, count visible after the ellipsis
|
||||
- [ ] mobile (390 px): label fills the pill, path ellipsized, `(n/m)` on-screen at the right edge (the E2E asserts bounding boxes)
|
||||
- [ ] `uv run pytest` green; no behavior change in completed phases
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 03 — E2E: the running sync label is fully visible with its count on mobile (and unchanged on desktop)
|
||||
|
||||
**Phase:** `128_sync_label_mobile` · **Source:** `TODO.md:4` — "The `Syncing… name/of/file/here (2/3)` doesn't show up on mobile, it needs to" (A1: fully visible on mobile, count never clipped)
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) proving the label's geometry: at a phone viewport the rendered `#sync-label` fits the viewport with the `(n/m)` count span visible in BOTH long spans (import + summaries); at a desktop viewport the label keeps its pre-phase strings and cap.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_label_mobile.py` (new) — follow `tests/e2e/test_sync_summary_visibility.py`'s house pattern (module app fixtures over the shared E2E Postgres, host temp-dir `kind=local` source, autouse truncate cleanup, mock-only). Use the SLOW app (the `tests/e2e/slow_llm.py` proxy, `SLOW_DELAY_S = 1.5` as there) so each phase span outlives the UI's 2 s poll — the rendered-label assertions are deterministic, not races.
|
||||
KB fixture — one local source `synclabel/` with TWO files in `alpha/` and TWO in `bravo/` (the summaries phase then has 3 candidate folders: root + 2 — a non-trivial `(n/m)`).
|
||||
2. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_mobile_label_fits_viewport_with_count_in_import` — admin page on the sources view at viewport **390×844** (mobile); click **Sync sources**; while the import phase is running (the page's own 2 s poll): the rendered `#sync-label`'s bounding box is INSIDE the viewport (left ≥ 0, right ≤ 390) — the bug's direct negation; `.sync-label-count` is visible (`boundingBox` non-null, not clipped: its right edge ≤ the button's right edge) with text matching `\(\d+/\d+\)` once a file is in flight; `.sync-label-main` carries `Syncing… <file>` (the file part may be visually ellipsized — assert the button `title` carries the UNTRUNCATED label, A4).
|
||||
- `test_mobile_label_shows_summaries_count` — same mobile viewport; through to the summaries phase: `.sync-label-main` matches `Summarizing folders…` (folder part optional), `.sync-label-count` matches `\(\d+/3\)` and stays visible; settle: the button leaves the running label (`Synced HH:MM` or the phase-32 terminal shape), the count span hidden/empty.
|
||||
- `test_desktop_label_unchanged` — viewport 1280×800; same sync: during import the FULL rendered label text (main + count concatenation) matches `Syncing… <full source/relative/path> (n/m)` — the pre-phase byte-identical string — and the label's width respects the `min(16rem, 40vw)` cap (bounding box width ≤ 256 px + slack); `title` carries the same untruncated string.
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_sync_label_mobile.py -v --no-cov`), the mock-only + slow-proxy rationale (re-sized from the phase-98 suite's comment), the fixture layout, the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% bar is carried by task 01's updated unit pins (frontend JS); this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_label_mobile.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the mobile tests fail against the pre-phase markup/CSS (the count ran off-screen) — i.e. the suite actually pins the fix
|
||||
- [ ] no behavior change in completed phases (label strings, phase machine, aria-live)
|
||||
@@ -0,0 +1,36 @@
|
||||
# Phase 129 — The sync label shows what chunk we're on out of how many
|
||||
|
||||
**Source:** `TODO.md` L5 — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (ASSUMPTION A6, owner-confirmed 2026-09-27: "chunk" = the current file's EMBEDDING CHUNKS; position reported at BATCH granularity — `n` = first chunk of the in-flight token-budgeted embed batch — because the transport embeds batches per HTTP call and per-chunk calls would be a latency regression; the summary-chunk embed is excluded from the counter; label `Syncing… <file> (2/3) · chunk 14/38`).
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append; a NEW feature — distinct from phase 128's mobile fix, per the owner).
|
||||
**Context:** `app/rag/llm.py` — `LLMClient.embed(texts)` (L346: iterates `self._batch_texts(texts)` — token-budgeted batches, `BOR_EMBED_BATCH_SIZE` cap, `_REQUEST_TOKEN_BUDGET` ≈ 2800 chars), `_embed_batch` (L289: the halving backstop on `_TooLarge`), `self.embed_batches` counter (L249); `app/rag/importer.py` — `import_sources` (L408, the per-file `progress(source, rel, done, total)` hook fired BEFORE each `_index_file`), `_index_file` (L647 — text branch → `_replace_content_chunks`, image branch → `_index_image_file`, PDF branch → `_index_pdf_file`), `_replace_content_chunks` (L603 — chunks the content, `llm.embed([c.content for c in doc.chunks])` in ONE call, token-cap re-chunk retry loop), `_store_summary` (embeds the ONE summary chunk — EXCLUDED from the counter); `app/api/sync.py` — `SyncStatus` (the phase-64/98 progress fields: `current_file`, `files_done/total`, `phase`, `current_summary`, `summaries_done/total`; `_run_sync`'s `_hook` closure convention; terminal-state clearing); `frontend/assets/sources.js` — the phase-128 part builders (`setSyncLabel`, `.sync-label-main` / `.sync-label-count`); `tests/e2e/slow_llm.py` + `tests/e2e/test_sync_summary_visibility.py` — the slow-proxy sampling pattern.
|
||||
|
||||
## Objective
|
||||
While a file is being imported, the sync status (endpoint + button label) shows the file's embedding-chunk position: `Syncing… <file> (2/3) · chunk 14/38` — 14 = the first chunk of the embed batch in flight, 38 = the file's total content chunks. The position climbs through the file's batches and is cleared when the file (or the run) moves on. Everything that passes no callback stays a zero-cost no-op (the CLI, the chat path).
|
||||
|
||||
## Dependencies
|
||||
- `128_sync_label_mobile` (todo) — the label's part structure (`.sync-label-main` + pinned count span) is the carrier; this phase adds the pinned chunk span.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The embed hook (task 01):** `LLMClient.embed(texts, on_progress: Callable[[int, int], None] | None = None)` — before EACH top-level batch request, `on_progress(first_1based, len(texts))` where `first_1based` = 1 + the cumulative chunk counts of the already-sent batches. `None` (every existing caller — the chat retrieval path, `embed_one`, the summarizer, the CLI) is a zero-cost guarded no-op. The `_embed_batch` halving backstop does NOT sub-report — the top-level loop reports batch starts; a halved batch means the position is approximate (document in the docstring; the backstop is rare by construction).
|
||||
- **Importer plumbing (task 02):** `import_sources(..., on_chunk_progress: Callable[[int, int], None] | None = None)` — a new keyword-only param, plumbed: `import_sources` → `_index_file(..., on_chunk=...)` (all three branches: text, image, PDF — the PDF branch's content embed is the same `_replace_content_chunks` call; the image branch's ONE description-chunk embed reports `1/1` — acceptable, it IS the file's only chunk) → `_replace_content_chunks(..., on_chunk=...)` → `llm.embed([...], on_progress=on_chunk)`. The token-cap re-chunk retry re-embeds with the NEW (smaller-target) chunk list — the hook naturally reports against the new total (no special case). `_store_summary`'s single summary-chunk embed NEVER receives the hook (the counter is content chunks only — no `1/1` noise after the real progress ends). The `scripts/import_docs.py` CLI passes nothing — byte-identical.
|
||||
- **Sync status (task 03):** `SyncStatus` gains `current_chunk: int = 0` and `chunks_total: int = 0` (documented: the current file's embed-batch position — 0/0 before the first batch and in terminal states). `_run_sync`: reset to 0/0 at run start (with the other progress fields); a closure `_chunk_hook(chunk, total)` (the file's `_hook` convention) sets them; passed as `on_chunk_progress` to `import_sources`. CLEARED (0/0) when the import phase ends — set at the `phase = "overview"` and `phase = "summaries"` transitions AND in both terminal states (the `current_file` clearing points — same convention, same spots). `GET /api/sync/status` exposes both fields (0/0 idle — the dataclass defaults). The `SyncStatus` docstring + module docstring get the chunk-position paragraph (the phase-64/98 style).
|
||||
- **UI (task 04):** `sources.js` — the import-phase label gains a THIRD pinned part `.sync-label-chunk` (markup in `index.html`'s `#sync-label`, `hidden` by default): when `status.chunks_total > 0` the import fall-through emits `· chunk ${status.current_chunk}/${status.chunks_total}` into it; every other phase/state leaves it hidden/empty (the summaries phase has its own `(n/m)` — no chunk part; the prelude has no file yet). The UNTRUNCATED concatenation (main + count + chunk part) rides the button `title` + `#sync-result` (A4). The pinned unit tests (`tests/unit/test_frontend_sync_upload.py`) gain the chunk-part pins.
|
||||
|
||||
## Tasks
|
||||
1. `01_llm_embed_hook.md` — `LLMClient.embed`'s optional per-batch `on_progress`.
|
||||
2. `02_importer_chunk_progress.md` — `import_sources` → `_index_file` → `_replace_content_chunks` → `embed` plumbing.
|
||||
3. `03_sync_status_chunk_fields.md` — `SyncStatus` fields, the closure, clearing, the endpoint.
|
||||
4. `04_ui_chunk_part.md` — the label's chunk part + unit pins.
|
||||
5. `05_e2e_chunk_progress.md` — dedicated Playwright E2E, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: the embed hook's call sequence (batch boundaries, `None` no-op, halving-approximate documented); the importer plumbing (hook fires per file with its own total; the CLI path unchanged; the summary-chunk embed never fires it); the sync status (fields in the endpoint JSON, reset per run, cleared at phase transitions + terminal, 0/0 idle).
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
- E2E: one dedicated Playwright file, run in isolation, slow-LLM sampling.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/sync/status` reports `current_chunk` climbing 1 → N with `chunks_total` = the file's true content-chunk count while a file imports (E2E, sampled)
|
||||
- [ ] the rendered mobile/desktop label shows `· chunk n/m` during the import phase only; the untruncated text rides `title` + `#sync-result`
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (chat embed path, CLI import, the phase-64/98 label contract minus the new part)
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 01 — `LLMClient.embed`'s optional per-batch `on_progress` hook
|
||||
|
||||
**Phase:** `129_sync_chunk_progress` · **Source:** `TODO.md:5` — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (A6: batch-granularity chunk position)
|
||||
|
||||
## Objective
|
||||
`embed` can report, before each top-level batch request, which chunk of the input list starts that batch — `on_progress(first_1based, total)` — so a caller can render `chunk n/m`. No callback → zero-cost no-op, byte-identical behavior for every existing caller.
|
||||
|
||||
## Work
|
||||
1. `app/rag/llm.py` — `LLMClient.embed` (L346): new keyword-only param `on_progress: Callable[[int, int], None] | None = None`. In the batch loop, BEFORE `await self._embed_batch(chunk)` (and before the try/except, so a failing batch still announced its start):
|
||||
```python
|
||||
if on_progress is not None:
|
||||
on_progress(len(out) + 1, len(texts)) # first chunk of THIS batch (1-based), total
|
||||
```
|
||||
(`len(out)` = chunks already embedded in earlier batches — the accumulator's length is exactly that, since each batch's vectors extend it 1:1.)
|
||||
Docstring: the contract — called once per TOP-LEVEL batch, before the request, with `(first_chunk_1based, total_chunks)`; the `_embed_batch` halving backstop does not sub-report (a halved batch makes the position approximate — documented; the backstop is rare by construction, phase-114 `_TooLarge` handling); `None` is a zero-cost guarded no-op (every pre-task caller — chat retrieval, `embed_one`, the summarizer, the importer — passes nothing and is byte-identical).
|
||||
2. Do NOT touch `_batch_texts` / `_embed_batch` internals, `embed_batches` counting, or `embed_one`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/` (extend the existing LLM-client test file — inspect `tests/unit/test_llm*.py` first):
|
||||
- N texts sized to force ≥ 3 batches (shrink the token budget via settings or pick chunk lengths against `_REQUEST_TOKEN_BUDGET`) → `on_progress` called with `(1, N)`, `(k2, N)`, … `total` always `len(texts)`, first-indices strictly increasing, each = previous first + previous batch size (assert against `_batch_texts`'s actual partition — call it directly to get the expected boundaries);
|
||||
- `on_progress=None` → the call sequence and return value are identical to pre-task (the existing suite is the pin);
|
||||
- empty `texts` → no hook calls (the early return).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the hook fires per top-level batch with the correct `(first, total)` pair (unit-pinned)
|
||||
- [ ] every existing `embed`/`embed_one` call site is untouched and its tests green
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 02 — Importer plumbing: `on_chunk_progress` from `import_sources` to the content-chunk embed
|
||||
|
||||
**Phase:** `129_sync_chunk_progress` · **Source:** `TODO.md:5` — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (A6: batch-granularity chunk position)
|
||||
|
||||
## Objective
|
||||
`import_sources` accepts an optional `on_chunk_progress` callback that fires with the CURRENT FILE's content-chunk position while that file's chunks embed — so the sync status can attribute the position to the file its per-file `progress` hook already named.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — `import_sources` (L408): new keyword-only param `on_chunk_progress: Callable[[int, int], None] | None = None` (documented in the docstring next to the `progress` paragraph: same "only when provided — `None` callers pay nothing" rule; it carries the in-flight FILE's chunk position, not the file identity — the per-file `progress` hook owns that).
|
||||
2. `app/rag/importer.py` — the processing loop's `_index_file` call: pass `on_chunk=on_chunk_progress` (a new keyword-only param on `_index_file`, default `None`).
|
||||
3. `app/rag/importer.py` — `_index_file`:
|
||||
- the TEXT branch → `_replace_content_chunks(session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk)`;
|
||||
- the IMAGE branch (`_index_image_file`) and the PDF branch (`_index_pdf_file`) → pass `on_chunk` through to THEIR `_replace_content_chunks` calls (the image branch's one description-chunk embed reports `1/1` — it IS the file's only chunk; the PDF branch's content embed is the primary target). Verify each branch's embed path: if a branch embeds chunks somewhere other than `_replace_content_chunks`, route the hook there too (one hook per branch, content chunks only).
|
||||
4. `app/rag/importer.py` — `_replace_content_chunks` (L603): new keyword-only param `on_chunk: Callable[[int, int], None] | None = None`, forwarded to `llm.embed([c.content for c in doc.chunks], on_progress=on_chunk)`. The token-cap re-chunk retry loop (the `while True` that halves `target`) re-embeds with the new chunk list — the hook reports against the NEW total automatically (no special case; document it in the docstring). The `doc.chunks = []` empty-content `break` never embeds (no hook fire).
|
||||
5. `_store_summary` (and its summary-chunk `embed_one`/`embed` call) NEVER receives the hook — the counter is content chunks only (A6: no `1/1` noise after the real progress ends). The `scripts/import_docs.py` CLI passes nothing — byte-identical (verify the call site).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_importer.py` (extend) —
|
||||
- a text file that chunks into ≥ 2 batches (size the fixture against the token budget, or monkeypatch `_batch_texts`) with a fake `on_chunk_progress` recorder → the recorder sees `(first, total)` pairs with `total == the file's chunk count`, one batch-run per file, and a SECOND file in the same run gets its OWN sequence (the position restarts per file — the recorder's `(first, total)` pairs never mix the two files' totals);
|
||||
- the image branch (images toggle on, a fixture image) → the recorder sees exactly `(1, 1)` for that file;
|
||||
- `on_chunk_progress=None` (the CLI shape) → no recorder, behavior byte-identical (the existing suite is the pin);
|
||||
- the summary-chunk embed never fires the hook (the recorder's call count == the content-embed batches only).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a multi-batch file's embed position is fully observable through `import_sources(on_chunk_progress=…)` (unit-pinned)
|
||||
- [ ] the CLI path and the summary-chunk embed are untouched (byte-identical, verified by the existing suite)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 03 — `SyncStatus` chunk fields: `current_chunk` / `chunks_total` on the status endpoint
|
||||
|
||||
**Phase:** `129_sync_chunk_progress` · **Source:** `TODO.md:5` — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (A6: batch-granularity chunk position)
|
||||
|
||||
## Objective
|
||||
`GET /api/sync/status` carries the in-flight file's embed-chunk position (`current_chunk`, `chunks_total`) so the UI can render `· chunk n/m` — reset per run, cleared when the import phase ends, 0/0 idle (the phase-64/98 field conventions).
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py` — `SyncStatus` dataclass: add
|
||||
```python
|
||||
# Phase 129: the current file's EMBED chunk position — the first
|
||||
# chunk of the in-flight token-budgeted batch and the file's total
|
||||
# content chunks (the LLMClient.embed hook, batch granularity).
|
||||
# 0/0 before the first batch, during the prelude/overview/summaries
|
||||
# phases, and in terminal states.
|
||||
current_chunk: int = 0
|
||||
chunks_total: int = 0
|
||||
```
|
||||
Update the `SyncStatus` docstring (the phase-64/98 progress-field paragraph) to name the new fields and their clearing contract.
|
||||
2. `app/api/sync.py` — `_run_sync`:
|
||||
- run start: `_status.current_chunk = 0`, `_status.chunks_total = 0` (with the other progress-field resets);
|
||||
- the closure (the file's `_hook` convention — closures capture the module `_status`):
|
||||
```python
|
||||
def _chunk_hook(chunk: int, total: int) -> None:
|
||||
_status.current_chunk = chunk
|
||||
_status.chunks_total = total
|
||||
```
|
||||
passed to `import_sources(..., on_chunk_progress=_chunk_hook)` (alongside the existing `progress=_hook`);
|
||||
- CLEARING (0/0) at every point the import phase ends — the `_status.phase = "overview"` assignment, the `_status.phase = "summaries"` assignment, and BOTH terminal states (the success and except blocks where `_status.current_file = None` — the phase-64 keep-final-FILE-counts convention does NOT extend to the sub-file chunk position: the file counts survive the terminal for the result line's context, the chunk position is a sub-position of a file that's done — it clears, like `current_file`);
|
||||
- the unchanged-walk / no-sync-idle paths never set them (defaults 0/0).
|
||||
3. `app/api/sync.py` — `sync_status` endpoint: add `"current_chunk": _status.current_chunk, "chunks_total": _status.chunks_total` to the returned dict (place them after the `files_*` fields; the dataclass defaults make idle 0/0 with no special-casing). Update the endpoint docstring's field list (the phase-98 paragraph) and the module docstring's status paragraph with the chunk-position contract.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: extend the sync unit suite (the file that fakes `import_sources` — inspect `tests/unit/test_sync_button.py` and the sync-pipeline test first):
|
||||
- idle → `current_chunk == 0`, `chunks_total == 0` in the JSON (the dataclass defaults — the endpoint shape test);
|
||||
- a running sync whose fake `import_sources` invokes `on_chunk_progress(1, 38)` then `(13, 38)` → the status between ticks reports the pair (assert on `_status` directly or via the endpoint through the existing app fixture);
|
||||
- phase transition to `overview`/`summaries` → the fields are 0/0 again; terminal success/failed → 0/0 while `files_done`/`files_total` keep their final values (the convention split, pinned).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the status endpoint JSON carries both fields; idle is 0/0; a running import reports the hook's pair
|
||||
- [ ] the fields clear at phase transitions + terminal states while the file counts keep-final (pinned test)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 04 — The label's chunk part: `Syncing… <file> (2/3) · chunk 14/38`
|
||||
|
||||
**Phase:** `129_sync_chunk_progress` · **Source:** `TODO.md:5` — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (A6: batch-granularity chunk position)
|
||||
|
||||
## Objective
|
||||
The running sync label shows the chunk position as a third pinned part — rendered ONLY while `chunks_total > 0` (the import phase, after the file's first batch), hidden in every other state — and the untruncated full label (main + count + chunk part) keeps riding the button `title` + `#sync-result` (A4).
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` — `#sync-label` (the phase-128 two-part markup): add the third part after the count span: `<span class="sync-label-chunk" hidden></span>`.
|
||||
2. `frontend/assets/styles.css` — `.sync-label-chunk { flex: 0 0 auto; white-space: nowrap; font-family: var(--mono); }` (the same pinned-part language as `.sync-label-count` — place it right after that rule; both mono parts read as data).
|
||||
3. `frontend/assets/sources.js` —
|
||||
- get the chunk part at `mount` scope (the phase-128 `querySelector` pattern, null-guarded);
|
||||
- `setSyncLabel(mainText, countText = "", chunkText = "")` — the new third arg: when `chunkText`, set `.sync-label-chunk.textContent` + `hidden = false`, else `""` + `hidden = true`; the UNTRUNCATED concatenation for `title` + `#sync-result` becomes `mainText` + (count? `" " + countText`) + (chunk? `" " + chunkText`) (A4 — the announcer hears the whole `Syncing… <file> (2/3) · chunk 14/38`);
|
||||
- the import fall-through (the `fmtSyncPhaseLabel` fall-through / the 202/409 click label site — everywhere the import label is emitted with a status): `chunkText = status.chunks_total > 0 ? `· chunk ${status.current_chunk}/${status.chunks_total}` : ""` — every OTHER state (prelude, overview, summaries, upload, idle, settle) passes no chunk text (the part hides): the summaries phase's own `(n/m)` count is its position — no chunk part there;
|
||||
- the `title`/`#sync-result` set on EVERY running label (the house rule) automatically carries the chunk part via the concatenation.
|
||||
4. `tests/unit/test_frontend_sync_upload.py` — extend the phase-128 part pins: the import state with `chunks_total > 0` emits the chunk part (`· chunk 14/38` for the fixture values) and the untruncated `title` string is `Syncing… <file> (2/3) · chunk 14/38`; the import state with `chunks_total == 0` (before the first batch) hides the part; every other state's pins stay (the part hidden).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above green; textContent-only (no HTML strings in the label path).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `· chunk n/m` renders only while `chunks_total > 0`; hidden in all other states (unit-pinned)
|
||||
- [ ] the untruncated full label (with the chunk part) rides `title` + `#sync-result` (A4, unit-pinned)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed phases (phase-128 pins still pass)
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 05 — E2E: the chunk position climbs through a large file's embed batches, endpoint + label
|
||||
|
||||
**Phase:** `129_sync_chunk_progress` · **Source:** `TODO.md:5` — "The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many" (A6: batch-granularity chunk position)
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) proving the feature at both surfaces: the endpoint's `current_chunk` climbs 1 → N with `chunks_total` = the file's true chunk count, and the rendered label shows `· chunk n/m` while the file's embed span runs (then the part clears).
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_chunk_progress.py` (new) — house pattern from `tests/e2e/test_sync_summary_visibility.py` (module app fixtures over the shared E2E Postgres, host temp-dir `kind=local` source, autouse truncate cleanup, mock-only, the `tests/e2e/slow_llm.py` proxy).
|
||||
Sizing (the phase-98 rationale, re-sized for batches): the fixture is ONE local source `chunkprog/` holding ONE large markdown file (`big.md`, ~60 KB, ~30+ chunks at the default target — compute the expected chunk count deterministically in the test by calling `chunk_document` with the app's settings, the same function the importer uses — and assert `chunks_total` equals it) plus ONE tiny second file (the file counter then reads `(1/2)` → `(2/2)`, proving the chunk position restarts per file). `SLOW_DELAY_S` sized so each embed batch outlives the recorder's ~100 ms cadence (the proxy delays EVERY LLM call — embeds included — so the import span is densely sampled).
|
||||
2. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_endpoint_reports_the_chunk_position` (the slow leg — tight-poll recorder on `GET /api/sync/status`, the phase-64 recorder pattern): while `phase == "import"`: some ticks have `chunks_total == <expected chunk count>` and `current_chunk` strictly climbing from 1 toward `chunks_total` (batch granularity — the sequence's LAST tick before the file ends is `chunks_total` or its final batch's first index; assert the climb + the total, not a contiguous run); while the second (tiny) file is in flight the position restarts (a later tick has `current_chunk == 1` with `chunks_total == 1`); the terminal state has `current_chunk == 0` and `chunks_total == 0` while `files_done == files_total == 2` (the keep-final-file-counts split, the phase-129 convention).
|
||||
- `test_label_shows_the_chunk_part` (the slow leg — UI, the page's own 2 s poll): at the import span the rendered `#sync-label` parts: `.sync-label-main` matches `Syncing… chunkprog/big.md`, `.sync-label-count` matches `(1/2)`, and at some tick `.sync-label-chunk` is visible matching `· chunk \d+/\d+` with the total == the expected chunk count; the button `title` carries the UNTRUNCATED concatenation (A4 — assert it contains `· chunk` on the same tick); after the file's embed span the chunk part is hidden (`.sync-label-chunk.hidden` true) while the file label continues.
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov`), the mock-only gate, the sizing rationale (why the file is ~60 KB, why the slow proxy's delay doubles as the batch sampler), the fixture layout, the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% `app/` bar is carried by tasks 01–03; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the endpoint's `chunks_total` equals the deterministic `chunk_document` count for the fixture (the total is the file's TRUE chunk count, not a batch count)
|
||||
- [ ] the chunk part renders only during the import phase and clears at the phase transition/terminal (UI-pinned)
|
||||
- [ ] no behavior change in completed phases (the phase-64/98 label contract + the keep-final-counts convention hold)
|
||||
@@ -0,0 +1,36 @@
|
||||
# Phase 130 — Chat answer clusters: chips row, images row, bottom-right actions cluster
|
||||
|
||||
**Source:** `TODO.md` L6 — "The chat output is getting messy. I think it's because the `source-chip`, `source-image`, `tune-btn`, `retry-btn`, and `save-as-doc-btn` end up in the same div which causes them to push each other in weird ways. Those three buttons should be in a button cluster in the bottom right. We don't need to display the `source-chip` for images, just the image itself is fine. `source-chips` should show up before images in their own cluster, then images in their own cluster to keep things organized."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `frontend/assets/app.js` — `appendSources` (~L1561: builds the FIRST `.msg-meta` row, interleaving a `.source-image` figure after each chip for refs carrying `image_url`), `appendRelated` (~L1618: the "Nearby docs" row, appended LAST among meta rows), `appendSourceImageFigure` (~L1660: the compact figure; on `img` error it removes the figure and the chip remains), `appendAnswerImage` (~L1735: the `.answer-image` figure — appended by callers right BEFORE `appendSources`), `appendTuneButton` (~L484, admin-gated), `appendRetryButton` (~L598), `appendSaveAsDocButton` (~L793, admin + docs-repo gated; `margin-inline-start: auto` pushes it right), `markLastRetryable` (~L620: re-appends the Retry button on the last bubble and re-appends the save button after it — "the auto margins split the free space"), `appendStoppedNote` / `appendFailedNote` (~L520/~L560: join the first `.msg-meta` row); `frontend/assets/styles.css` — the `.msg-meta`, `.source-chip`, `.source-image`, `.tune-btn`, `.retry-btn`, `.save-as-doc-btn` rules (grep each — the auto-margin rules live here); `frontend/index.html` — no chat markup changes expected (the bubble DOM is JS-built); E2E image-KB fixture patterns in `tests/e2e/test_chat_image_questions.py` / `test_image_documents.py` (the mock's `IMAGE_DESCRIPTION_MODE` canned vision reply makes image docs retrievable).
|
||||
|
||||
## Objective
|
||||
A completed brain bubble's metadata reads as organized clusters, top to bottom: the answer text → the answer figure (phase 124, unchanged) → the **source chips row** (text documents only — an image document gets NO chip, just its figure) → the **source images row** (all retrieved image figures) → the "Nearby docs" row (unchanged) → the **actions row** (a right-aligned cluster holding Tune / Retry / Save-as-doc and the Stopped/Failed notes) — never one soup div where buttons push chips around.
|
||||
|
||||
## Dependencies
|
||||
- `129_sync_chunk_progress` (todo) — pipeline order only; code dependencies are the completed chat-render contract (phases 26/49/59/113/120/122/124).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Row structure per brain bubble (top → bottom):** (1) answer text; (2) `.answer-image` figure — the phase-124 position is UNCHANGED (below the text, before the source rows — A3); (3) `.msg-meta` chips row (`role=list`, `aria-label="Sources"`) — one `.source-chip` per NON-image ref; refs with `image_url` contribute NO chip (ASSUMPTION A3 — the figure is the citation for image docs); (4) `.msg-meta.msg-images` images row (`role=list`, `aria-label="Source images"`) — one `.source-image` figure per image ref, in ref order; the row is absent when there are no image refs; (5) `.msg-meta.related-docs` — the phase-113 row, UNCHANGED in content, placed after the images row (ASSUMPTION A4); (6) `.msg-meta.msg-actions` actions row (`role=list`, `aria-label="Answer actions"`) — the ONLY row the action buttons + the stopped/failed notes join (ASSUMPTION A7).
|
||||
- **Call order at every render site** (the `done` branch, the empty-answer fallback, the stop finalize, the phase-14 restore path): answer image → `appendSources` (now builds the chips row AND the images row) → `appendRelated` → the action buttons/notes. The current "buttons claim the FIRST `.msg-meta` row" behavior is REPLACED — the buttons get-or-create `.msg-actions` instead.
|
||||
- **Image-load failure (ASSUMPTION A3):** `appendSourceImageFigure`'s `img` error handler replaces the figure IN PLACE with a plain `.source-chip` anchor for that document (same `documentUrl` href, same modal click behavior, the label as text) — the citation survives a lost byte copy; no broken-image icon.
|
||||
- **Actions row:** one get-or-create `.msg-actions` per bubble (the house pattern). DOM order = Tune, Retry, Save-as-doc (each present per its gate — non-admins see only Retry; a stopped bubble sees the note + Retry). The note (`appendStoppedNote` / `appendFailedNote`) joins this row. `markLastRetryable`'s re-append logic moves the Retry button into `.msg-actions` and DROPS the save-button re-append hack (row-level alignment replaces the per-button `margin-inline-start: auto` — CSS task). ARIA: `role=list` + `role=listitem` on every child (the existing convention).
|
||||
- **CSS:** `.msg-actions { justify-content: flex-end; }` (the right-aligned cluster — "bottom right" = the bubble's last row, right edge); remove the per-button auto margins (`.save-as-doc-btn`'s `margin-inline-start: auto` + the tune/retry auto margins the `markLastRetryable` comment names); the chips row keeps its wrap; `.msg-images` wraps with the `.msg-meta` gap; the row ORDER is DOM order (no positioning tricks); mobile: rows stack naturally, the actions row stays right-aligned.
|
||||
|
||||
## Tasks
|
||||
1. `01_sources_clusters.md` — `appendSources`/figure split: chips-only row + images-only row, no chip for image docs, the in-place chip fallback.
|
||||
2. `02_actions_cluster.md` — the `.msg-actions` row: the three buttons + the notes, `markLastRetryable` rework, ARIA kept.
|
||||
3. `03_css_clusters.md` — the cluster CSS (actions right-aligned, auto margins removed, images row).
|
||||
4. `04_e2e_clusters.md` — dedicated Playwright E2E, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the frontend unit suite — the pinned behaviors that move (the button-row target, the no-chip-for-images rule, the fallback) — inspect `tests/unit/test_frontend_*.py` for the chat-render pins and update them.
|
||||
- E2E: one dedicated Playwright file, run in isolation, image-KB fixture (text doc + image doc both cited).
|
||||
- Coverage: **>90%** on new/modified code (frontend: unit pins + E2E; `app/` untouched this phase).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a bubble citing a text doc + an image doc renders: chips row (text chip only, NO chip for the image) → images row (the figure) → actions row (bottom right) — asserted in the DOM, not the pixels
|
||||
- [ ] Tune/Retry/Save-as-doc sit in ONE `.msg-actions` row, right-aligned; non-admin sees only Retry; the restore path renders the identical structure
|
||||
- [ ] an image whose bytes 404 renders as its plain chip (the citation survives)
|
||||
- [ ] `uv run pytest tests/e2e/test_chat_answer_clusters.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (chips still open the modal, the related row, the answer figure's position, the ARIA names)
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 01 — Sources split: chips-only row, then images-only row; no chip for image docs
|
||||
|
||||
**Phase:** `130_chat_answer_clusters` · **Source:** `TODO.md:6` — "...We don't need to display the `source-chip` for images, just the image itself is fine. `source-chips` should show up before images in their own cluster, then images in their own cluster to keep things organized."
|
||||
|
||||
## Objective
|
||||
`appendSources` stops interleaving figures after chips. It builds the chips row from the NON-image refs only, then (when any image ref exists) a separate images row holding all the figures — chips cluster first, images cluster second, image documents cited by their figure alone.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — `appendSources(wrap, sources)`:
|
||||
- Partition the refs: `imageRefs = sources.filter(s => s.image_url)`, `textRefs = sources.filter(s => !s.image_url)` (the `image_url` presence is the existing image-doc signal — phase 122).
|
||||
- CHIPS ROW: build it (the existing `.msg-meta` + `role=list` + `aria-label="Sources"` + per-chip `documentUrl`/modal click/`title`/clipped `aria-label` code) from `textRefs` ONLY. Skip the row entirely when `textRefs` is empty (an image-only citation still gets its images row + the actions row).
|
||||
- IMAGES ROW: when `imageRefs.length > 0`, build a SECOND `.msg-meta` row — add the `msg-images` class, `role=list`, `aria-label="Source images"` — and append one `appendSourceImageFigure` per image ref (in ref order). No chips in this row.
|
||||
- Preserve the existing post-build clipped-chip `aria-label` pass (the `scrollWidth > clientWidth` check) over the chips row only.
|
||||
- The row ORDER in `.msg-body` is: answer figure (appended by the caller before `appendSources`, phase 124 — unchanged) → chips row → images row. `appendRelated` (the next caller) lands after both (task 02's call-order change guarantees the actions row is last).
|
||||
2. `frontend/assets/app.js` — `appendSourceImageFigure(meta, s)`: the `img` error handler currently does `fig.remove()` (the chip remains — but there is no chip for image docs anymore). Change it to the IN-PLACE CHIP FALLBACK: build a plain `.source-chip` anchor for `s` (the same `documentUrl` href + `title` + left-click → modal handler the chips row uses; `textContent`/`aria-label` = the `source/path` label) and `replaceWith` the figure (the citation survives a lost byte copy — ASSUMPTION A3). The figure's fetch-summary swap is untouched.
|
||||
3. `frontend/assets/app.js` — `appendRelated`: the related tier's refs ALSO carry `image_url` for image docs (phase 122 — `appendSourceImageFigure(row, s)` is called there today). Keep the related row's own behavior (its chips + figures) — the related row is a separate tier and is NOT reorganized by this phase (ASSUMPTION A4: it stays where it is, below the citation clusters). Do not change `appendRelated`'s chip/figure logic.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the frontend unit suite — inspect `tests/unit/test_frontend_*.py` for any `appendSources`/figure pins and update: no chip emitted for an `image_url` ref; the images row is built only when image refs exist; the in-place fallback replaces the figure with a chip (the figure's `img` error path).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a mixed citation (1 text + 2 image refs) renders a 1-chip row then a 2-figure row — no chip for either image
|
||||
- [ ] an image-only citation renders NO chips row and a figures-only images row
|
||||
- [ ] a figure whose bytes 404 becomes its plain chip in place (the citation survives)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,23 @@
|
||||
# Task 02 — The actions row: Tune / Retry / Save-as-doc + the notes in one bottom-right cluster
|
||||
|
||||
**Phase:** `130_chat_answer_clusters` · **Source:** `TODO.md:6` — "...Those three buttons should be in a button cluster in the bottom right."
|
||||
|
||||
## Objective
|
||||
The three action buttons (and the Stopped/Failed notes) stop joining whatever the first `.msg-meta` row happens to be. They get-or-create a dedicated `.msg-actions` row — the bubble's LAST row, right-aligned by CSS (task 03) — so chips and images can never push the buttons around.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — a shared helper next to the button appenders: `getActionsRow(wrap)` — find or create the bubble's `.msg-meta.msg-actions` row inside `.msg-body` (`role=list`, `aria-label="Answer actions"`), the house get-or-create pattern (the same shape as the current "reuse the `.msg-meta` row when it exists" code, but scoped to the `msg-actions` class so it NEVER grabs the chips row).
|
||||
2. `frontend/assets/app.js` — `appendTuneButton` (~L484), `appendRetryButton` (~L598), `appendSaveAsDocButton` (~L793): replace their "reuse the first `.msg-meta` row" logic with `getActionsRow(wrap)`; keep everything else (the gates, the one-button-per-bubble guards, the icons, the `role=listitem` when the row is `role=list`).
|
||||
3. `frontend/assets/app.js` — `appendStoppedNote` (~L520) + `appendFailedNote` (~L560): join the actions row via `getActionsRow(wrap)` instead of the first `.msg-meta` row (ASSUMPTION A7 — the notes are answer-status markers; the actions row is their home). Keep the `failed-detail` textContent contract (no HTML from error strings).
|
||||
4. `frontend/assets/app.js` — `markLastRetryable` (~L620): the retry button is removed from EVERY bubble and re-appended — but now into `getActionsRow(lastBrainWrap)`. DELETE the save-button re-append hack (the `saveDocBtn.parentElement.appendChild(saveDocBtn)` block) — the row's `justify-content: flex-end` (task 03) right-aligns the whole cluster; DOM order inside the row is the insertion order (Tune, Retry, Save-as-doc per their gates). Update the function's comment (the "auto margins split the free space" rationale is gone).
|
||||
5. Call-order audit: at every site that renders a COMPLETED brain bubble (the `done` branch, the empty-answer fallback, the stop finalize, the phase-14 restore), confirm the order is answer figure → `appendSources` → `appendRelated` → action buttons/notes, so `.msg-actions` is the LAST row (the "bottom right" = the bubble's bottom, right edge). If a site appends actions before `appendRelated`, reorder the calls (the DOM order decides the visual order — no CSS tricks).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the frontend unit suite — update the pins that target the button row: the buttons/notes land in `.msg-actions` (not the chips row); `markLastRetryable` moves the Retry button into the actions row and no longer re-appends the save button.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a bubble with sources + all three buttons renders the actions row LAST, containing exactly the gated buttons (the notes when present)
|
||||
- [ ] a non-admin bubble's actions row holds only Retry (the gate unchanged)
|
||||
- [ ] the restore path renders the identical row (a saved chat re-opens with the buttons in `.msg-actions`)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 03 — CSS: the cluster rows (actions right-aligned, auto margins removed, images row)
|
||||
|
||||
**Phase:** `130_chat_answer_clusters` · **Source:** `TODO.md:6` — "...Those three buttons should be in a button cluster in the bottom right. ... `source-chips` should show up before images in their own cluster, then images in their own cluster to keep things organized."
|
||||
|
||||
## Objective
|
||||
The cluster layout is CSS: the actions row right-aligns its buttons as a cluster (no per-button auto margins), the images row wraps its figures like the chips row wraps chips, and the rows stack in DOM order with the existing `.msg-meta` rhythm.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/styles.css` — grep the current button-row rules (`.tune-btn`, `.retry-btn`, `.save-as-doc-btn`, and any `margin-inline-start: auto` on them — the `markLastRetryable`/phase-59 comments name the auto-margin mechanism; `.save-as-doc-btn`'s is explicit in the app.js comment at ~L653). Remove the per-button auto margins (the buttons become plain flex items of the actions row).
|
||||
2. `frontend/assets/styles.css` — add the cluster rules (place them with the other `.msg-meta` / chip / figure rules — the house grouping):
|
||||
```css
|
||||
/* Phase 130: the actions cluster — the bubble's LAST row, right-aligned.
|
||||
The row's justify does the "bottom right" (no per-button auto margins). */
|
||||
.msg-meta.msg-actions {
|
||||
justify-content: flex-end;
|
||||
}
|
||||
/* Phase 130: the source-IMAGES cluster — wraps like the chips row. */
|
||||
.msg-meta.msg-images {
|
||||
flex-wrap: wrap;
|
||||
gap: 0.4rem; /* match the .msg-meta chip gap (read the existing value) */
|
||||
}
|
||||
```
|
||||
(Match the EXISTING `.msg-meta` gap/align values — read the rule first; do not invent new spacing values that diverge from the chips row.)
|
||||
3. Audit: confirm the chips row (`.msg-meta` with `role=list`) and the related row (`.msg-meta.related-docs`) rules are untouched; the `.source-image` figure styles are untouched (they were already sized for a row context); mobile: the rows stack naturally (the `.msg-body` flex gap handles it) — verify in the mobile media query block that no rule pins the actions row left; the actions row stays right-aligned at every width.
|
||||
|
||||
## Testing & Quality
|
||||
- Visual gate: the E2E (task 04) — bounding-box assertions (the actions row's buttons' right edge ≈ the row's right edge; the row is the last `.msg-meta` in the bubble).
|
||||
- Coverage: **>90%** on new/modified code (CSS has no direct coverage — the E2E + unit pins carry it).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the actions cluster is right-aligned by `.msg-actions` alone (the per-button auto margins are gone — grep confirms)
|
||||
- [ ] the images row wraps its figures with the chips row's gap
|
||||
- [ ] `uv run pytest` green; no behavior change in completed phases
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 04 — E2E: the answer renders as organized clusters (DOM-asserted), live + restored
|
||||
|
||||
**Phase:** `130_chat_answer_clusters` · **Source:** `TODO.md:6` — "The chat output is getting messy. ... Those three buttons should be in a button cluster in the bottom right. We don't need to display the `source-chip` for images ... `source-chips` should show up before images in their own cluster, then images in their own cluster to keep things organized."
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) proving the owner-visible fix in the real DOM: a grounded answer citing a text doc AND an image doc renders the chips row (text chip only, NO chip for the image) before the images row (the figure), and the buttons sit in one right-aligned `.msg-actions` row that is the bubble's last row — on the live turn and after a page reload (the phase-14 restore path).
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_chat_answer_clusters.py` (new) — house pattern (module app fixture over the shared E2E Postgres, autouse cleanup, mock-only). KB fixture: follow `tests/e2e/test_chat_image_questions.py` / `test_image_documents.py`'s image-KB pattern — a local source with ONE text doc (the mock's grounded-answer trigger, e.g. the existing "use your tools" / read flow that cites a doc) and ONE image doc (the mock's `IMAGE_DESCRIPTION_MODE` canned description makes it retrievable; the done frame's `sources` must carry both refs — the image ref with `image_url`). Inspect those suites first and reuse their fixture builders — do not invent a new mock flow unless the existing done-frame shape cannot cite both docs in one answer (if a new mock trigger is needed, add it to `tests/e2e/mock_llm.py` in the house style: a user-message marker, a deterministic reply, a module-header docstring entry).
|
||||
Run as ADMIN (the whoami gate — the Tune + Save-as-doc buttons exist; the Retry button exists for everyone) with a configured docs repo (the `docsRepoConfigured` gate for Save-as-doc — follow the existing admin chat E2E setup).
|
||||
2. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_answer_clusters_render_in_order` — send the question; on `done`: the LAST brain bubble's `.msg-body` direct children include, in order: the `.answer-image` (if the mock offers one — else skip) → a `.msg-meta` chips row containing EXACTLY the text doc's `.source-chip` and NO `.source-chip` for the image doc (assert by the chip's `source/path` label) → a `.msg-meta.msg-images` row containing the image's `.source-image` figure (the `img` loaded, the caption = the canned description/summary) → (the related row, if present) → a `.msg-meta.msg-actions` row that is the bubble's LAST `.msg-meta`, containing the `.tune-btn`, `.retry-btn`, `.save-as-doc-btn` (all three — admin); the actions row's buttons are right-aligned (the rightmost button's bounding-box right edge ≈ the row's right edge, slack for the row padding) and the row's bounding box is BELOW the images row's (the "bottom right" geometry).
|
||||
- `test_image_without_chip_keeps_its_citation_via_figure` — in the same bubble: the image doc is cited (the figure's `href` = its `documentUrl`, left-click opens the same-page modal — the phase-26 affordance) and there is NO `.source-chip` whose label is the image's `source/path` (the owner's exact ask).
|
||||
- `test_restore_renders_the_same_clusters` — the turn auto-saves (phase-66); reload the page; the restored last brain bubble has the IDENTICAL structure (chips row text-only, images row, `.msg-actions` last with the gated buttons — the restore path renders through the same appenders).
|
||||
- `test_non_admin_sees_retry_only_actions` — a second, non-admin browser context on a FRESH chat (no admin cookie): send the same question; the `.msg-actions` row holds ONLY the `.retry-btn` (Tune/Save gated out — the phase-16/59 gates unchanged).
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_chat_answer_clusters.py -v --no-cov`), the mock-only gate, the fixture + mock-trigger notes, the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% bar is carried by tasks 01–02's updated frontend unit pins; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_chat_answer_clusters.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the DOM order assertion (chips → images → actions-last) is pinned for BOTH the live and the restored bubble
|
||||
- [ ] the suite fails against the pre-phase markup (buttons inside the chips row) — i.e. it actually pins the fix
|
||||
- [ ] no behavior change in completed phases (modal navigation, the related tier, the ARIA names)
|
||||
@@ -0,0 +1,51 @@
|
||||
# Phase 131 — `read` gains line ranges: grep finds the line, read(offset, limit) reads it
|
||||
|
||||
**Source:** `TODO.md` L7 — "The agent is struggling with huge documents. It can search for the text it wants and it finds the text, including line number, but then it gets stuck saying 'However, read takes a path, not a line number.' and can't figure out how to read that part of the document. Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck. I have a huge test pdf (the DND player's handbook) in test_sources called 'pdf_tests.tar' that you can use for testing. It's a massive pdf. The question the agent got stuck on was 'What are the components of the find traps spell?' which is toward the end of the players handbook so it will need to search and read specific lines. I've been setting BOR_READ_MAX_CHARS to 24000 to keep things relatively quick so the agent realizes quickly its answer isn't in the first few pages."
|
||||
(LOCKED A2, owner-confirmed 2026-09-27: the parameter names are `offset`/`limit` — 1-based line number + line count — mirroring the pi-harness `read` tool the model was trained on, the phase-70 "harness-aligned surface" anchor.)
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/agent.py` — `AGENT_TOOLS` (L258+: the `ls`/`read`/`grep` pi.dev-shaped schemas; the `read` schema at ~L304 takes ONLY `path`; the `grep` description promises `source/path:line: text` locators), `_execute_tool` (L1266+, the `read` branch at ~L1348: `arg` resolution, the `ALREADY_IN_CONTEXT` dedupe on `holder.read_docs`, the `BOR_READ_MAX_CHARS` truncation with `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` (~L506 — today it points at `grep` and then dead-ends), the `holder.read_truncations` tuples), `AgentHolder` (the `read_docs` / `read_truncations` bookkeeping); `app/rag/llm.py` — `ToolCallPiece` (the `arguments` dict the model sends); `tests/e2e/mock_llm.py` — the scripted tool-call flows (the "use your tools" / multi-read / grep patterns — the module header documents each trigger); `tests/unit/test_agent.py` — the read-branch unit suite (the `ALREADY_IN_CONTEXT` pins, the truncation pins); `test_sources/players_handbook.pdf` (the DND player's handbook from `pdf_tests.tar` — on disk, untracked); `settings.read_max_chars` (`BOR_READ_MAX_CHARS`).
|
||||
|
||||
## Objective
|
||||
The agent stops dead-ending on huge documents: `read` accepts an optional 1-based line `offset` + line-count `limit`, the truncation notice and tool descriptions teach the grep → read(offset/limit) workflow, and the dedupe rules allow the ONE case that matters — re-reading a range of a document whose first read was truncated — while still refusing pointless re-reads of text already in context. The owner's repro (the handbook, "find traps", `BOR_READ_MAX_CHARS=24000`) passes as an integration test.
|
||||
|
||||
## Dependencies
|
||||
- `130_chat_answer_clusters` (todo) — pipeline order only; code dependencies are the completed agent surface (phases 70/72/95/118).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Schema + teaching (task 01):** `read` gains two OPTIONAL integer params — `offset` ("The first line to read, 1-based. Omit it to read from the beginning of the document.") and `limit` ("The maximum number of lines to read. Omit it to read as many lines as fit under the character cap."). `required` stays `["path"]`. The `read` description keeps its existing teaching and GAINS the workflow: for a very large document, use `grep` to find where your text is (it reports line numbers), then `read` that part with `offset` + `limit`; re-reading a document already read in full adds nothing. `READ_TRUNCATION_NOTICE` (L506) is REWRITTEN to end the dead-end: "…The rest is NOT shown. Use grep (pattern) to find what you need — it reports line numbers — then read that part with `offset` (1-based first line) and `limit` (number of lines)." The `grep` description gets the light touch: "read the winner with `read` — for a large document, use the reported line number as `offset`."
|
||||
- **Execution (task 02):** in `_execute_tool`'s read branch:
|
||||
- Validation: `offset`/`limit`, when present, must be ints ≥ 1 (NOT bools — the house `isinstance(x, int) and not isinstance(x, bool)` guard); a bad value → a teaching refusal echoing the bad value ("`offset` and `limit` are whole line numbers ≥ 1 — you passed `…`") — counts in nothing, consumes a round (the phase-72 refusal style).
|
||||
- The dedupe matrix (`known` = the `read_docs` (source, path) set; `truncated` = the set of (source, path) from `holder.read_truncations`):
|
||||
| call | doc in `known` | previously truncated | result |
|
||||
|---|---|---|---|
|
||||
| plain `read(path)` | no | — | full read (truncate at the cap as today) |
|
||||
| plain `read(path)` | yes | yes | NEW teaching line (no re-emit, no count): "You already have the first {shown} characters of {source}/{path} in your context. To read the rest, find where your text is with grep (it reports line numbers), then call read with `offset` (1-based first line) and `limit` (lines)." — `{shown}` from the matching `read_truncations` tuple |
|
||||
| plain `read(path)` | yes | no | `ALREADY_IN_CONTEXT` (byte-identical to today) |
|
||||
| `read(path, offset, limit)` | yes | yes | ALLOWED (the stuck-scenario fix — the range the model names is exactly the part it does NOT have) |
|
||||
| `read(path, offset, limit)` | yes | no | `ALREADY_IN_CONTEXT` (the full text is already in context — the range is a subset) |
|
||||
| `read(path, offset, limit)` | no | — | ALLOWED (a range read of an unread doc) |
|
||||
- `offset > total_lines` (the doc's `len(content.splitlines())`) → teaching refusal: "{source}/{path} is {total_lines} lines long — `offset` {offset} is past the end. Use a smaller offset (grep reports the exact line numbers)." Counts in nothing.
|
||||
- Result shape: line 1 `Document {source}/{path}:` + line 2 `date: YYYY-MM-DD` stay BYTE-IDENTICAL for plain reads (the E2E mock's `_READ_RESULT_PREFIX` contract); a RANGE read inserts a third line `lines: {offset}-{end} of {total_lines}` (the actual end = min(offset + limit − 1, total_lines); the model sees the true extent), then the image-doc marker (as today), then the slice (`"\n".join(lines[offset-1 : offset-1+limit])`).
|
||||
- The cap still applies to the SLICE: `len(slice) > settings.read_max_chars` → cut + `TRUNCATION_MARKER` + the (rewritten) notice; the `read_truncations` tuple for a range read is `(arg, cap, len(slice))` — the slice's requested size (documented: for range reads the tuple's `chars_total` is the slice length, not the document's).
|
||||
- A range read appends the doc to `read_docs` ONCE (the `known` set prevents a re-append) — so a later plain read of a range-read doc is `ALREADY_IN_CONTEXT` (the model has committed to this doc; the range result's `lines:` line told it what it holds).
|
||||
- **Handbook repro (task 03):** an integration test that imports `test_sources/players_handbook.pdf` as a local source (the mock LLM's deterministic embed/summary fakes — the in-process `tests/fakes.py` pattern; the handbook is ~1.5 MB of text ≈ 1200 chunks — in-process embedding is fast, keep the test under ~60 s) and drives the agent turn "What are the components of the find traps spell?" with `BOR_READ_MAX_CHARS=24000`: a scripted fake (the `test_agent.py` pattern) — round 1 → a `grep` tool call (pattern "find traps"), round 2 (the grep result carries the real line number) → a `read(path, offset=<that line>, limit=40)` call, round 3 → the answer echoing the read result's content (the phase-94 "read document reached the context" echo convention). Assert the echo contains the handbook's actual find-traps text (grep it from the extracted PDF text in the test — do not hardcode the handbook wording).
|
||||
- **Mock flow + E2E (task 04):** `tests/e2e/mock_llm.py` gains ONE scripted flow (house style — a user-message marker, deterministic reply, module-header docstring entry): the marker `read the lines` → request 1: a `grep` tool call (a canned pattern); request 2 (a `tool`-role grep result with a `source/path:line:` line present): a `read` tool call with `offset` = the parsed line number and `limit` = 30; request 3 (the read result present): the answer "Found it at line N." + the first 80 chars of the read result (the existing read-echo shape). `tests/e2e/test_read_line_ranges.py` (dedicated, in isolation, mock-only): a KB fixture with ONE large doc (> 24000 chars, the marker line `TRAP_COMPONENTS_LINE` near line ~1500), `BOR_READ_MAX_CHARS=24000`; ask a question containing the marker → assert the SSE `tool` frames (grep, then read with `offset`/`limit` args) and the final answer echoing the marker line; a second test: a plain read of the large doc truncates with the REWRITTEN notice (the notice names `offset`/`limit` — assert the `tool_result`/result surface carries it).
|
||||
|
||||
## Tasks
|
||||
1. `01_tool_surface.md` — the schema, the descriptions, the rewritten truncation notice.
|
||||
2. `02_range_read_execution.md` — the read branch: validation, the dedupe matrix, the slicing, the result shape.
|
||||
3. `03_handbook_repro.md` — the integration repro (the handbook + the owner's exact question) + the unit matrix.
|
||||
4. `04_mock_flow_e2e.md` — the mock-LLM `read the lines` flow + the dedicated E2E.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the full dedupe matrix (6 rows), the validation refusals, the slice math (boundaries, offset past EOF, 1-line docs), the result shapes (plain reads byte-identical — the existing pins must pass unchanged), the cap on slices.
|
||||
- Integration: the handbook repro (the owner's scenario, the owner's `BOR_READ_MAX_CHARS=24000`).
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `read(path, offset, limit)` returns exactly the named lines with the `lines: a-b of N` header; plain reads are byte-identical to pre-phase
|
||||
- [ ] a truncated first read followed by a range read SUCCEEDS (the stuck scenario is gone); every other re-read is refused with the teaching
|
||||
- [ ] the handbook integration test answers the find-traps question from the real PDF text (no hardcoded wording)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the grep/ls surfaces, the plain-read contract, the `ALREADY_IN_CONTEXT` full-read refusal)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 01 — The tool surface: `offset`/`limit` on `read`, the descriptions, the rewritten notice
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. It can search for the text it wants and it finds the text, including line number, but then it gets stuck saying 'However, read takes a path, not a line number.' ... Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck."
|
||||
|
||||
## Objective
|
||||
The model's contract for `read` names the line-range capability (LOCKED A2: `offset`/`limit`, 1-based, pi-harness shape) and the truncation dead-end is rewritten into the working workflow: grep for the line numbers, then read that range.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `AGENT_TOOLS`, the `read` entry (~L304): add to `properties`:
|
||||
```python
|
||||
"offset": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The first line to read, 1-based. Omit it to read from the "
|
||||
"beginning of the document. Use it to read a specific part "
|
||||
"of a large document — grep reports the line numbers of "
|
||||
"what you found."
|
||||
),
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The maximum number of lines to read, starting at offset. "
|
||||
"Omit it to read as many lines as fit under the character "
|
||||
"cap. Keep it small — you are reading a part, not the "
|
||||
"document."
|
||||
),
|
||||
},
|
||||
```
|
||||
`required` stays `["path"]`.
|
||||
2. `app/rag/agent.py` — the `read` `description`: keep the existing sentences (the `<documents>`-summaries teaching, the no-re-read rule, the truncation/`grep` paragraph — but REPLACE the truncation paragraph with the workflow): "Very large documents are truncated: you receive the first part plus a TRUNCATED notice naming how many more characters exist. To read the rest: use `grep` (pattern) to find where your text is — it reports line numbers — then call `read` again on the same document with `offset` (1-based first line) and `limit` (number of lines) to read exactly that part. Do not re-read a document you have already read in full — its full text is already in your prompt."
|
||||
3. `app/rag/agent.py` — `READ_TRUNCATION_NOTICE` (~L506): rewrite the tail (keep the `TRUNCATED — this document is {total} characters; only the first {shown} are in your context. The rest is NOT shown.` opening): "Use grep (pattern) to find what you need — it reports line numbers — then read that part with `offset` (1-based first line) and `limit` (number of lines)."
|
||||
4. `app/rag/agent.py` — the `grep` `description`: the "read the winner with `read`" sentence becomes "read the winner with `read` — for a large document, use the reported line number as `offset`".
|
||||
5. `tests/unit/test_agent.py` — update/extend the `AGENT_TOOLS` schema pins: the `read` schema has `offset`/`limit` (integer, optional — `required` is still `["path"]`), the descriptions contain the `offset`/`limit` teaching (the pinned substrings), and `READ_TRUNCATION_NOTICE` names both parameters.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the schema/notice pins above green; the `ls`/`grep` schemas' OTHER pins unchanged (byte-identical where not rewritten).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a model reading `AGENT_TOOLS` knows: grep → line number → `read(offset, limit)` (schema + descriptions + notice all agree)
|
||||
- [ ] the old dead-end sentence ("use grep to locate the rest" with no way to act on the line number) is gone from the notice
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,49 @@
|
||||
# Task 02 — Range-read execution: validation, the dedupe matrix, the slice, the result shape
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. ... then it gets stuck saying 'However, read takes a path, not a line number.' and can't figure out how to read that part of the document."
|
||||
|
||||
## Objective
|
||||
`_execute_tool`'s read branch executes `offset`/`limit`: it validates the arguments with teaching refusals, applies the six-row dedupe matrix (the truncated-then-range case is the one that must succeed), slices the document's lines, and shapes the result so plain reads stay byte-identical while range reads carry their `lines: a-b of N` extent.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `_execute_tool`, the `read` branch (~L1348): after `arg` is resolved and before the dedupe check, parse the range args:
|
||||
```python
|
||||
offset = call.arguments.get("offset")
|
||||
limit = call.arguments.get("limit")
|
||||
```
|
||||
with a helper `_read_range_args(offset, limit)` returning `(ok, error_line, offset, limit)`:
|
||||
- absent/None → ok, `(1, None)` semantics = plain read (no range);
|
||||
- present but not `isinstance(x, int) and not isinstance(x, bool)`, or `< 1` → the teaching refusal: "`offset` and `limit` are whole line numbers — pass integers ≥ 1 (you passed offset={offset!r}, limit={limit!r})." (echo the bad values; counts in nothing, consumes a round — the phase-72 refusal style, a new pinned constant `BAD_RANGE_ARGS`);
|
||||
- ok → the (offset, limit) pair (either may be None — `offset` without `limit` reads to the end, `limit` without `offset` is treated as `offset=1` — document both in the constant's docstring).
|
||||
2. `app/rag/agent.py` — the dedupe matrix (the phase-131 design table; `truncated_pairs = {(source, path) of holder.read_truncations}` — build it from the holder's tuples, matching on the RESOLVED pair, not the raw arg string):
|
||||
- plain read + in `known` + truncated → the new pinned teaching line `REREAD_TRUNCATED` ("You already have the first {shown} characters of {source}/{path} in your context. To read the rest, find where your text is with grep (it reports line numbers), then call read with `offset` (1-based first line) and `limit` (lines).") — `{shown}` from the matching tuple; no re-emit, `tool_calls` NOT incremented (a refusal);
|
||||
- plain read + in `known` + not truncated → `ALREADY_IN_CONTEXT` (the existing byte-identical path — do not touch that constant);
|
||||
- range read + in `known` + truncated → PROCEED (the fix);
|
||||
- range read + in `known` + not truncated → `ALREADY_IN_CONTEXT`;
|
||||
- range read + not in `known` → PROCEED.
|
||||
(The current code's dedupe check runs on the RAW arg before resolution for speed — keep that fast path for the plain-read `ALREADY_IN_CONTEXT` case; the truncated-set check needs the resolved pair, so it runs after `_resolve_path`, exactly where the `doc` is known. Restructure minimally: the raw-arg fast path handles "plain read + known"; everything else resolves first, then consults the matrix.)
|
||||
3. `app/rag/agent.py` — the slice + result (after `doc` is resolved and the matrix passed):
|
||||
- `lines = doc.content.splitlines()`; `total_lines = len(lines)`;
|
||||
- `offset > total_lines` → the teaching refusal `OFFSET_PAST_END` ("{source}/{path} is {total_lines} lines long — `offset` {offset} is past the end. Use a smaller offset (grep reports the exact line numbers).") — no count;
|
||||
- `end = min(offset + (limit or 1) - 1, total_lines)`; `slice_lines = lines[offset - 1 : end]`; `slice_text = "\n".join(slice_lines)`;
|
||||
- `holder.read_docs.append(doc)` (the `known` check already guarantees no duplicate); `holder.tool_calls += 1` (a range read is a successful call — the plain-read convention);
|
||||
- result: plain read → EXACTLY today's bytes (header, date, marker, content/truncated-content — the existing code path untouched); range read →
|
||||
```
|
||||
Document {source}/{path}:
|
||||
date: YYYY-MM-DD
|
||||
lines: {offset}-{end} of {total_lines}
|
||||
{marker}{slice_text}
|
||||
```
|
||||
(the `lines:` line replaces the position the marker/`TRUNCATION_MARKER` occupy in the plain read — for a range read the marker line comes after `lines:` exactly as after the date line in the plain read);
|
||||
- the cap on the slice: `len(slice_text) > settings.read_max_chars` → `slice_text[:cap]` + `\n{TRUNCATION_MARKER}\n{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))}` and `holder.read_truncations.append((arg, cap, len(slice_text)))` — for range reads the tuple's `chars_total` is the SLICE's requested size (document in the `AgentHolder.read_truncations` field comment: "for range reads the slice length, not the document's").
|
||||
4. `tests/unit/test_agent.py` — the full matrix (six rows, each pinned): the six dedupe outcomes; `BAD_RANGE_ARGS` (a float, a string, a bool, a 0, a negative); `OFFSET_PAST_END` (offset = total+1 on a 10-line doc); the slice math (offset=1/limit=None → the whole doc; offset=total → the last line; limit overshoot → `end` clamps to `total_lines`; a 1-line doc; offset past EOF); the result shapes (the plain-read pins pass UNCHANGED — byte-identical; a range read's `lines: a-b of N` line + slice; the cap on a huge slice + the tuple's slice-length `chars_total`).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the matrix + the pins above — this task's suite.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the six-row matrix is pinned one test per row (the truncated-then-range row PASSES — the stuck scenario's unit proof)
|
||||
- [ ] plain reads are byte-identical to pre-phase (the existing pins green, unmodified)
|
||||
- [ ] a range read's result names its true extent (`lines: a-b of N`) and respects the cap
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 03 — The handbook repro: "find traps" on the real D&D player's handbook
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "I have a huge test pdf (the DND player's handbook) in test_sources called 'pdf_tests.tar' that you can use for testing. It's a massive pdf. The question the agent got stuck on was 'What are the components of the find traps spell?' which is toward the end of the players handbook so it will need to search and read specific lines. I've been setting BOR_READ_MAX_CHARS to 24000 to keep things relatively quick so the agent realizes quickly its answer isn't in the first few pages."
|
||||
|
||||
## Objective
|
||||
The owner's exact incident is an integration test: the real `players_handbook.pdf` is indexed, the agent is asked the find-traps question with the owner's `BOR_READ_MAX_CHARS=24000`, and the scripted agent loop performs the grep → read(offset/limit) dance against the REAL document and lands an answer grounded in the handbook's actual find-traps text.
|
||||
|
||||
## Work
|
||||
1. `tests/integration/test_read_line_ranges_handbook.py` (new) — the house integration pattern (inspect `tests/integration/` for the agent-loop test that runs a scripted fake LLM against a real-ish pipeline — `tests/unit/test_agent.py`'s fake-LLM/scripted-rounds pattern is the model; lift it into an integration test that ALSO runs the real PDF extraction + real chunking/embedding fakes from `tests/fakes.py`):
|
||||
- Fixture: a temp source dir containing `test_sources/players_handbook.pdf` (copy — never mutate the untracked fixture), registered/imported with the REAL importer (`import_sources`, the mock embed/summary fakes — the in-process fakes make the ~1200-chunk embed fast; keep the whole test under ~60 s; if the import is slower, the test may pre-import once at module scope and share it across its tests).
|
||||
- The scripted fake (deterministic, the `test_agent.py` round pattern):
|
||||
- round 1 (no tool results yet) → a `grep` tool call, `pattern="find traps"` (lowercase — grep is case-insensitive);
|
||||
- round 2 (the grep result carries `…:LINE: …find traps…` — parse the REAL line number from the grep result the loop produced, do not hardcode it) → a `read` tool call: `path` = the handbook's `source/path`, `offset` = the parsed line, `limit` = 40;
|
||||
- round 3 (the read result present) → the answer: the read result's content, verbatim (the phase-94 "the read document reached the context" echo convention — the loop's reply IS the slice).
|
||||
- Settings: `BOR_READ_MAX_CHARS=24000` (the owner's env — the handbook is far longer, so round 1's plain-read alternative would truncate; the script goes straight to grep, but the setting must be in force for the notice/teaching path to be realistic).
|
||||
2. Assertions (the observable contract):
|
||||
- the final answer contains the handbook's REAL find-traps text — derive the expected substring at test time: extract the PDF's text (the phase-126 `extract_pdf_text`), grep it for the find-traps passage in the test itself (the same way the model does), and assert the ANSWER contains that passage's first 60 chars (NO hardcoded handbook wording — the test stays true if the fixture is re-extracted);
|
||||
- the agent's `read` call carried `offset`/`limit` (assert on the fake's captured round-2 tool call: `offset == the grep-reported line`, `limit == 40`);
|
||||
- the loop made exactly 3 rounds (grep → range read → answer) with `tool_calls == 2` (the owner's stuck loop made 2 and dead-ended — the fixed loop closes in 3).
|
||||
3. `tests/integration/test_read_line_ranges_handbook.py` — a SECOND test, the plain-read-first variant: the fake round 1 does a PLAIN `read(path)` (the handbook truncates at 24000 with the rewritten notice), round 2 does `read(path, offset=<grep line>, limit=40)` AFTER a grep — asserting the truncated-then-range matrix row works on the REAL document (the `REREAD_TRUNCATED` teaching is never needed because the model went straight to the range — the matrix row itself is unit-pinned in task 02; here the end-to-end path is the point).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this file IS the task's suite (plus task 02's unit matrix).
|
||||
- Coverage: **>90%** on new/modified code (the test file is covered by the suite run; `app/` changes are task 02's).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_read_line_ranges_handbook.py -v` green in ~60 s (the owner's scenario passes on the real PDF)
|
||||
- [ ] the expected text is derived from the fixture at test time (grep the extracted text — no hardcoded handbook wording)
|
||||
- [ ] the read call's `offset` is the grep-reported line (parsed, not literal)
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — The mock flow `read the lines` + the dedicated E2E
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. ... Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck."
|
||||
|
||||
## Objective
|
||||
The E2E mock deterministically performs the grep → read(offset/limit) dance, and a dedicated, in-isolation Playwright suite pins the full SSE surface: the `tool` frames carry the range args, the answer is grounded in the large doc's tail, and the truncation notice now teaches the workflow instead of dead-ending.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new scripted flow in the house style (a user-message marker + a module-header docstring entry, exactly like the existing "use your tools" / multi-read / grep flows — read the module header first and match its format):
|
||||
- Marker: the user message containing `read the lines` (a dedicated trigger — no overlap with the existing grep-flow trigger; if the existing grep flow's trigger could also match, make the new marker more specific, e.g. `read the lines of` — the header documents the exact string).
|
||||
- Flow (classified by the count of `tool`-role results in the messages, the house pattern):
|
||||
- 0 results → a `grep` tool call, `pattern="LINE_MARKER"` (a canned constant the fixture doc contains);
|
||||
- a `tool`-role grep result present (a line matching `^[\w/]+:(\d+):` — parse the FIRST match's line number) → a `read` tool call: `path` = the grep result's `source/path`, `offset` = the parsed line number, `limit` = 30 (structured `tool_calls`, no reasoning — the terminal-shape convention);
|
||||
- the read result present (a `tool`-role result whose content starts with `Document `) → the forced answer, byte-stable: `Found it at line N.` (N = the offset the model passed — parse it from the `read` call's arguments in the messages) + the first 80 chars of the read result's content (the phase-94 read-echo convention — proves the RANGE reached the model's context).
|
||||
2. `tests/e2e/test_read_line_ranges.py` (new) — dedicated, in isolation, mock-only, house pattern (module app fixture, temp local source, autouse cleanup):
|
||||
- KB fixture: ONE large markdown doc `linedoc/big.md` — ~100 KB, generated deterministically in the test (numbered filler lines `Line <i>: …`, so line numbers are exact), with the marker line `LINE_MARKER the trap components are documented here` at a known line ~1500 (far beyond the 24000-char cap — the doc's first 24000 chars hold NO marker);
|
||||
- app env: `BOR_READ_MAX_CHARS=24000` (the owner's setting — the plain read MUST truncate).
|
||||
- Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_range_read_flow_over_sse` — send a question containing `read the lines` → the SSE `tool` frames, in order: the `grep` call (args `pattern="LINE_MARKER"`), then the `read` call (args: `path` = the doc's `source/path`, `offset` an int == the marker's true line, `limit` == 30 — assert on the frame's args, not a substring); the `done` frame's answer contains `Found it at line <N>` (N == the marker's line) and the marker text's first 80 chars (the read-echo — the range reached the context); NO truncation-notice dead-end (the answer is the grounded echo, not a "I can't read that part" shape — assert the marker text is present).
|
||||
- `test_plain_read_truncation_notice_teaches_the_workflow` — a SECOND trigger (reuse the mock's existing plain-read flow on this large doc — the `read` of `linedoc/big.md` truncates at 24000): the surfaced truncation (the `tool_result` frame / the result the model sees — assert on the SSE `tool_result` surface per A15) contains the REWRITTEN notice text naming `offset` and `limit` (the pinned substrings `"offset"` + `"limit"` in the notice — the old dead-end wording is gone).
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov`), the mock-only gate, the fixture's determinism (the generated line numbers), the `BOR_READ_MAX_CHARS=24000` rationale (the owner's env), the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% `app/` bar is carried by tasks 01–02; the handbook integration repro by task 03; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the `read` tool frame carries REAL `offset`/`limit` ints (parsed from the grep result — the mock does the owner's exact dance)
|
||||
- [ ] the truncation notice surfaces the workflow teaching (the dead-end is gone)
|
||||
- [ ] no behavior change in completed phases (the existing mock flows' triggers still fire unchanged)
|
||||
@@ -0,0 +1,35 @@
|
||||
# Phase 132 — Chat image upload: visible "uploading" feedback
|
||||
|
||||
**Source:** `TODO.md` L8 — "Uploading an image in the chat provides no feedback to the user. The user clicks the paperclick, selects and image, presses 'send' and then nothing happens. No loading bar, no 'uploading image', no indication. This results in the user spamming 'send' over and over because they think it's frozen. We need to visually show the user's image is uploading."
|
||||
(A5, owner-confirmed 2026-09-27: the affordance = a spinner + "Uploading image…" in the preview strip AND the Send button disabled with the label "Uploading…" — the double-spam is blocked visually, not just by the internal guard.)
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `frontend/assets/app.js` — `handleSend` (~L2700: the `attachedImage` branch — the `attachUpload` double-fire guard, `await uploadAttachedImage(attachedImage.file)` with NO visual state between the await's start and its resolution, then `image = { path, src, alt }` + `runTurn`), `uploadAttachedImage` (~L2641: the `POST /api/chat-images` multipart — every failure path is an out-of-turn banner + null), `showAttachPreview` / `clearAttachedImage` (~L2680: the strip's reveal/clear — static markup, only `textContent`/`src` move), `setUiState` (the turn-state owner — NOT used for the upload: the upload is a PRE-turn operation, `uiState` stays `idle`, the turn loader must not spin during the upload); `frontend/index.html` L280–322 — the static attach markup (`#attach-preview` with `img` + `.attach-preview-name` + `#attach-remove`; `#send-btn` with `#send-label`); `frontend/assets/styles.css` — the `.attach-preview*` / `.send-btn` rules (the existing `:disabled` styles); `tests/unit/test_frontend_sync_upload.py` or a sibling — the pinned frontend JS unit pattern.
|
||||
|
||||
## Objective
|
||||
Between the user pressing Send and the upload resolving, the UI proves the image is moving: the preview strip shows a spinner + "Uploading image…" (the remove button disabled), and the Send button is disabled with the label "Uploading…". When the upload resolves — success proceeds to the turn exactly as today; failure clears the state and keeps the existing banner + the typed text + the attachment for the retry. No frozen-looking gap, no visible double-send.
|
||||
|
||||
## Dependencies
|
||||
- `131_read_line_ranges` (todo) — pipeline order only; code dependencies are the completed phase-123 attach/upload contract.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **State scope (task 01):** the uploading state is the `attachedImage` branch of `handleSend` — from just after the `attachUpload = true` guard until the `attachUpload = false` line (success AND failure paths — the helper never throws). `setUiState` is NOT touched (the upload is pre-turn; `uiState` stays `idle`; the turn's own thinking/streaming feedback starts only when `runTurn` does). The `attachUpload` guard stays exactly as is (the visual state reinforces it — the disabled Send is the user-facing half).
|
||||
- **Preview strip (task 01):** the static markup gains ONE status element inside `#attach-preview`: `<span class="attach-preview-status" hidden><svg spinner…></svg><span class="attach-preview-status-text">Uploading image…</span></span>`. The JS sets `attachPreview.classList.add("is-uploading")` + un-hides the status + `attachRemove.disabled = true` on entry; on exit it removes the class + hides the status + `attachRemove.disabled = false`. The thumbnail + filename stay visible (the user watches THEIR image uploading).
|
||||
- **Send button (task 01):** `sendBtn.disabled = true` + `sendLabel.textContent = "Uploading…"` on entry; on exit (both paths) `sendBtn.disabled = false` + `sendLabel.textContent = "Send"` — restored BEFORE the success path calls `runTurn` (the turn's `setUiState("thinking")` then takes over the button as usual) and BEFORE the failure path returns (the banner + editable input + the still-attached image).
|
||||
- **Reduced motion (task 02):** the spinner animation is still under `prefers-reduced-motion` (the house `@media (prefers-reduced-motion: reduce)` block pattern).
|
||||
|
||||
## Tasks
|
||||
1. `01_upload_state.md` — the static markup + the uploading state in `handleSend` (enter/exit on both paths).
|
||||
2. `02_css_upload_state.md` — the strip's uploading styles + the disabled Send + reduced motion.
|
||||
3. `03_e2e_upload_feedback.md` — dedicated Playwright E2E, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pinned frontend JS suite — the state transitions (the strip's class/status/remove-disabled, the Send's disabled/label) on entry, on success exit, on failure exit (the banner path).
|
||||
- E2E: one dedicated Playwright file, run in isolation, with a route-delayed upload (Playwright `page.route` delaying `POST /api/chat-images`) so the in-flight window is observable.
|
||||
- Coverage: **>90%** on new/modified code (frontend: unit pins + E2E; `app/` untouched this phase).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] while the upload is in flight: the strip shows the spinner + "Uploading image…", the remove button is disabled, the Send button is disabled with "Uploading…" (E2E-asserted mid-flight)
|
||||
- [ ] a second Send press during the flight does nothing (no second upload request — E2E request count)
|
||||
- [ ] success: the indicator clears and the turn proceeds exactly as pre-phase (the user bubble carries the image); failure: the indicator clears, the banner shows, the text + attachment remain
|
||||
- [ ] `uv run pytest tests/e2e/test_image_upload_feedback.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the phase-123 upload contract, the A5 path-never-bytes rule, the out-of-turn banner)
|
||||
@@ -0,0 +1,49 @@
|
||||
# Task 01 — The uploading state: markup + `handleSend` enter/exit on both paths
|
||||
|
||||
**Phase:** `132_image_upload_feedback` · **Source:** `TODO.md:8` — "Uploading an image in the chat provides no feedback to the user. ... We need to visually show the user's image is uploading."
|
||||
|
||||
## Objective
|
||||
The `attachedImage` branch of `handleSend` enters a visible uploading state the moment the upload starts and clears it the moment the upload resolves — success or failure — so the user always sees whether the app is moving.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` (L280–284, the `#attach-preview` block) — add the status element INSIDE the strip (after the name span, before the remove button), static + hidden by default (the house "ships hidden" rule):
|
||||
```html
|
||||
<span class="attach-preview-status" hidden>
|
||||
<svg class="attach-preview-status-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M21 12a9 9 0 1 1-6.2-8.56"/></svg>
|
||||
<span class="attach-preview-status-text">Uploading image…</span>
|
||||
</span>
|
||||
```
|
||||
(The spinner arc is the house loading glyph — check `#turn-loader`'s icon for the exact shape and reuse it; the spinner's spin animation is the CSS task. `aria-hidden` on the svg — the text carries the meaning; the strip's `role`/label are untouched.)
|
||||
2. `frontend/assets/app.js` — get the new elements at module scope with the existing attach-family consts (L313–316, the null-guard pattern): `attachStatus = document.querySelector(".attach-preview-status")`, `attachStatusText` (not needed — the text is static), alongside `attachPreview`/`attachRemove`.
|
||||
3. `frontend/assets/app.js` — two tiny helpers next to `showAttachPreview`/`clearAttachedImage`:
|
||||
```js
|
||||
function enterUploadState() {
|
||||
if (attachPreview) attachPreview.classList.add("is-uploading");
|
||||
if (attachStatus) attachStatus.hidden = false;
|
||||
if (attachRemove) attachRemove.disabled = true;
|
||||
sendBtn.disabled = true;
|
||||
sendLabel.textContent = "Uploading…";
|
||||
}
|
||||
function exitUploadState() {
|
||||
if (attachPreview) attachPreview.classList.remove("is-uploading");
|
||||
if (attachStatus) attachStatus.hidden = true;
|
||||
if (attachRemove) attachRemove.disabled = false;
|
||||
sendBtn.disabled = false;
|
||||
sendLabel.textContent = "Send";
|
||||
}
|
||||
```
|
||||
(The label literals "Send"/"Uploading…" — check the idle label's exact current text in the markup/`setUiState` and match it byte-for-byte; if `setUiState` owns the idle label restore, use its restore helper instead of a literal — inspect first.)
|
||||
4. `frontend/assets/app.js` — `handleSend`, the `attachedImage` branch (~L2700):
|
||||
- after the `attachUpload = true;` line, BEFORE `await uploadAttachedImage(...)`: `enterUploadState();`
|
||||
- on the `attachUpload = false;` line (the helper never throws — this is reached on BOTH success and failure): `exitUploadState();` — the success path then continues to `image = { path, src, alt }` + the rest of the send (the turn's `setUiState` takes the button over); the failure path (`path === null`) returns with the banner shown, the input text intact, and the attachment still attached (the strip is back in its normal preview state — the user can retry or remove).
|
||||
- Do NOT touch the `attachUpload` guard, the `uploadAttachedImage` helper, or `clearAttachedImage` (the New-chat reset already hides the strip — the uploading class is transient and never persists: if a New chat happened mid-upload, the strip is hidden; the class on a hidden element is harmless, and `exitUploadState` still clears it when the upload resolves).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pinned frontend JS suite (extend `tests/unit/test_frontend_sync_upload.py` or the chat-frontend pin file — inspect what exists): the state transitions — entry sets the class/status/unhides/disables remove/disables Send with "Uploading…"; exit (success) clears everything and restores "Send"; exit (failure) clears everything the same way (the banner is the failure's signal, not the strip's); a second `handleSend` during the flight is a no-op (the existing `attachUpload` guard pin).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the uploading state is entered exactly once per upload (after the guard, before the await) and exited on BOTH resolution paths
|
||||
- [ ] `uiState`/`setUiState`/the turn loader are untouched (the upload is pre-turn — the loader spins only when the turn starts)
|
||||
- [ ] the idle label after exit matches the pre-phase idle label byte-for-byte
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,54 @@
|
||||
# Task 02 — CSS: the strip's uploading look, the disabled Send, reduced motion
|
||||
|
||||
**Phase:** `132_image_upload_feedback` · **Source:** `TODO.md:8` — "Uploading an image in the chat provides no feedback to the user. ... We need to visually show the user's image is uploading."
|
||||
|
||||
## Objective
|
||||
The uploading state reads as "in motion": the strip's thumbnail dims with a spinning arc + the "Uploading image…" text beside it, and the Send button shows the disabled affordance the app already has for a disabled Send — no new color, no new weight, the existing `--ink-soft`/opacity language.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/styles.css` — with the other `.attach-preview*` rules (grep the existing block first — read the strip's current layout: the thumbnail, the name, the remove button):
|
||||
```css
|
||||
/* Phase 132: the uploading state — the strip proves the image is
|
||||
moving (the TODO: no feedback → the user spams Send). The
|
||||
thumbnail dims, the spinner arc spins, the status text stands in
|
||||
the name's place. */
|
||||
.attach-preview.is-uploading .attach-preview-img {
|
||||
opacity: 0.5;
|
||||
}
|
||||
.attach-preview.is-uploading .attach-preview-name {
|
||||
visibility: hidden; /* keep the layout; the status takes its spot */
|
||||
}
|
||||
.attach-preview-status {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 0.35rem;
|
||||
color: var(--ink-soft);
|
||||
font-size: 0.85rem; /* the strip's small text (match the name's size) */
|
||||
min-width: 0;
|
||||
}
|
||||
.attach-preview-status-icon {
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
flex: 0 0 auto;
|
||||
animation: spin 1s linear infinite; /* the .sync-icon spinner (the house spin keyframes) */
|
||||
}
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.attach-preview-status-icon { animation: none; }
|
||||
}
|
||||
.attach-preview.is-uploading .attach-preview-remove {
|
||||
opacity: 0.5; /* the disabled attribute's visual half (the house :disabled language) */
|
||||
}
|
||||
```
|
||||
(Match the EXISTING values — the strip's font sizes, the `spin` keyframes' name (verify they exist — the `.sync-icon.is-spinning` rule uses `spin`), the disabled-button opacity. If the strip's name span has `overflow: hidden; text-overflow: ellipsis` (a long filename), the status span inherits the row's clamp — verify the "Uploading image…" text never clips at 320 px; if it does, let the status text wrap or ellipsize per the strip's existing pattern.)
|
||||
2. `frontend/assets/styles.css` — verify the `#send-btn:disabled` rule exists and reads as disabled (the Send is disabled via the `disabled` attribute — the existing rule applies; if the disabled Send currently looks identical to the enabled one, add the house `opacity`/`cursor: wait` there — inspect the current rule first, do not invent a new visual).
|
||||
3. Theme audit: every value used is a theme variable or a neutral (no raw hex/rgb — the B4 byte-identical built-in theme contract; the status text's `--ink-soft` on the strip's background meets 4.5:1 — the strip's bg is `--bg-soft` or similar, verify the pair's contrast the way the theme phases did).
|
||||
|
||||
## Testing & Quality
|
||||
- Visual gate: the E2E (task 03) — the status is visible mid-flight, the thumbnail is dimmed (an `opacity` computed-style assertion is optional — the visibility of the status text is the pin).
|
||||
- Coverage: **>90%** on new/modified code (CSS has no direct coverage — the E2E + unit pins carry it).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the uploading strip shows the spinning arc + "Uploading image…" with the dimmed thumbnail (E2E-asserted mid-flight)
|
||||
- [ ] `prefers-reduced-motion` stills the arc (the house block)
|
||||
- [ ] the disabled Send reads as disabled (the existing rule, verified)
|
||||
- [ ] `uv run pytest` green; no behavior change in completed phases
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 03 — E2E: the upload window shows "uploading", blocks the double-send, then the turn proceeds
|
||||
|
||||
**Phase:** `132_image_upload_feedback` · **Source:** `TODO.md:8` — "Uploading an image in the chat provides no feedback to the user. The user clicks the paperclick, selects and image, presses 'send' and then nothing happens. ... This results in the user spamming 'send' over and over because they think it's frozen. We need to visually show the user's image is uploading."
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) proving the owner-visible fix: while `POST /api/chat-images` is in flight (route-delayed so the window is observable), the strip shows the uploading state, the Send button is disabled with "Uploading…", a second press produces NO second upload request — and once the upload resolves, the state clears and the turn proceeds exactly as pre-phase.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_image_upload_feedback.py` (new) — house pattern (module app fixture over the shared E2E Postgres, autouse cleanup, mock-only). The app boots with `BOR_IMAGES=1` (the `/api/config` `images: true` gate reveals `#attach-btn` — follow `tests/e2e/test_chat_image_questions.py`'s app env). KB fixture: a minimal local source with ONE text doc (the mock's grounded answer for a plain question — the turn after the upload needs to complete normally).
|
||||
2. The upload window (Playwright's `page.route`): intercept `POST /api/chat-images` — `route.waitForRequest`-style: the handler DELAYS (e.g. `await asyncio.sleep(2.0)` in the route handler, or `page.route` + a fixture that holds the response) then `route.continue_()` — the server's real handler runs, the response is just late. Count the intercepted upload requests (a module counter) for the double-send assertion. A small PNG fixture (the house test image — reuse the existing one the image E2E suites use; do not commit a new binary if one exists).
|
||||
3. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_upload_window_shows_uploading_and_blocks_resend` — admin or anonymous (chat is public — anonymous is the simpler context) page on the chat view; click `#attach-btn` → `#attach-file` set_input_files (the small PNG); the strip is revealed (the phase-123 preview — `#attach-preview` not hidden, the thumbnail's `src` set); type a question; click Send. MID-FLIGHT (poll for the state within the delayed window): `#attach-preview` has the `is-uploading` class; `.attach-preview-status` is visible (not hidden) with the text "Uploading image…"; `#attach-remove` is disabled; `#send-btn` is disabled with `#send-label` text "Uploading…". THEN press `#send-btn` twice more (the user's exact spam — the button is disabled so the presses are no-ops; also try an Enter-to-submit while disabled — the form submit's `sendBtn.disabled` guard) → after the window, the upload-request counter is EXACTLY 1 (no double upload, no second turn).
|
||||
- `test_upload_success_clears_and_turn_proceeds` — the SAME flow, past the window: the `is-uploading` class is gone, `.attach-preview-status` hidden, `#attach-remove` enabled, `#send-label` back to the idle "Send"; the strip HIDEs with the send (the phase-123 contract — the preview clears when the question goes out); the user bubble renders with the attached image (the phase-123 `attachBubbleImage` — the data-URL `img`), and the turn completes (the mock's grounded answer streams to `done` — the brain bubble is non-empty).
|
||||
- `test_upload_failure_clears_state_and_keeps_the_draft` — a SECOND app/route that FAILS the upload (the route aborts or the handler is pointed at a 413 shape — the simplest: `route.abort("failed")` → `uploadAttachedImage`'s network-drop banner path): mid-flight the uploading state is visible (same assertions); after resolution: the state is CLEARED (no `is-uploading`, Send re-enabled with "Send"), the out-of-turn banner is shown ("Couldn't attach the image — try again." — the network-drop copy), the input text is INTACT, and the strip is back in its normal preview (the attachment remains for the retry — `#attach-preview` not hidden, no `is-uploading`).
|
||||
4. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_image_upload_feedback.py -v --no-cov`), the mock-only gate, the route-delay rationale (the in-flight window is otherwise ~50 ms — unsamplable), the fixture notes, the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% bar is carried by task 01's unit pins; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_image_upload_feedback.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the mid-flight assertions are deterministic (the route delay makes the window ≥ 2 s — the UI's no-poll direct state, asserted by waiting for the class, not a race)
|
||||
- [ ] exactly ONE upload request per Send-press sequence (the spam is visibly AND functionally dead)
|
||||
- [ ] no behavior change in completed phases (the phase-123 upload contract, the A5 path-never-bytes rule, the out-of-turn banner)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Phase 133 — Garbage PDF text is detected; the page-screenshot vision fallback extracts it
|
||||
|
||||
**Source:** `TODO.md` L9 — "test_sources/hnb_2026_guide.tar.gz contains a pdf that is being read as binary. Figure out why. It contains images, tables, and text but none of that is being extracted correctly. Instead, the raw xml is being embedded and the preview shows the entire document as lines like `<illustrator:StartupProfile>Web</illustrator:StartupProfile>`. This project should support this kind of PDF and should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
**FINDING (diagnosed 2026-09-27, before phase authoring — the evidence, verified against the real fixture):**
|
||||
1. The archive holds ONE file: `HBI-2026-Guide-Year-Round_260704_131221.pdf` (6.2 MB, 48 pages, PDFium-produced, **Adobe Illustrator export** — the raw bytes carry 222 × `illustrator:StartupProfile` + 138 × `<x:xmpmeta` uncompressed XMP blocks).
|
||||
2. The CURRENT code (phase 126 onward) extracts it CORRECTLY: `extract_pdf_text` returns 2,585 lines, **0 XML-ish**; a full real-importer run stores clean content (`---PAGE-N---` blocks, "2026 U.S. Benefits Guide" first page), the title, and vision-described embedded images.
|
||||
3. The garbage the owner saw is the **pre-126 import**: `pdf` was already in `BOR_IMPORT_EXTENSIONS` in the local `.env` (untracked — predates the parser) BEFORE `126_pdf_support` shipped (commit `e0f11e7`, 2026-09-27 09:36). The generic text path (`importer.py` L689: `read_text(encoding="utf-8", errors="replace").replace("\x00", "")`) read the raw PDF bytes — the uncompressed XMP/structure blocks survive as literal text (reproduced: 138,170 non-empty lines, **46% XML-ish** — `<</Pages 2 0 R /Type/Catalog>>`, `<illustrator:StartupProfile>Web</…>`, exactly the owner's preview) — and that garbage was chunked + embedded.
|
||||
4. **Production fix is a re-sync**: the importer's content-hash delta sees the old garbage row's digest ≠ the newly extracted text's digest → the next sync re-indexes the file as `updated` with clean content (deploy phase 126 first — it is already on `main`). No migration needed. This is documented here, not a code task.
|
||||
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/pdf.py` (phase 126 — `extract_pdf_text` (L109, `page.get_text("text")` per page, `PAGE_SEPARATOR` blocks, fail-soft `""`), `extract_pdf_images` (L145), the fitz import); `app/rag/importer.py` — `_index_pdf_file` (the phase-126 PDF branch: extract → doc row → digest over extracted text → chunk + embed → lite summary → embedded images via `_describe_or_skip` (L1779)); `app/rag/summarizer.py` — `describe_image` (L208: ONE chat-model call, multimodal user message, `data: bytes` + `mime`, returns `str | None`, no app-level retry); `ImportSummary` (L212, the stats counters + `log()` line); `tests/e2e/mock_llm.py` — the `IMAGE_DESCRIPTION_MODE` canned vision reply (deterministic for the E2E); `test_sources/hnb_2026_guide.tar.gz` (the untracked fixture, 2.6 MB compressed).
|
||||
|
||||
## Objective
|
||||
Two guarantees, one for today and one for the class of problem: (a) the owner's guide is pinned as a regression — it imports through the real pipeline with CLEAN content (the root cause is documented above: a pre-126 binary-text import, self-healed by a re-sync); (b) any future PDF whose extracted text is GARBAGE (XML/binary soup, empty text layer) is DETECTED by a deterministic gate and falls back to the owner's specified path — every page rendered to a screenshot and read by the vision model, preserving what the text layer couldn't.
|
||||
|
||||
## Dependencies
|
||||
- `132_image_upload_feedback` (todo) — pipeline order only; code dependencies are the completed phase-126 PDF pipeline + the phase-122 vision path.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The gate (task 02):** `app/rag/pdf.py` gains `is_usable_pdf_text(text: str) -> bool` — pure, deterministic, unit-testable. Unusable when ANY of: `text.strip() == ""` (the empty text layer — the fallback's other target); the fraction of non-empty lines that look like XML is ≥ 0.30 (a line is XML-ish: `re.search(r"</?[A-Za-z_][\w:.-]*[\s/>=]", line)` — covers `<tag>`, `</tag>`, `<tag attr=`, and the PDF-structure `<</Key…` noise); the non-printable fraction (U+FFFD replacement chars + C0 control chars, over the whole text) is ≥ 0.05 (the binary-soup signal — the pre-126 read is 46% XML-ish AND heavy on U+FFFD; a real document about XML with ≥ 30% tag-like lines is a documented false-positive cost — ASSUMPTION A8). Applied in `_index_pdf_file` AFTER `extract_pdf_text` (the extractor stays pure).
|
||||
- **The fallback (task 03):** when the gate fails, `_index_pdf_file` takes the owner's specified path — "screenshotting the pages and reading each one as an image": `app/rag/pdf.py` gains `render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]` (one PNG per page, PyMuPDF `page.get_pixmap(dpi=dpi).tobytes("png")`, in page order; an unreadable page → a zero-byte entry the caller replaces with the placeholder — no page index may shift); for each page, `describe_image(llm, data=png, mime="image/png")` — the phase-122 vision call, fail-soft per page (`None` → the placeholder line). The stored `content` = `PAGE_SEPARATOR` blocks of the page descriptions (`---PAGE-N---\n<description or "page N — description unavailable">`) — the SAME separator convention as the text path, so the chunker's page-boundary handling works unchanged. `is_image` stays `False` (it is a PDF document whose content happens to be vision text). The digest is over the constructed content (the phase-126 convention — the fallback output is stable for a stable file, so re-syncs don't re-burn vision: a cached row re-imports as unchanged). ONE loud log line (PLAN §9): `pdf: text layer unusable for <source>/<path> (xml_frac=… nonprint=…) — falling back to N page screenshots`. `ImportSummary` gains `pdfs_screenshotted: int = 0` (the counter + the `log()` field; the sync `detail` shape stays untouched — the stats log-only convention).
|
||||
- **The regression pin (task 01):** the real guide, imported through the REAL importer (the fake LLM with a multimodal `chat` override for the embedded images), must land clean content — 0/low XML-ish fraction + the real first-page text present. The fixture stays the tar.gz (the test extracts it into `tmp_path` — never the repo, `test_sources/` is untracked).
|
||||
|
||||
## Tasks
|
||||
1. `01_diagnosis_regression_test.md` — the FINDING is documented in this overview; the task pins the real guide's clean import as a regression integration test.
|
||||
2. `02_pdf_text_gate.md` — `is_usable_pdf_text` (the heuristic) + the pre-126-garbage reproduction fixture.
|
||||
3. `03_page_screenshot_fallback.md` — `render_pdf_pages` + `_index_pdf_file`'s fallback branch + the stats counter.
|
||||
4. `04_e2e_garbage_pdf.md` — dedicated Playwright E2E, run in isolation (the real guide's happy path + a synthetic garbage PDF's fallback path).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the gate (clean text, the REAL pre-126 garbage string — reproduced by reading the fixture with the old `errors="replace"` recipe, XML-heavy text, empty text, a borderline document about XML); `render_pdf_pages` (page count, PNG magic bytes, the zero-byte unreadable-page placeholder); the fallback content assembly (separators, placeholders, the digest stability).
|
||||
- Integration: the real-guide regression (task 01).
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only (the `IMAGE_DESCRIPTION_MODE` canned reply is the fallback's vision surface).
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the real guide imports with clean content through the real pipeline (the regression test green — the owner's exact file)
|
||||
- [ ] a PDF whose text layer is XML/binary garbage (synthetic) lands with the vision-described page content — the owner's specified fallback fires end to end
|
||||
- [ ] a clean PDF NEVER burns a single vision call for the fallback (the gate is the only gate — the happy path is byte-identical to phase 126)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the phase-126 clean-PDF path, the image pipeline, the sync)
|
||||
- [ ] the production stale row is fixed by a re-sync (documented above — verify post-deploy: the guide's content no longer shows XML lines)
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 01 — The regression pin: the real guide imports CLEAN through the real pipeline
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "test_sources/hnb_2026_guide.tar.gz contains a pdf that is being read as binary. Figure out why. ... This project should support this kind of PDF and should be able to extract it correctly"
|
||||
|
||||
## Objective
|
||||
The owner's exact file is a permanent regression: importing `test_sources/hnb_2026_guide.tar.gz`'s PDF through the REAL importer (extraction, digest, chunking, summary, embedded images) lands CLEAN content — the pre-126 XML-garbage import can never come back, and the phase-126 happy path on this 48-page Illustrator PDF is pinned.
|
||||
|
||||
## Work
|
||||
1. `tests/integration/test_pdf_hnb_guide_regression.py` (new) — the house integration pattern (the local compose Postgres, the `tmp_path` fixture, the `tests/fakes.py` `FakeEmbedder`):
|
||||
- Fixture: extract `test_sources/hnb_2026_guide.tar.gz` into `tmp_path` (the test's local source dir — `test_sources/` is untracked, the tar.gz is the stable reference; the extraction is ~6 MB, seconds). The inner file keeps its name (`HBI-2026-Guide-Year-Round_260704_131221.pdf`).
|
||||
- The fake LLM: `FakeEmbedder` with (a) `settings = Settings(_env_file=".env")` (the real extension set — `pdf` is in the owner's live config; a `_env_file=None` fake would silently skip the PDF — the exact trap that hid this class before) and (b) a `chat` override that returns a deterministic string for MULTIMODAL user content (the `list`-of-parts shape `describe_image` sends — the fake's string-only `chat` crashes on it; the override: `"A rendered page of the guide (test vision description)."` — the docstring names it as the task-03-style vision mock the `fakes.py` comment anticipates).
|
||||
- Import with `import_sources([dir], llm)` (the real walk + the real PDF branch).
|
||||
2. Assertions (the observable contract):
|
||||
- `summary.added == 1` (the guide PDF) — the file is ADMITTED by the walk and indexed through the PDF branch (not the text branch: if the text branch ever ran on it again, the content assertions below fail);
|
||||
- the stored `content` is CLEAN: the XML-ish line fraction (< the task-02 gate's 0.30 threshold — assert < 0.05 to be strict: the real extraction is 0/2,585) AND the non-printable fraction < 0.01;
|
||||
- the real first-page text is present: `"2026 U.S. Benefits Guide"` in `content` (pinned fixture text — the file is a stable untracked fixture; if the owner ever re-cuts it, the test failure names the regression, which is the point);
|
||||
- `is_image is False`, `title == "HBI-2026-Guide-Year-Round_260704_131221"` (the stem fallback — the PDF's metadata carries no title, only PDFium creator/producer), the content starts with `---PAGE-1---`;
|
||||
- the embedded-image docs (the phase-126 `#page=…#img=…` rows) got the FAKE vision description (the multimodal `chat` was called — assert `llm.chat_calls` contains ≥ 1 multimodal message) or are absent when the fake "fails" (not this test's scope — the fail-soft is the phase-126 contract).
|
||||
- Wall-time guard: the test completes in < 120 s (48 pages × in-process fakes; if it drifts, the test is too heavy — the suite must stay fast).
|
||||
3. Header docstring: the FINDING summary (the pre-126 root cause in two lines — see the phase overview's FINDING block), the run command (`uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v`), the fixture provenance.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this file IS the task's suite.
|
||||
- Coverage: **>90%** on new/modified code (test-only task — the bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v` green (DB up: `podman compose up -d db`)
|
||||
- [ ] the test fails if the PDF is ever routed through the text branch again (a temporary experiment moving the branch order must break it — the pin has teeth)
|
||||
- [ ] the test does not commit the extracted PDF (it lives in `tmp_path`; `git status` stays clean of binaries)
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,54 @@
|
||||
# Task 02 — `is_usable_pdf_text`: the deterministic garbage gate
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
## Objective
|
||||
A pure, deterministic predicate decides whether an extracted PDF text layer is real content or garbage (the pre-126 XML/binary soup class, the empty text layer class) — so the screenshot fallback has exactly one trigger, unit-testable without any PDF.
|
||||
|
||||
## Work
|
||||
1. `app/rag/pdf.py` — new module-level predicate (place it after `extract_pdf_text`, with a docstring citing the FINDING — the pre-126 `read_text(errors="replace")` soup, the 46%-XML-ish reproduction, the owner's `<illustrator:StartupProfile>` preview):
|
||||
```python
|
||||
#: A line "looks like XML" when it carries a tag-shaped token — the
|
||||
#: pre-126 garbage's signature (raw XMP blocks, PDF-structure noise).
|
||||
_XMLISH_LINE_RE = re.compile(r"</?[A-Za-z_][\w:.-]*[\s/>=]")
|
||||
|
||||
def is_usable_pdf_text(text: str) -> bool:
|
||||
"""Whether an extracted PDF text layer is REAL content (not garbage).
|
||||
|
||||
Unusable (→ the page-screenshot fallback) when ANY of:
|
||||
* ``text.strip() == ""`` — the empty text layer (a scanned/image-
|
||||
only PDF has nothing to read);
|
||||
* ≥ :data:`XMLISH_LINE_FRACTION` (0.30) of the non-empty lines
|
||||
match :data:`_XMLISH_LINE_RE` — the raw-XMP / binary-soup
|
||||
signature (the FINDING's reproduced pre-126 read is 46%);
|
||||
* ≥ :data:`NONPRINT_CHAR_FRACTION` (0.05) of all characters are
|
||||
non-printable (U+FFFD replacement chars + C0 controls) — the
|
||||
``errors="replace"`` binary-soup signal.
|
||||
|
||||
Pure + deterministic (no PDF, no I/O) — the unit tests pin the
|
||||
thresholds against the REAL reproduced garbage string, not a
|
||||
hand-drawn one. A real document ABOUT xml whose tag-like lines
|
||||
reach 30% is a documented false positive (ASSUMPTION A8 — the
|
||||
fallback still extracts it correctly via vision; the cost is a
|
||||
few vision calls).
|
||||
"""
|
||||
```
|
||||
with the two threshold constants (`XMLISH_LINE_FRACTION = 0.30`, `NONPRINT_CHAR_FRACTION = 0.05`) — named, documented, not magic. Non-empty-line counting: `line.strip()` truthy (house convention); the non-printable set: `"\ufffd"` + `unicodedata.category(ch).startswith("C")` (or an explicit C0 range check — pick one, document it; exclude the structural `\\n`/`\\t` the extractor legitimately emits — Cc category would catch them, so use the explicit set: `"\ufffd"` + control chars EXCEPT `\\t\\n\\r` — document the exclusion).
|
||||
2. `tests/unit/test_pdf.py` (extend — the phase-126 suite) — the gate's pins:
|
||||
- clean text (a few real paragraphs + the `---PAGE-N---` separators) → `True`;
|
||||
- THE REAL GARBAGE: reproduce the pre-126 read in the test — read `test_sources/hnb_2026_guide.tar.gz`'s PDF (extract to `tmp_path` — share a module-scoped fixture with the integration test's pattern, or a smaller inline reproduction: the test may instead read the tar.gz's PDF bytes with `read_text(errors="replace").replace("\\x00", "")` and pin `is_usable_pdf_text(garbage) is False` — the REAL string, the owner's exact file, the airtight pin);
|
||||
- the empty text layer (`""` / whitespace) → `False`;
|
||||
- an XML-heavy layer (≥ 30% tag-like lines) → `False`;
|
||||
- a borderline document ABOUT xml (29% tag-like lines, printable) → `True` (the threshold's edge, from below) and 31% → `False` (from above);
|
||||
- the U+FFFD-heavy soup (5%+ replacement chars, few tags) → `False` (the second signal fires alone);
|
||||
- the structural `\\n`/`\\t` exclusion (text full of newlines/indents is printable).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above — this task's suite.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `is_usable_pdf_text` is pure (no PDF dependency) and pinned on the REAL reproduced garbage string (the owner's file, not a lookalike)
|
||||
- [ ] both failure signals (XML-ish fraction, non-printable fraction) are tested independently
|
||||
- [ ] the thresholds are named constants with the false-positive cost documented
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,51 @@
|
||||
# Task 03 — The screenshot fallback: unusable text layer → render pages, read each by vision
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working."
|
||||
|
||||
## Objective
|
||||
When the gate (task 02) says the text layer is garbage or empty, `_index_pdf_file` takes the owner's specified path: every page is rendered to a PNG screenshot and read by the vision model — the stored content is the page-anchored descriptions, so tables/images/whatever the text layer lost are preserved as the model saw them on the page.
|
||||
|
||||
## Work
|
||||
1. `app/rag/pdf.py` — new pure renderer:
|
||||
```python
|
||||
def render_pdf_pages(pdf_path: Path, dpi: int = 150) -> list[bytes]:
|
||||
"""Render every page of *pdf_path* to a PNG (page order, 1-based
|
||||
meaning — list index i is page i+1).
|
||||
|
||||
The fallback's visual input (ASSUMPTION A8: fixed 150 DPI — the
|
||||
balance of vision legibility vs data-URL payload; a page that
|
||||
cannot be rendered yields b"" (the caller's placeholder — the
|
||||
page index NEVER shifts, so ---PAGE-N--- stays truthful).
|
||||
A file that cannot be OPENED yields [] (the caller's existing
|
||||
empty-content path — the doc lands with the no-extractable-text
|
||||
behavior, unchanged).
|
||||
"""
|
||||
```
|
||||
PyMuPDF: `page.get_pixmap(dpi=dpi).tobytes("png")` per page, per-page try/except (the `extract_pdf_text` fail-soft style — log a warning per skipped page).
|
||||
2. `app/rag/importer.py` — `_index_pdf_file`: after `content = extract_pdf_text(full_path)`, insert the gate branch (BEFORE the doc-row upsert, so the digest/chunk/summary pipeline runs on the FINAL content exactly once):
|
||||
```python
|
||||
if not is_usable_pdf_text(content):
|
||||
summary.pdfs_screenshotted += 1
|
||||
content = await _fallback_page_screenshots(session, source=source, rel=rel,
|
||||
full_path=full_path, llm=llm)
|
||||
```
|
||||
with the new module-level helper `_fallback_page_screenshots` (place it near `_index_pdf_file`):
|
||||
- `pages = render_pdf_pages(full_path)`;
|
||||
- the loud one-line log (PLAN §9, the FINDING's signature): `logger.warning("pdf: text layer unusable for %s/%s — falling back to %d page screenshots", source, rel, len(pages))` (the gate's fractions ride the log at DEBUG: `logger.debug("pdf: gate xml_frac=%.2f nonprint=%.2f …")` — computed by a small `_gate_fractions(text)` helper the unit tests pin too, if the warning wants them — optional, keep the required line exact);
|
||||
- per page (index i, page = i+1): `png = pages[i]`; if `png == b""` → `block_text = f"page {page} — description unavailable"`; else `desc = await describe_image(llm, data=png, mime="image/png")` (the phase-122 call — `settings` default, `context` None); `desc is None` or blank → the same placeholder (fail-soft per page — a dead vision endpoint must not sink the doc, the phase-122 fail-soft spirit); else `block_text = desc.strip()`;
|
||||
- `content = "\n\n".join(f"{PAGE_SEPARATOR.format(page=p)}\n{block}" for p, block in ...)` — the SAME separator convention as `extract_pdf_text` (the chunker's page-boundary handling works unchanged);
|
||||
- an ALL-failed render (`pages == []` or every page placeholder) → return `""` (the existing empty-content path — the doc lands as a zero-text doc, the phase-126 no-extractable-text behavior);
|
||||
- the embedded-image extraction (`extract_pdf_images` + `_describe_or_skip`) still runs AFTER the gate branch on BOTH paths — the fallback doc's embedded images join the pipeline exactly as a clean PDF's do (the screenshots are the CONTENT; the extracted images are the citation figures — both, no conflict).
|
||||
3. `app/rag/importer.py` — `ImportSummary`: add `pdfs_screenshotted: int = 0` (documented: "PDFs whose text layer failed the gate and took the page-screenshot vision fallback (phase 133)") + the field in `log()`'s format string (append to the existing line — the house ample-logging shape).
|
||||
4. `app/rag/importer.py` — the digest: the fallback's constructed content is what the sha256 runs over (the existing `digest = hashlib.sha256(content.encode(...))` line — no change needed, just verify it runs AFTER the gate branch so a fallback doc's row is stable across re-syncs: same pages + same vision model → same descriptions → same digest → unchanged on the next sync, no vision re-burn. If `describe_image`'s reply is non-deterministic in production, note it in the docstring — the re-burn is then a cost, not a bug).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_pdf.py` (extend) — `render_pdf_pages`: a 2-page temp PDF (built with fitz in the test — the phase-126 fixture pattern) → 2 PNGs (magic bytes `\\x89PNG`), page order; an unreadable path → `[]`; a corrupted page (if constructible — else the b"" path via a monkeypatched `get_pixmap` raising) → the zero-byte entry at its index, the index intact.
|
||||
- Unit: `tests/unit/test_importer.py` (extend) — the fallback branch, with a REAL tiny garbage PDF (build one in the test: a fitz page whose inserted text is ONLY XML-ish lines — `<foo:bar>x</foo:bar>` repeated — so the gate fails deterministically) + a fake LLM whose multimodal `chat` returns a canned page description: the stored doc's content = `---PAGE-1---\\n<canned>` (the separator convention), `summary.pdfs_screenshotted == 1`, the loud warning logged (the caplog pin); a CLEAN tiny PDF (the phase-126 fixtures) → `pdfs_screenshotted == 0` and the vision multimodal `chat` is NEVER called for the fallback (only the embedded-image describes — the gate never burns vision on a clean layer).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a garbage-text PDF lands with the page-screenshot content (the synthetic test: `---PAGE-N---` + the canned descriptions)
|
||||
- [ ] a clean PDF's path is byte-identical to phase 126 (no fallback, no extra vision calls, `pdfs_screenshotted == 0`)
|
||||
- [ ] the fallback is fail-soft per page (a dead vision call → the placeholder line, the doc still lands)
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 04 — E2E: the real guide stays clean, a synthetic garbage PDF takes the screenshot path
|
||||
|
||||
**Phase:** `133_pdf_screenshot_fallback` · **Source:** `TODO.md:9` — "...the raw xml is being embedded and the preview shows the entire document as lines like `<illustrator:StartupProfile>Web</illustrator:StartupProfile>`. This project should support this kind of PDF and should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image..."
|
||||
|
||||
## Objective
|
||||
A dedicated, mock-only Playwright suite (A16 — one per phase, run in isolation) pins both legs of the guarantee at the user surface: (a) the owner's REAL guide, imported through the admin sync, previews CLEAN text (no XML lines — the exact complaint, negated); (b) a synthetic PDF whose text layer is pure XML garbage takes the screenshot fallback end to end — its content is the mock's canned vision descriptions, and the preview shows them.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_garbage_pdf.py` (new) — house pattern (module app fixture over the shared E2E Postgres, host temp-dir `kind=local` source registered via the admin git-sources API, autouse truncate cleanup, mock-only — `E2E_REAL_LLM=1` unsupported: the fallback's vision surface is the deterministic `IMAGE_DESCRIPTION_MODE` canned reply).
|
||||
App env: `BOR_IMPORT_EXTENSIONS` must include `pdf` (the owner's live config — the app fixture's env, the phase-126/127 suites' pattern) and `BOR_IMAGES=1` (the embedded-image pipeline is live).
|
||||
2. Fixtures (two local sources, two tests):
|
||||
- `hnb/` — the REAL guide: extract `test_sources/hnb_2026_guide.tar.gz` into the temp dir (module-scoped, once — the 6 MB extraction is seconds).
|
||||
- `garbagedoc/` — a synthetic garbage PDF built IN the test with PyMuPDF (the fixture pattern from task 03's unit test, lifted here): 2 pages, each page's ONLY text = repeated XML-ish lines (`<illustrator:StartupProfile>Web</illustrator:StartupProfile>` × ~20 — the owner's exact line shape, so the test is a faithful reproduction of the report), plus one small embedded raster image per page (fitz `insert_image` from a tiny PNG built in the test — the fallback's embedded-image leg runs too).
|
||||
3. Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_real_guide_previews_clean_text` — admin sync over `hnb/` settles `success`; `GET /api/documents/content?source=hnb&path=<the guide's pdf>` (the PREVIEW surface — the endpoint the document viewer's content uses): the body contains `"2026 U.S. Benefits Guide"` (the real first-page text) and NO line matching the XML-ish regex (the owner's complaint, asserted on the served content, not the DB); the document viewer page (`/document.html?source=hnb&path=…`) renders that content (the preview shows real text, not `<illustrator:…>` lines).
|
||||
- `test_garbage_pdf_takes_the_screenshot_path` — admin sync over `garbagedoc/` settles `success`; the stored doc's content (the same content endpoint): starts with `---PAGE-1---`, contains the mock's CANNED vision description (the `IMAGE_DESCRIPTION_MODE` reply's deterministic text — assert the first 40 chars) for BOTH pages (2 separators, 2 descriptions), and contains NO `<illustrator:StartupProfile>` line (the garbage text layer is NOT what got stored — the screenshot path replaced it); the sync's `detail` is `success` with `added == 1` (the garbage PDF is a normal import — the fallback is invisible in the status shape; the loud log line is server-side, PLAN §9).
|
||||
- `test_clean_pdf_never_burns_the_fallback` — a third source `cleandoc/` with ONE small clean PDF (fitz, real prose text): sync → the stored content is the PROSE (the text layer, byte-faithful to the extraction) — NOT the canned vision description (the fallback did not fire on a clean layer — the gate's no-false-negative pin at the E2E level).
|
||||
4. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`), the mock-only gate + why (the canned vision reply makes the fallback deterministic), the fixture provenance (the real tar.gz + the synthetic builder), the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% `app/` bar is carried by tasks 02–03; the real-guide regression by task 01; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the real guide's PREVIEW (the owner's exact surface) is clean — the `<illustrator:StartupProfile>` line is asserted ABSENT on the served content
|
||||
- [ ] the synthetic garbage PDF's stored content is the canned vision descriptions (the fallback fired end to end, mock-deterministic)
|
||||
- [ ] the clean PDF is byte-faithful to its text layer (no false-positive fallback)
|
||||
- [ ] no behavior change in completed phases (the phase-126 pipeline, the sync, the document viewer)
|
||||
@@ -0,0 +1,36 @@
|
||||
# Phase 134 — The owner can cancel a running sync from the Sources page
|
||||
|
||||
**Source:** `TODO.md` L3 — "Need to be able to cancel a sync after I've clicked the button."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/api/sync.py` — `SyncStatus` (the four-state dataclass: `idle` / `running` / `success` / `failed`, phase-64 progress fields, phase-98 phase-machine fields), the module-level `_status` + `_task`, `POST /api/sync` (202 / 409 one-sync-at-a-time), `GET /api/sync/status`, and `_run_sync` — the in-process `asyncio` background task: the blocking git prelude (`scripts.git_sync.clone_or_pull` → `subprocess.run`, no await inside), then `await import_sources(...)` (per-file LLM awaits — a cancellation point between files), then `await regenerate_overview(...)` and `await generate_folder_summaries(...)` (both only on a changed KB / gap). The module docstring's shutdown contract: `CancelledError` is deliberately NOT swallowed — app shutdown cancels the task, and swallowing would mask a real stop. `import_sources` closes its own private session in a `finally` (leak-safe under a mid-import cancel). `frontend/assets/sources.js` — the sync-button lifecycle: `enterSyncRunningState` (live click + re-attach + every running tick), `settleSyncButton` (every terminal), `applySyncSuccess` / `applySyncFailure` / `applySyncIdle`, `startSyncPolling` (the 2 s two-job decision tree, 7 branches), `initSyncButton` (load-time re-attach). `frontend/index.html` — the `#sync-btn` markup (the Sources view section). E2E patterns: `tests/e2e/test_sync_button.py` (per-module app on a dedicated port, the `file://` fixture repo, the `_wait_sync_done` status-poll helper), `tests/e2e/slow_llm.py` (the delay-injecting proxy that makes a run's duration deterministic), `tests/e2e/mock_llm.py`. Unit: `tests/unit/test_sync_button.py`; integration: `tests/integration/test_sync_api.py`.
|
||||
|
||||
## Objective
|
||||
A sync that is already running can be stopped by the owner: while a run is in flight the Sources page shows a **Cancel** button; one click requests the cancel, the pipeline stops at the next boundary, the run lands in a new `cancelled` terminal state (reported by the status endpoint and rendered by the UI as a neutral result — cancel is intent, not failure), and the controls settle back to the retry-ready idle state. A subsequent sync starts cleanly (the partial run leaves a consistent KB — the importer's content-hash delta makes the next run idempotent).
|
||||
|
||||
## Dependencies
|
||||
- `133_pdf_screenshot_fallback` (todo) — pipeline order only; no code dependency.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The fifth state — `cancelled`** (ASSUMPTION A1): `SyncStatus.state` becomes `Literal["idle", "running", "success", "failed", "cancelled"]`. It is TERMINAL like `success` / `failed`: carries `finished_at`, clears `current_file` / `phase` / `current_summary` but keeps the run's final `files_done` / `files_total` and summary counts (the phase-64/98 keep-final-counts convention), `detail = {}`, and **`error = None`** (no error string — the UI never treats it as a failure: no error banner, no error modal, just the neutral "Sync cancelled" result line). `GET /api/sync/status` gains nothing structurally — the `state` key simply reports `"cancelled"`.
|
||||
- **`POST /api/sync/cancel`** (same `router`, so `require_admin` — 403 anonymous): while a run is in flight → set the module-level `_cancel_event` (an `asyncio.Event`, next to `_status` / `_task`) AND call `_task.cancel()` (unblocks whatever `await` the pipeline is parked on — e.g. a per-file LLM call inside the import) → **202** `{"detail": "cancel requested"}`. No run in flight (`_task` None or done) → **409** `{"detail": "no sync is running"}`. `start_sync` resets `_cancel_event` (clear) when it starts a run.
|
||||
- **Cancellation semantics — cooperative, best-effort** (ASSUMPTION A2): `_run_sync` gains (a) a private `_check_cancelled()` helper that raises `asyncio.CancelledError()` when `_cancel_event.is_set()`, called at the pipeline boundaries — the top of the per-source loop (before each clone/pull or local-dir check), immediately before `import_sources(...)`, and immediately before the `if summary.added + summary.updated > 0:` block (covers the overview + summaries spans); and (b) an `except asyncio.CancelledError:` clause (placed before the existing `except Exception`): if `_cancel_event.is_set()` → land the run in the `cancelled` terminal state + one INFO log line (PLAN §9: `sync: cancelled by user state=cancelled`), then `raise` (the task ends cancelled — `_task.done()` is True, the status object stays authoritative); otherwise (no user cancel — the app-shutdown path) → `raise` unchanged (the module docstring's shutdown contract holds: an unrequested cancellation is never swallowed). Consequence the UI/tests may rely on: a git subprocess already running at the cancel moment runs to completion (the blocking `subprocess.run` has no await inside) — the cancel takes effect at the NEXT boundary (after that source's git ops, between files, or before overview/summaries). During the import span the `_task.cancel()` from the endpoint interrupts at the next per-file await.
|
||||
- **The UI** (ASSUMPTION A3): a `#sync-cancel-btn` button in the same sync-control area as `#sync-btn` (`frontend/index.html`), `hidden` by default; secondary button styling consistent with the theme (contrast ≥4.5:1, `:focus-visible` ring — house UI rules). `sources.js` rules: SHOWN + ENABLED only while a SYNC run is running (`enterSyncRunningState` with `kind === "sync"` — the upload job is NOT cancellable in this phase, so the cancel button stays hidden while only the upload run drives the button); HIDDEN + DISABLED by `settleSyncButton` (every terminal/idle settle) and by every `applySync*` path. Click → `requestCancel()`: the cancel button disables IMMEDIATELY (never stale, no double request), `POST /api/sync/cancel`; 403 → the whoami-backstop hide-everything branch (same as the other endpoints); 409 → a harmless race (the run ended between ticks — the poll settles on the next tick, no error copy); 202 → the button stays disabled until the terminal (the poll's new `cancelled` branch settles everything); any other non-2xx → the phase-55 neutral retry copy in `#sync-result` ("Couldn't cancel the sync — try again.") + the cancel button re-enables. New poll-tree branch (after the `failed` branch, before the idle branch): `syncStatus.state === "cancelled"` → stop polling, settle the main button to "Sync sources", `#sync-result` = "Sync cancelled" (the aria-live announcer reads it), hide the error banner, hide the cancel button, `emitSyncStatus`. `initSyncButton` re-attach: `state === "cancelled"` → the same neutral settle (a page reload during/right after a cancel shows "Sync cancelled", never a stuck running state).
|
||||
|
||||
## Tasks
|
||||
1. `01_backend_cancel_state.md` — the `cancelled` state, `POST /api/sync/cancel`, and the cooperative cancel mechanics in `_run_sync`.
|
||||
2. `02_frontend_cancel_button.md` — the `#sync-cancel-btn` UI: show/hide rules, the cancel request lifecycle, the `cancelled` terminal branch in the poll tree + re-attach.
|
||||
3. `03_e2e_sync_cancel.md` — the dedicated Playwright E2E (the slow-LLM cancel window, the API contract, the full UI lifecycle, the clean re-sync after cancel).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cancel endpoint's contract (202/409/403), `_check_cancelled`, the `CancelledError` conversion (event set → `cancelled` state + re-raise; event unset → re-raise with the state untouched), the `SyncStatus` reset on a fresh start (a stale cancel event cannot bleed into the next run).
|
||||
- Integration: a real run started through the API is cancelled mid-run → the status reports `cancelled` (with `error: None`); a second run after the cancel completes successfully (the restart is clean).
|
||||
- E2E: one dedicated Playwright file, run in isolation (`tests/e2e/test_sync_cancel.py`) — the mock-only app with the `slow_llm` proxy for a deterministic cancel window.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST /api/sync/cancel` (admin): 202 while a run is in flight, 409 `no sync is running` otherwise, 403 anonymous
|
||||
- [ ] a cancelled run lands in the `cancelled` terminal state (`error: None`, final counts kept) and the status endpoint reports it
|
||||
- [ ] the app-shutdown cancel contract is preserved (an unrequested `CancelledError` is never swallowed — a unit test pins it)
|
||||
- [ ] the UI: the Cancel button is visible exactly while a sync run is running, the `cancelled` terminal settles to "Sync cancelled" with no error banner/modal, and a re-sync after a cancel succeeds
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the uncancelled run is byte-identical: same states, same labels, same 409 double-trigger)
|
||||
@@ -0,0 +1,63 @@
|
||||
# Task 01 — Backend: the `cancelled` state, `POST /api/sync/cancel`, and the cooperative cancel in `_run_sync`
|
||||
|
||||
**Phase:** `134_sync_cancel` · **Source:** `TODO.md:3` — "Need to be able to cancel a sync after I've clicked the button."
|
||||
|
||||
## Objective
|
||||
The sync API can stop its own in-flight run: a new admin-only `POST /api/sync/cancel` requests the cancel; `_run_sync` converts a user-requested cancellation into a new `cancelled` terminal state (never an error); the shutdown-cancel contract is untouched.
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py` — the state machine:
|
||||
- `SyncStatus.state`: `Literal["idle", "running", "success", "failed", "cancelled"]` (the docstring's four-state paragraph becomes five; `cancelled` is terminal like `success` / `failed`).
|
||||
- Module level, next to `_status` / `_task`: `_cancel_event = asyncio.Event()`.
|
||||
- `start_sync`: reset the run's cancellation — `_cancel_event.clear()` before `_task = asyncio.create_task(_run_sync())` (a stale event from a previous cancelled run must never bleed into the next run).
|
||||
2. `app/api/sync.py` — the endpoint (same `router` → `require_admin` applies, 403 anonymous):
|
||||
```python
|
||||
@router.post("/cancel", status_code=202)
|
||||
async def cancel_sync() -> dict[str, str]:
|
||||
if _task is None or _task.done():
|
||||
raise HTTPException(status_code=409, detail="no sync is running")
|
||||
_cancel_event.set()
|
||||
_task.cancel()
|
||||
return {"detail": "cancel requested"}
|
||||
```
|
||||
202 is answered the moment the request is REGISTERED (the run may already be terminal — the UI's poll then settles on the terminal state; a 202-then-immediate-404-style race is impossible because the status object, not the task, is authoritative).
|
||||
3. `app/api/sync.py` — `_run_sync` mechanics:
|
||||
- A private helper: `def _check_cancelled() -> None:` — `if _cancel_event.is_set(): raise asyncio.CancelledError()`.
|
||||
- Call it at the boundaries: (a) the top of the per-source loop (before the `kind == "git"` / `kind == "local"` split), (b) immediately before `await import_sources(...)`, (c) immediately before the `if summary.added + summary.updated > 0:` block (covers both the overview and the summaries spans).
|
||||
- New `except asyncio.CancelledError:` clause placed BEFORE the existing `except Exception`:
|
||||
```python
|
||||
except asyncio.CancelledError:
|
||||
if _cancel_event.is_set():
|
||||
_status.state = "cancelled"
|
||||
_status.finished_at = datetime.now(UTC)
|
||||
_status.error = None
|
||||
_status.current_file = None
|
||||
_status.phase = None
|
||||
_status.current_summary = None
|
||||
logger.info(
|
||||
"sync: cancelled by user state=cancelled files_done=%d files_total=%d",
|
||||
_status.files_done, _status.files_total,
|
||||
)
|
||||
raise
|
||||
raise # app shutdown — the module-docstring contract: never swallow an unrequested cancel
|
||||
```
|
||||
The terminal keeps `files_done` / `files_total` and the summary counts (the phase-64/98 keep-final-counts convention); `detail` stays `{}` (a partial run reports no counts — ASSUMPTION A1: `error` is None, `detail` is empty).
|
||||
- Update the module docstring: the state machine is five-state; the cancel semantics are cooperative (a git subprocess in flight at the cancel moment runs to completion — the cancel lands at the next boundary); the `CancelledError` paragraph now names BOTH conversion paths (user-requested → `cancelled` state + re-raise; unrequested → re-raise).
|
||||
4. `GET /api/sync/status` — no shape change; update its docstring (the `state` value set now includes `cancelled`; `error` is null in that state).
|
||||
5. Unit tests — `tests/unit/test_sync_button.py` (extend) or a new `tests/unit/test_sync_cancel.py` (your call — the existing file owns the endpoint's contract, so extending it is the default):
|
||||
- `POST /api/sync/cancel` anonymous → 403; idle (never started, and after a terminal run) → 409 with the exact detail `no sync is running`.
|
||||
- A started run: the endpoint returns 202 `{"detail": "cancel requested"}`, sets the event, and cancels the task (drive `_run_sync` with a controllable await — e.g. a monkeypatched `import_sources` that awaits a `asyncio.Event` the test controls — then cancel via the endpoint and assert the task finished + `_status.state == "cancelled"`, `error is None`, `finished_at` set, final counts kept, `detail == {}`).
|
||||
- The shutdown contract pin: cancel the task WITHOUT setting the event (bare `task.cancel()`) → `_status.state` is NOT `cancelled` (stays `running` — the task died mid-run without a terminal state, exactly as today's shutdown path does) and the `CancelledError` propagated.
|
||||
- The stale-event pin: a run that was cancelled, then a FRESH run starts → `_cancel_event` is cleared (the new run completes to `success` with no cancel request).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the behaviors in Work step 5 (the suite above IS the task's unit layer).
|
||||
- Integration: `tests/integration/test_sync_api.py` (extend) — start a real run against the `file://` fixture pattern the file already uses (a slow/many-file fixture or a monkeypatched delay at the per-file hook seam — follow the file's existing fakes), `POST /api/sync/cancel` while running → poll `GET /api/sync/status` → `state == "cancelled"` (and `error is None`); then a second full run → `state == "success"` (the restart after a partial cancel is clean — the importer's hash delta re-imports what the cancel skipped).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_sync_button.py tests/integration/test_sync_api.py -v` green (plus the new test file, if split)
|
||||
- [ ] the five-state machine is exercised in tests: `cancelled` is reachable ONLY through a user-requested cancel (the unrequested-cancel pin is green)
|
||||
- [ ] the module docstring names the cooperative boundary semantics (a future reader sees why a long clone can outlive the click)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change for the uncancelled run (the existing sync tests stay green untouched)
|
||||
@@ -0,0 +1,45 @@
|
||||
# Task 02 — Frontend: the `#sync-cancel-btn` and the `cancelled` terminal branch in the poll tree
|
||||
|
||||
**Phase:** `134_sync_cancel` · **Source:** `TODO.md:3` — "Need to be able to cancel a sync after I've clicked the button."
|
||||
|
||||
## Objective
|
||||
The Sources page shows a Cancel button exactly while a sync run is in flight (live click, re-attached run, every running poll tick); clicking it requests the cancel and the UI settles into the neutral "Sync cancelled" terminal — no error banner, no modal, never stale.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` (the Sources view section, next to `#sync-btn`): add the cancel control —
|
||||
```html
|
||||
<button type="button" id="sync-cancel-btn" class="sync-cancel-btn" hidden>Cancel</button>
|
||||
```
|
||||
- `hidden` by default (anonymous-safe + the idle state; the module reveals it, the same convention as `#sync-btn`).
|
||||
- A visible "Cancel" label (no icon-only — WCAG 2.4 name/role; the accessible name is the text).
|
||||
- `frontend/assets/styles.css`: `.sync-cancel-btn` — a secondary button style consistent with the theme (the house tokens, contrast ≥4.5:1 on `--surface`, `:focus-visible` ring like the other buttons, disabled state). No layout regression on the sync-control row (mobile included — the row already wraps; verify the 360 px case).
|
||||
2. `frontend/assets/sources.js` — the lifecycle (all inside the existing sync section; the §7.4 never-stale contract applies exactly as to the main button):
|
||||
- Element lookup: `const syncCancelBtn = root.querySelector("#sync-cancel-btn");`
|
||||
- Show/hide helpers: `showSyncCancelBtn()` (unhide + enable + `aria-label="Cancel sync"`) / `hideSyncCancelBtn()` (hide + disable).
|
||||
- `enterSyncRunningState(kind, …, status)`: when `kind === "sync"` → `showSyncCancelBtn()`; when `kind === "upload"` → `hideSyncCancelBtn()` (the upload job is NOT cancellable in this phase — the cancel button belongs to the sync run only).
|
||||
- `settleSyncButton(label)`: always `hideSyncCancelBtn()` (every terminal/idle settle — sync success/failed/cancelled, upload success/failed, both idle).
|
||||
- `requestCancel()` + the click listener (armed next to `syncBtn.addEventListener("click", startSync)`):
|
||||
- `syncCancelBtn.disabled = true` FIRST (never stale, one request at a time), `#sync-result` cleared.
|
||||
- `POST /api/sync/cancel`: 403 → the whoami-backstop branch (stop polling, hide the main button, `applySyncIdle()`) — the same depth as every other 403 in this module; 409 → NO-OP (the run ended between the tick and the click — the poll settles on the next tick; the cancel button is hidden by that settle; no error copy); 202 → nothing (the button stays disabled — the poll's `cancelled` branch settles everything); any other non-2xx / network failure → `#sync-result` = "Couldn't cancel the sync — try again." (the phase-55 neutral retry copy) + `syncCancelBtn.disabled = false` (the run is still going — the owner can retry).
|
||||
- New terminal branch in `startSyncPolling`'s tick — placed AFTER the `failed` branch and BEFORE the upload-success branch:
|
||||
```js
|
||||
if (syncStatus.state === "cancelled") {
|
||||
stopSyncPolling();
|
||||
applySyncCancelled(syncStatus);
|
||||
return;
|
||||
}
|
||||
```
|
||||
- `applySyncCancelled(status)`: `settleSyncButton("Sync sources")` (re-enables the main button, hides the cancel button), `syncResult.textContent = "Sync cancelled"`, `hideSyncError()`, `emitSyncStatus(status)`. No modal, no error styling (ASSUMPTION A1 — cancel is intent).
|
||||
- `initSyncButton` re-attach: `status.state === "cancelled"` → `applySyncCancelled(status)` (a reload during/right after a cancel shows the neutral result, never a stuck running state).
|
||||
- Update the section-header docstring (the decision tree gains the `cancelled` branch — the tree's "7" becomes "8" — and the cancel button's show/hide contract is documented there, the house pattern).
|
||||
3. No change to the upload job, the main sync button's existing labels, or the `bor:sync-status` event shape (the `cancelled` state simply flows through `emitSyncStatus` — the event's `detail` is the status object, as for every other state).
|
||||
|
||||
## Testing & Quality
|
||||
- No JS unit layer in this repo — the UI contract is pinned by the phase's E2E (task 03) and by the never-stale rules above.
|
||||
- Coverage: **>90%** on new/modified code (test-only-for-`app/` task — the bar is carried by the suite run; the JS is covered by the E2E).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the Cancel button is visible + enabled ONLY while a sync run is running (hidden on boot/idle, hidden while only the upload run drives the button, hidden on every terminal)
|
||||
- [ ] clicking Cancel settles to "Sync cancelled" in `#sync-result` with the main button re-enabled, the cancel button hidden, no error banner and no modal
|
||||
- [ ] a 409 cancel (race) is silent; a failed cancel request shows the neutral retry copy and re-enables the button (the run keeps going)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean; the page renders with no console errors (verified in the E2E, task 03)
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 03 — E2E: `tests/e2e/test_sync_cancel.py` (the dedicated Playwright suite, run in isolation)
|
||||
|
||||
**Phase:** `134_sync_cancel` · **Source:** `TODO.md:3` — "Need to be able to cancel a sync after I've clicked the button."
|
||||
|
||||
## Objective
|
||||
The owner's cancel path is pinned end to end against the REAL pipeline: a real `file://` fixture sync whose duration is long enough to cancel (the `slow_llm` proxy), the API contract (403/409/202), the UI lifecycle (Cancel button visible → click → "Sync cancelled" settle), and the clean re-sync after a partial run.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_cancel.py` (new) — the `test_sync_button.py` module-app pattern, adapted:
|
||||
- **Ports/env:** per-module app on its own port — `APP_PORT = int(os.environ.get("E2E_APP_PORT_SYCNCEL", "8133"))`; `BOR_STATIC_DIR`, `BOR_RELEVANCE_THRESHOLD=0.30` (mock-calibrated, the conftest pattern), `BOR_ADMIN_PASSWORD` / `BOR_SESSION_SECRET` from `e2e.conftest`, `BOR_GIT_SOURCES=file://<fixture repo>`, a dedicated `BOR_SOURCES_DIR`.
|
||||
- **The slow-LLM proxy:** spawn `tests/e2e/slow_llm.py` the conftest way (the `SLOW_LLM_DELAY_S` env — 0.3 s is a good default: ~40 fixture files → ~12+ s of import — a wide, deterministic cancel window against the UI's 2 s poll) in front of the mock LLM (the `mock_llm` fixture from `e2e.conftest`); `BOR_LLM_BASE_URL` points at the proxy.
|
||||
- **The fixture repo** (module-scoped, `tmp_path_factory`, real `git` subprocess — the `sync_git_repo` pattern): ~40 small A9-format `.md` files under a `notes/` folder (each one unique enough to import independently; ONE sentinel file — `Marker: RESE-CANCEL-SENTINEL-4f1e` — the assertion target for the re-sync).
|
||||
- **`_wait_sync_state(page, app_url, states, timeout_s)`** — the `test_sync_button._wait_sync_done` helper generalized to a set of terminal states (here `{"cancelled"}` and `{"success", "failed"}` for the re-sync).
|
||||
2. Tests (Playwright Mapping Rule — each test names the behavior it pins):
|
||||
- `test_anonymous_cancel_403` — anonymous: `POST /api/sync/cancel` → 403 (the admin-only surface, same as the other two endpoints); `#sync-cancel-btn` is never revealed (the whoami gate).
|
||||
- `test_cancel_when_idle_409` — logged-in, no run in flight: `POST /api/sync/cancel` → 409 with the exact detail `no sync is running`; the Cancel button is hidden in the UI (boot + idle).
|
||||
- `test_admin_cancel_lifecycle` — the owner's exact flow:
|
||||
1. login (lands on `/sources.html`);
|
||||
2. click `#sync-btn` → `#sync-label` "Syncing…", the main button disabled, **`#sync-cancel-btn` visible + enabled** (within the 2 s poll tick — the running state is already entered synchronously by the 202);
|
||||
3. confirm via the status endpoint that the run is `running` (the deterministic layer — `page.request` poll at ~100 ms, the `slow_llm` module's convention);
|
||||
4. click `#sync-cancel-btn` → the cancel button disables immediately (never stale);
|
||||
5. `_wait_sync_state(…, {"cancelled"})` → the status endpoint reports `state == "cancelled"` with `error is None`;
|
||||
6. the UI settles (timeout-bound): main button ENABLED + label "Sync sources", `#sync-cancel-btn` hidden, `#sync-result` text "Sync cancelled", the error banner hidden, no error modal in the DOM;
|
||||
7. **the re-sync is clean:** click `#sync-btn` again → "Synced HH:MM" (the `SYNCED_LABEL` regex) + `#sync-result` includes "40 added" (or the fixture's file count — the first FULL pass; the cancelled run's partial import is idempotent under the hash delta — assert the sentinel row is present in the catalog via the drill-down tree, the `_expect_fixture_row` pattern) → the cancel never corrupted the KB.
|
||||
- Wall-time guard: the whole module < 120 s (40 files × 0.3 s proxy delay + the re-sync's 40 × 0.3 s — if it drifts, cut the file count, not the guard).
|
||||
3. Header docstring: the story ("the owner's TODO item, L3"), the run command (`uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov`), the isolation note (per-module app, dedicated port — the combined-run convention), the slow-LLM provenance (the proxy is the only thing that makes the cancel window deterministic).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite; it runs in ISOLATION (`--no-cov`, the AGENTS.md rule 9 gate) with the DB up (`podman compose up -d db`) and git on PATH.
|
||||
- Coverage: **>90%** on new/modified code (test-only task — the bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` green in isolation
|
||||
- [ ] the cancel window is deterministic (the test does not race the 2 s poll — the `slow_llm` delay is the gate; a rerun on a quiet machine passes)
|
||||
- [ ] the re-sync after a cancel imports the FULL fixture (the sentinel row present — a partial cancel never leaves a corrupted or permanently-incomplete KB)
|
||||
- [ ] the `cancelled` state's `error` is null end to end (the UI can never render it as a failure)
|
||||
@@ -0,0 +1,40 @@
|
||||
# Phase 135 — A git source can pin the branch it mirrors
|
||||
|
||||
**Source:** `TODO.md` L4 — "Need to be able to specify the branch for a git source"
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/models.py::GitSource` (id, url, kind, path, ignore_paths, include_hidden, token, added_at — no branch column; the phase-121 `token` column is the nullable-write-only precedent); `alembic/versions/` (0023 is the newest; every migration is reversible — the house rule); `app/schemas.py` — `GitSourceIn` / `GitSourceOut` / `GitSourceRow` / `GitSourcePatchIn` (the token field is the write-trimmed precedent; the output shapes use `extra="forbid"`); `app/api/git_sources.py` — `POST` (the `_create_git_row` / `_create_local_row` split, the fixed-detail 422s, the token normalization), `PATCH /{source_id}` (the per-field present-wins tri-state — the token's absent/None = no change, `""` = clear, non-empty = replace), `GET` (DB rows vs the env-fallback synthetic rows); `scripts/git_sync.py::clone_or_pull(url, dest)` (fresh clone with full history; the shallow probe + one-time `git fetch --unshallow` self-heal (phase 107 D11); `git pull --ff-only` — the fail-loud D12 contract); the TWO call sites that resolve effective sources and clone git rows: `app/api/sync.py::_run_sync` and `scripts/import_docs.py::_resolve_sources` (both already pass `clone_url_for(row)` + `repo_name(row.url)`); `file_commit_dates` (walks the checkout — untouched); the UI: `frontend/index.html` — the add form `#git-source-form` (the url + token inputs) and the per-row editor dialog `#ignore-editor-dialog` (the source line, the ignore textarea, the token field); `frontend/assets/git-sources.js` — `loadSources` / `makeRow` (the "hidden on" text-tag idiom in the source cell), `wireAddForm` (`opts.body(value, token)` builds the POST body), the dialog open/save handlers (the PATCH body builder), the `toggleHidden` never-stale lifecycle as the per-row control precedent.
|
||||
|
||||
## Objective
|
||||
A git source can specify the branch it mirrors: the owner gives a branch when adding a git source (or edits it per-row later), the clone/pull honors it (a fresh clone checks the branch out; an existing checkout tracks it), BOTH entry points — the Sync button and the `import_docs` CLI — honor it, and every row without a branch (all existing rows + the env-fallback rows) behaves byte-identically to today (default branch, `git pull --ff-only`).
|
||||
|
||||
## Dependencies
|
||||
- `134_sync_cancel` (todo) — pipeline order only; no code dependency (independent surfaces).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The column** (ASSUMPTION B1): `git_sources.branch` — `TEXT NULL`, server default NULL; alembic **0024** (reversible — `add_column` / `drop_column`, no data migration: NULL = "the repo's default branch", so every existing row is unchanged by definition). `GitSource.branch: Mapped[str | None] = mapped_column(Text, nullable=True)`.
|
||||
- **Validation** (ASSUMPTION B2): a pure helper `validate_branch_name(raw: str) -> str` in `app/rag/git_sources.py` (next to `normalize_credential` — the write-path helper home; the API layer maps its `ValueError` to the fixed-detail 422). Contract: trim → reject empty; reject length > 200; reject anything not matching `^[A-Za-z0-9._/-]+$` (alphanumerics + dot, underscore, hyphen, slash — whitespace/`@`/`~`/`^`/`:`/`?`/`*`/`[`/backslash are all out); reject a name containing `..` or `//`; reject a leading `-`, `.`, or `/`; reject a trailing `/` or `.`; reject `HEAD` (case-insensitive — git's refname; a branch literally named `head` is a documented cost of the simple rule). Returns the trimmed value. The API's 422 detail is the FIXED string `not a valid branch name` (never echoes the input — the house credential-safety discipline, applied for consistency).
|
||||
- **The write paths** (ASSUMPTION B3): `GitSourceIn.branch: str | None` (optional at create — absent/empty → NULL; trimmed in the schema before the length constraint, the `_trim_url` precedent). `GitSourcePatchIn.branch: str | None = None` — TRI-STATE exactly like the token: absent/None = no change; `""` = clear (stored NULL); non-empty = set (re-validated). A `branch` on a LOCAL row (POST or PATCH) → 422 fixed detail `branch is only valid for git sources` (local rows are walked, not cloned). `GitSourceOut.branch: str | None` + `GitSourceRow.branch: str | None` (env-fallback synthetic rows report `None` — no DB row to store one on, the `ignore_paths: []` precedent).
|
||||
- **`clone_or_pull(url, dest, branch: str | None = None)`** (ASSUMPTION B4): `branch=None` → the EXISTING code path byte-identical (same argv: clone without `--branch`; probe; unshallow-if-shallow; `git pull --ff-only`). A fresh clone with a branch → `git clone --branch <b> url dest`. An existing checkout with a branch → (1) the HEAD GUARD first: `git rev-parse --abbrev-ref HEAD` — if the result is not `<b>` → `GitSyncError("checkout <dest> is on branch '<current>' but the source specifies branch '<b>' — remove and re-add the source to switch branches")` (ASSUMPTION B5 — fail loud, the D12 convention; NO silent checkout switch in this phase: the checkout may serve other rows' documents, and a switch would silently re-map the source's file set; phase 136's explicit-ref reset makes the ref unambiguous later without changing this guard); (2) the shallow probe + one-time unshallow (unchanged); (3) `git fetch origin <b>`; (4) `git merge --ff-only FETCH_HEAD` (the single-fetch ff pull — the sequence phase 136 extends with its reset fallback).
|
||||
- **Wiring:** both call sites pass `row.branch` — `app/api/sync.py::_run_sync` and `scripts/import_docs.py::_resolve_sources`: `clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url), row.branch)`. Env-fallback synthetic rows carry `branch=None` (the `GitSource(url=…, kind="git")` constructor default). `file_commit_dates`, the checkout naming (`repo_name` stays on the BARE url), and the ignore/hidden/date maps are untouched.
|
||||
- **The UI** (ASSUMPTION B6): the add form gains an optional `#git-source-branch` text input (maxlength 200, `autocomplete="off"`, hint "optional — default: the repo's default branch") between the URL and Token fields; the submit body becomes `(url, token, branch) => ({ url, ...(branch ? { branch } : {}), ...(token ? { token } : {}) })` — a blank branch OMITS the key (NULL). The per-row editor dialog gains a `#ignore-editor-branch` field (visible for GIT rows only — local rows' dialogs hide it; it opens prefilled from `s.branch ?? ""`; the PATCH body for a git row ALWAYS includes `branch: <trimmed value>` — empty = the tri-state's clear leg, so clearing a branch is a deliberate empty save, unlike the token's blank = keep). Every STORED git row with a branch shows a `branch: <name>` text tag in the source cell (the "hidden on" tag idiom — text + background, WCAG: the text IS the signal, never color alone).
|
||||
|
||||
## Tasks
|
||||
1. `01_branch_column.md` — the alembic 0024 migration, the `GitSource.branch` column, the schema fields, the migration round-trip test.
|
||||
2. `02_branch_api.md` — `validate_branch_name` + POST/PATCH/GET wiring (the tri-state, the local-row 422, the env rows).
|
||||
3. `03_clone_pull_branch.md` — `clone_or_pull(…, branch)` (the `--branch` clone, the HEAD guard, the fetch+ff sequence) + both call sites.
|
||||
4. `04_branch_ui.md` — the add-form field, the per-row editor field, the row tag.
|
||||
5. `05_e2e_git_source_branch.md` — the dedicated Playwright E2E (the two-branch `file://` fixture, the UI add-with-branch → sync → branch content, the fail-loud mismatch, the API contracts).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `validate_branch_name` (the full accept/reject matrix); `clone_or_pull`'s argv sequences with mocked `subprocess` (the `tests/unit/test_git_sync.py` pattern — fresh-clone `--branch`, the existing-checkout guard → probe → fetch → merge, the mismatch raise, `branch=None` byte-identical).
|
||||
- Integration: the migration round-trip (0024); the API round-trips (create/patch/list, the tri-state, the 422s); a real `file://` two-branch clone through `clone_or_pull` (the `tests/integration/test_import_docs_git.py` pattern) + the sync pipeline importing the branch's content.
|
||||
- E2E: one dedicated Playwright file, run in isolation.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a git source added with a branch mirrors THAT branch (fresh clone + subsequent pulls) through both the Sync button and the CLI — pinned by the E2E + the integration test
|
||||
- [ ] a row with no branch (all existing rows + env rows) behaves byte-identically to pre-phase (the `branch=None` argv pin is green)
|
||||
- [ ] invalid branch names are rejected with the fixed 422 (never echoing input); a branch on a local row is a 422; the PATCH tri-state (set/change/clear) round-trips
|
||||
- [ ] setting a branch on an existing checkout on a different branch fails the sync LOUDLY with the remove-and-re-add message (no silent switch)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_git_source_branch.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the token/ignore/hidden round-trips, the sync pipeline, the CLI — all existing tests green untouched)
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 01 — The `git_sources.branch` column: migration 0024, the model field, the schema fields
|
||||
|
||||
**Phase:** `135_git_source_branch` · **Source:** `TODO.md:4` — "Need to be able to specify the branch for a git source"
|
||||
|
||||
## Objective
|
||||
The registry can store a per-source branch: a reversible alembic 0024 adds the nullable `branch` column, the ORM model carries it, and the four API schema shapes report/accept it — with NULL meaning "the repo's default branch" so every existing row is unchanged.
|
||||
|
||||
## Work
|
||||
1. `alembic/versions/0024_git_source_branch.py` (new) — follow the shape of `0021_git_source_token.py` (read it first):
|
||||
- `revision = "0024"`, `down_revision = "0023"` (verify the actual latest revision id in `alembic/versions/` at execution time — the file numbering is the convention, the `down_revision` chain is the truth).
|
||||
- `upgrade`: `op.add_column("git_sources", sa.Column("branch", sa.Text(), nullable=True))`.
|
||||
- `downgrade`: `op.drop_column("git_sources", "branch")` (reversible — the house rule; NULL = default branch means no data migration in either direction).
|
||||
2. `app/models.py::GitSource` — add:
|
||||
```python
|
||||
#: The branch to mirror (phase 135): NULL = the repo's default branch
|
||||
#: (every pre-phase-135 row and every env-fallback row — byte-identical
|
||||
#: clone/pull behavior). A non-NULL value is validated at the API
|
||||
#: boundary (app.rag.git_sources.validate_branch_name) — the column
|
||||
#: itself is plain Text.
|
||||
branch: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
```
|
||||
3. `app/schemas.py` (the four shapes — each docstring gains its phase-135 paragraph):
|
||||
- `GitSourceIn.branch: str | None = Field(default=None, max_length=200)` — optional at create (absent → NULL); a `field_validator` trims before the length constraint runs (the `_trim_url` precedent) — note the API layer treats a trimmed-EMPTY value as absent at create and as the tri-state's clear leg at PATCH (document that split in the docstring).
|
||||
- `GitSourceOut.branch: str | None` — the stored value (NULL → null in the response); the shape keeps `extra="forbid"` (no new omission risk — the field is explicit).
|
||||
- `GitSourceRow.branch: str | None` — DB rows report the stored value; env-fallback synthetic rows report `None` (the `ignore_paths: []` / `include_hidden: False` precedent — no DB row to store one on).
|
||||
- `GitSourcePatchIn.branch: str | None = None` — the tri-state (absent/None = no change; `""` = clear; non-empty = set — the token's precedent, documented in the docstring).
|
||||
4. `uv run alembic upgrade head` against the dev DB (and `downgrade -1` + `upgrade head` once — the reversibility proof).
|
||||
5. `tests/integration/test_migration_0024.py` (new — the `test_migration_0021.py` pattern): the upgrade adds the nullable column to a table WITH pre-existing rows (all report NULL — the byte-identical guarantee), a row can take a branch value, the downgrade drops the column.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the migration round-trip (the suite above).
|
||||
- Coverage: **>90%** on new/modified code (the model/schema surface is exercised by task 02's API tests — this task's own gate is the migration test + the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run alembic upgrade head && uv run alembic downgrade -1 && uv run alembic upgrade head` succeeds on a scratch DB (the reversibility proof)
|
||||
- [ ] pre-existing rows read `branch: None` (the integration test asserts on seeded rows — no backfill, no default)
|
||||
- [ ] `uv run pytest tests/integration/test_migration_0024.py -v` green
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 02 — API: branch validation, POST/PATCH/GET wiring (the tri-state, the local-row 422, the env rows)
|
||||
|
||||
**Phase:** `135_git_source_branch` · **Source:** `TODO.md:4` — "Need to be able to specify the branch for a git source"
|
||||
|
||||
## Objective
|
||||
The admin API accepts, validates, stores, and reports a branch per git source: a pure `validate_branch_name` helper is the single choke point, POST stores it (git rows only), PATCH edits it with the token's tri-state, and GET/POST-201/PATCH-200 report it — invalid names get a fixed-detail 422 that never echoes the input.
|
||||
|
||||
## Work
|
||||
1. `app/rag/git_sources.py` — the pure validator (next to `normalize_credential`; no DB, no logging of the raw value beyond the fixed reason):
|
||||
```python
|
||||
_BRANCH_RE = re.compile(r"^[A-Za-z0-9._/-]+$")
|
||||
|
||||
def validate_branch_name(raw: str) -> str:
|
||||
"""Trim + validate a branch name (phase 135, ASSUMPTION B2).
|
||||
|
||||
Returns the trimmed name; raises ValueError (fixed reason strings,
|
||||
for logs only — the API maps to the fixed-detail 422) when the
|
||||
name is empty, > 200 chars, outside the safe charset
|
||||
([A-Za-z0-9._/-]+), contains ".." or "//", starts with "-"/"."/"",
|
||||
ends with "/" or ".", or is HEAD (case-insensitive).
|
||||
"""
|
||||
```
|
||||
(The leading-slash check is `startswith("/")`; write the helper with the checks in this order: trim → empty → length → charset → `..`/`//` → leading → trailing → HEAD.)
|
||||
2. `app/api/git_sources.py`:
|
||||
- `_create_git_row`: after the URL contract (and the token normalization — order: URL shape → credential normalization → duplicate check → branch): `branch = (payload.branch or "").strip()`; if `branch`: `validate_branch_name` → store; else store `None`. A `ValueError` → `HTTPException(422, detail="not a valid branch name")` (FIXED detail — the input is never echoed).
|
||||
- `_create_local_row`: `payload.branch` present and non-empty after strip → `HTTPException(422, detail="branch is only valid for git sources")` (local rows are walked, not cloned — the fixed detail, the kind-field rules' precedent).
|
||||
- `patch_git_source`: a new independent present-wins field — `if payload.branch is not None:` the row must be `kind == "git"` (local → the same fixed 422); `""` → `row.branch = None` (the tri-state's clear leg); non-empty → `validate_branch_name` → set. No Integrity interaction (branch never touches a unique index) — no 409 path. Update the docstring's field list (ignore_paths / include_hidden / token / branch).
|
||||
- `list_git_sources`: DB rows report `branch=row.branch`; env-fallback rows report `branch=None`. Update the docstring.
|
||||
- `create_git_source` / `patch_git_source` responses: include `branch=row.branch` in the `GitSourceOut` construction.
|
||||
- The module docstring's route summary gains the branch clauses (the house one-paragraph-per-route contract).
|
||||
3. Unit tests — `tests/unit/test_git_sources_branch.py` (new; the validator has no existing home):
|
||||
- Accept: `main`, `release/1.2`, `feature_x`, `a.b/c`, `v2-rc.1` (trim applied: `" main "` → `main`).
|
||||
- Reject (each → `ValueError`): `""`, `" "`, 201 chars, `a b`, `a@b`, `a~b`, `a:b`, `a?b`, `a*b`, `a[b`, `a\b`, `a..b`, `a//b`, `-x`, `.x`, `/x`, `x/`, `x.`, `HEAD`, `head`, `HeAd`.
|
||||
4. Integration tests — `tests/integration/test_git_sources_api.py` (extend — the file already owns this router's contract):
|
||||
- POST a git source with `branch: "release"` → 201 reports `branch: "release"`; GET reports it.
|
||||
- POST with an invalid branch (`"a..b"`) → 422 `not a valid branch name` (the detail is EXACT — no input echo); the row is not created.
|
||||
- POST a LOCAL source with a branch → 422 `branch is only valid for git sources`.
|
||||
- PATCH the tri-state: set (`"hotfix/x"`) → 200 reports it; change (`"hotfix/y"`); clear (`""`) → 200 reports `null`; absent (PATCH `{"include_hidden": true}` only) → the branch is untouched.
|
||||
- PATCH a local row's branch → 422.
|
||||
- Env-fallback rows (empty table + `BOR_GIT_SOURCES` env): GET reports `branch: null` for every synthetic row.
|
||||
- The existing token/ignore/hidden round-trips stay green (no regression).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the validator matrix (step 3).
|
||||
- Integration: the API round-trips (step 4).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_git_sources_branch.py tests/integration/test_git_sources_api.py -v` green
|
||||
- [ ] every 422 detail is a fixed string (a grep of the new code shows no f-string echoing the branch input)
|
||||
- [ ] the tri-state matches the token's semantics field-for-field (the PATCH docstring documents both)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 03 — `clone_or_pull(…, branch)` and both call sites (sync pipeline + CLI)
|
||||
|
||||
**Phase:** `135_git_source_branch` · **Source:** `TODO.md:4` — "Need to be able to specify the branch for a git source"
|
||||
|
||||
## Objective
|
||||
The git plumbing honors a row's branch: a fresh clone checks it out (`--branch`), an existing checkout is tracked (fetch + ff-merge of that ref) with a fail-loud guard when the checkout is on a different branch, and `branch=None` keeps the pre-phase argv byte-identical — wired through BOTH entry points (the Sync button and the `import_docs` CLI).
|
||||
|
||||
## Work
|
||||
1. `scripts/git_sync.py::clone_or_pull` — the new optional parameter (the module docstring's clone/pull paragraph + the function docstring gain the branch contract):
|
||||
```python
|
||||
def clone_or_pull(url: str, dest: Path | str, branch: str | None = None) -> Path:
|
||||
```
|
||||
- **`branch is None`** — the EXISTING code path, byte-identical: clone without `--branch`; `rev-parse --is-shallow-repository` probe; `fetch --unshallow` while shallow; `git pull --ff-only`. (The argv pin in the unit tests is the proof.)
|
||||
- **Fresh clone + branch** (`dest` absent or no `.git`): `git clone --branch <b> url dest`.
|
||||
- **Existing checkout + branch**, in order:
|
||||
1. HEAD GUARD: `git rev-parse --abbrev-ref HEAD` → let `current` be the result. If `current != branch` → `GitSyncError(f"checkout {dest} is on branch '{current}' but the source specifies branch '{branch}' — remove and re-add the source to switch branches")` (ASSUMPTION B5 — fail loud per D12; the message names the checkout and both branch names — no URL, so no credential exposure).
|
||||
2. the shallow probe + one-time `git fetch --unshallow` self-heal (UNCHANGED from today — the phase-107 D11 contract).
|
||||
3. `git fetch origin <b>` → `git merge --ff-only FETCH_HEAD` (the single-fetch ff pull of the specified ref — the sequence phase 136 extends with its reset fallback; a non-ff merge still fails loud HERE — the force-push fix is phase 136, not this one).
|
||||
- All commands go through `run_git` (the A11 single invocation site — unchanged).
|
||||
2. The call sites — pass `row.branch`:
|
||||
- `app/api/sync.py::_run_sync` (the per-source loop, the `kind == "git"` arm): `root = clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url), row.branch)`.
|
||||
- `scripts/import_docs.py::_resolve_sources` (the same shape): `root = clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url), row.branch)`.
|
||||
- Both files' docstrings/comments name the branch argument (the phase-106/121 comment style). Env-fallback synthetic rows carry `branch=None` (the `GitSource(url=…, kind="git")` constructor — no argument needed, the model default is None — verified by the env-fallback integration path).
|
||||
- `file_commit_dates(root)` is untouched (it walks whatever the checkout holds — for a branch checkout that is the branch's tree, exactly the head-of-branch semantics the owner asked for).
|
||||
3. Unit tests — `tests/unit/test_git_sync.py` (extend — the mocked-`subprocess` argv pattern the file already uses):
|
||||
- `branch=None` fresh + existing: the argv sequence is byte-identical to the pre-phase expectations (the existing tests stay green + add an explicit argv-equality pin if the file doesn't already pin it).
|
||||
- Fresh clone + `branch="release"`: `["git", "clone", "--branch", "release", url, dest]`.
|
||||
- Existing + matching HEAD (`rev-parse --abbrev-ref HEAD` → `"release"`): the sequence `rev-parse --abbrev-ref HEAD` → `rev-parse --is-shallow-repository` → (`fetch --unshallow` when the probe says shallow) → `fetch origin release` → `merge --ff-only FETCH_HEAD`.
|
||||
- Existing + mismatched HEAD (`"main"` vs `branch="release"`): raises `GitSyncError` whose message contains both branch names and the checkout path — and NO fetch/merge argv was invoked (the guard is first).
|
||||
- The mismatch message never contains the URL (credential-safety — even for a plain URL the message is URL-free).
|
||||
4. Integration tests — `tests/integration/test_import_docs_git.py` (extend — the real-`git`-`file://` pattern the file already uses):
|
||||
- Build a two-branch fixture repo (real git: `main` with file A, `release` with file B — different content per branch).
|
||||
- `clone_or_pull(url, dest, "release")` fresh → the checkout's HEAD branch is `release` and file B is present (file A absent).
|
||||
- A second `clone_or_pull` after the origin gains a commit on `release` → ff-merged (file B updated, no error).
|
||||
- The mismatch case: clone fresh with no branch (HEAD = `main`), then `clone_or_pull(url, dest, "release")` → `GitSyncError` (the remove-and-re-add message), the checkout untouched (still on `main`).
|
||||
- The full sync pipeline with a branch row: a `git_sources` row (`kind="git"`, `branch="release"`) → `_run_sync`'s source resolution imports file B (the `tests/integration/test_sync_api.py` fixture shape) — or at minimum `_resolve_sources`-level for the CLI path (follow whichever seam the file already exercises; the E2E, task 05, pins the button path end to end).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the argv matrix (step 3) — the `branch=None` byte-identical pin is the regression gate for every existing deployment.
|
||||
- Integration: the real-git two-branch scenarios (step 4).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_import_docs_git.py -v` green
|
||||
- [ ] `branch=None` produces the exact pre-phase argv (the pin is in the suite — a future refactor that rewrites the default path breaks it)
|
||||
- [ ] the mismatch fails loud BEFORE any fetch (the unit test asserts no fetch argv after the guard)
|
||||
- [ ] both entry points pass `row.branch` (a grep of the two call sites + the integration test covering each)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 04 — UI: the add-form field, the per-row editor field, and the row tag
|
||||
|
||||
**Phase:** `135_git_source_branch` · **Source:** `TODO.md:4` — "Need to be able to specify the branch for a git source"
|
||||
|
||||
## Objective
|
||||
The Sources manager (git-sources view) lets the owner see, set, change, and clear a source's branch: an optional Branch input in the add form, a Branch field in the per-row editor dialog (git rows only, prefilled, empty-save = clear), and a `branch: <name>` text tag on every stored git row that has one — all on the §7.4 never-stale lifecycle.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` (the `#view-git-sources` section):
|
||||
- **The add form** (`#git-source-form`) — between the URL field and the Token field:
|
||||
```html
|
||||
<label for="git-source-branch">Branch <span class="field-hint">optional — default: the repo's default branch</span></label>
|
||||
<input id="git-source-branch" name="branch" type="text" maxlength="200" autocomplete="off" placeholder="main, release/1.2, …">
|
||||
```
|
||||
(The visible `<label for=…>` is the WCAG input-label rule, same as the url/token fields.)
|
||||
- **The per-row editor dialog** (`#ignore-editor-dialog`) — after the source line, before the ignore textarea:
|
||||
```html
|
||||
<label for="ignore-editor-branch">Branch</label>
|
||||
<input id="ignore-editor-branch" type="text" maxlength="200" autocomplete="off" placeholder="(default branch)">
|
||||
<p class="ignore-editor-branch-hint">Leave empty for the repo's default branch.</p>
|
||||
```
|
||||
The hint uses the existing `.field-hint` / muted-copy idiom (match the token field's hint styling — no new CSS class unless the existing one doesn't fit).
|
||||
2. `frontend/assets/git-sources.js`:
|
||||
- **Element lookups:** `const branchInput = root.querySelector("#git-source-branch");` and `const ignoreBranchInput = root.querySelector("#ignore-editor-branch");` (next to the existing token lookups).
|
||||
- **Add form:** thread the branch through `wireAddForm` — the opts object gains `branchInput`, and the git-kind body builder becomes `body: (url, token, branch) => ({ url, ...(branch ? { branch } : {}), ...(token ? { token } : {}) })` (a blank branch OMITS the key — the token's blank-omits precedent; the submit handler reads `branchInput.value.trim()` and clears it on 201 alongside the url/token — the credential-free inputs clear together).
|
||||
- **Per-row editor dialog:**
|
||||
- Open: for a git row (`s.kind === "git"`) — prefill `ignoreBranchInput.value = s.branch ?? ""` and show the field; for a local row — hide the field (label + input + hint) and the save body OMITS `branch` (local rows can't take one — the API's 422 is the backstop, the UI simply doesn't offer it).
|
||||
- Save: the PATCH body for a git row ALWAYS includes `branch: ignoreBranchInput.value.trim()` (empty = the tri-state's clear leg — a deliberate empty save clears the branch, unlike the token's blank = keep; the docstring/section comment documents the asymmetry), composed with the existing `ignore_paths` + conditional `token` (the token stays the blank = keep tri-state — untouched).
|
||||
- The never-stale lifecycle mirrors the existing save handler: the Save button disables in flight, 200 → close the dialog + `loadSources()` (the row re-renders from the server — the tag appears/disappears), failure → the dialog's error line (the existing pattern) + the inputs keep the user's text.
|
||||
- **The row tag** (`makeRow` / the source-cell builder): when `s.id && s.branch` — append a text tag `branch: <name>` in the source cell, next to the existing "hidden on" tag (the SAME tag idiom: a span with the tag class — text + background, never color alone; the tag text is the signal). Env-fallback rows (no `s.id`) and NULL-branch rows show no tag. Update the source cell's `aria-label`/`title` to include the branch (the hover precision idiom).
|
||||
- The module's section-header docstring gains the phase-135 paragraph (the house pattern — the field, the tri-state asymmetry, the tag rule).
|
||||
3. No new API surface, no new fetches — the branch rides the existing POST/PATCH/GET payloads (task 02's shapes).
|
||||
|
||||
## Testing & Quality
|
||||
- No JS unit layer in this repo — the UI contract is pinned by the phase's E2E (task 05).
|
||||
- Coverage: **>90%** on new/modified code (frontend-only task — the bar is carried by the suite run + the E2E).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] adding a git source with a branch sends `branch` (non-blank) / omits it (blank) — pinned by the E2E's request assertions
|
||||
- [ ] the dialog prefills the stored branch; saving an empty value clears it (the row's tag disappears after `loadSources`); local rows' dialogs never show the field
|
||||
- [ ] a stored git row with a branch shows the `branch: <name>` text tag (visible, text-based, in the source cell)
|
||||
- [ ] the 360 px case renders (the form + dialog don't overflow — verified in the E2E's viewport assertions where applicable)
|
||||
- [ ] no console errors on the view (the E2E pins it)
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 05 — E2E: `tests/e2e/test_git_source_branch.py` (the dedicated Playwright suite, run in isolation)
|
||||
|
||||
**Phase:** `135_git_source_branch` · **Source:** `TODO.md:4` — "Need to be able to specify the branch for a git source"
|
||||
|
||||
## Objective
|
||||
The branch feature is pinned end to end against the REAL pipeline: a two-branch `file://` fixture repo, the owner adding a source WITH a branch through the UI (→ sync → the branch's content is indexed, not the default branch's), the row tag, the per-row editor's branch field (change + the fail-loud mismatch), and the API contracts (the tri-state, the fixed 422s, the local-row rejection).
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_git_source_branch.py` (new) — the per-module app pattern (`test_git_sources_admin.py` + `test_sync_button.py` combined):
|
||||
- **Ports/env:** `APP_PORT = int(os.environ.get("E2E_APP_PORT_GITBR", "8134"))`; the standard module env (`BOR_STATIC_DIR`, the mock LLM base URL, `BOR_RELEVANCE_THRESHOLD=0.30`, `BOR_ADMIN_PASSWORD` / `BOR_SESSION_SECRET`, a dedicated `BOR_SOURCES_DIR`). **No `BOR_GIT_SOURCES` env** — the sources under test are DB rows (the UI adds them), so the env fallback stays empty.
|
||||
- **The fixture repo** (module-scoped, `tmp_path_factory`, real `git`): two branches — `main` (file `notes/main-only.md` with `Marker: RESE-BRANCH-MAIN-7a31`) and `release` (file `notes/release-only.md` with `Marker: RESE-BRANCH-REL-9c2d`, plus a shared `notes/common.md` whose content DIFFERS between the branches — `main` says `VERSION: 1`, `release` says `VERSION: 2`). Both branches pushed to the repo (a local bare-ish fixture: the test builds `main`, commits, then `git checkout -b release`, edits, commits — a normal two-branch repo; the app clones it via `file://`).
|
||||
- **Helpers:** `add_git_source(page, url, branch=None, token=None)` (fills the UI form — the branch field when given), `sync_and_wait(page, app_url)` (clicks `#sync-btn` and polls the status endpoint to a terminal state — the `_wait_sync_done` pattern), `catalog_has_doc(page, doc_name)` (the drill-down tree assertion, the `_expect_fixture_row` pattern).
|
||||
2. Tests:
|
||||
- `test_add_with_branch_syncs_that_branch` — the owner's exact flow:
|
||||
1. login (lands on the git-sources view — the `login` helper's `next` param or a nav click, per the existing suites);
|
||||
2. add a source: url = `file://<fixture>`, branch = `release` → 201; the row appears with the `branch: release` tag (the text is in the source cell);
|
||||
3. switch to the Sources/RAG view, `sync_and_wait` → `state == "success"`;
|
||||
4. the catalog contains `release-only.md` (the branch's doc) AND `common.md` shows `VERSION: 2` (drill in and assert on the document's stored content via the document modal or the `GET /api/docs` row — follow the assertion idiom the dates suite uses for content checks); `main-only.md` is ABSENT (the default branch's doc was never indexed);
|
||||
5. re-sync (idempotent): "0 added · 2 unchanged" (release's two files).
|
||||
- `test_default_branch_source_unaffected` — a SECOND source row for the SAME url is impossible (the unique URL 409) — so this test uses a CLONE of the fixture (a second `file://` repo, identical history, `main` only): add it with NO branch → sync → `main-only.md` present, `release-only.md` absent (the `branch=None` path mirrors the pre-phase behavior — the byte-identical guarantee, observable through the UI).
|
||||
- `test_row_editor_branch_change_fails_loud` — the fail-loud mismatch (ASSUMPTION B5, end to end):
|
||||
1. take the `release` source from test 1's state (or a fresh module state — order the tests so the fixture state is explicit; the KB-truncate autouse fixture keeps them independent: re-add the source fresh);
|
||||
2. open the per-row editor, change the branch field to `main`, save → the dialog closes, the row re-renders with the `branch: main` tag (the PATCH landed — the registry is updated);
|
||||
3. sync → `state == "failed"`; the sanitized error banner/modal shows the fixed mismatch copy (assert on the user-visible fragment `remove and re-add the source` — the message is a fixed string, safe to pin);
|
||||
4. restore: editor → branch field `release` → save → sync → success again (the guard is symmetric — back to the matching branch, the ff path works).
|
||||
- `test_api_branch_contracts` — the API layer through `page.request` (admin cookie):
|
||||
- POST a git source with `branch: "a..b"` → 422, detail EXACTLY `not a valid branch name` (no echo); the list is unchanged;
|
||||
- POST a git source with `branch: "HEAD"` → 422 (same detail);
|
||||
- PATCH the tri-state on a stored row: set `hotfix/x` → 200 reports it; clear with `""` → 200 reports `null`; a PATCH of `{"include_hidden": true}` leaves the branch untouched;
|
||||
- POST a LOCAL source (an existing tmp dir) with `branch: "x"` → 422 `branch is only valid for git sources`;
|
||||
- the token field is unaffected by all of the above (the existing token tests stay green — no regression pin needed here).
|
||||
- Wall-time guard: the whole module < 180 s (three full syncs of a 3-file repo against the mock LLM — the `slow_llm` proxy is NOT needed: the branch assertions are about WHICH files land, not about a running window).
|
||||
3. Header docstring: the story (the TODO item, L4), the run command (`uv run pytest tests/e2e/test_git_source_branch.py -v --no-cov`), the isolation note (per-module app, dedicated port, DB rows not env), the fixture's two-branch shape.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite; it runs in ISOLATION (`--no-cov`, the AGENTS.md rule 9 gate) with the DB up (`podman compose up -d db`) and git on PATH.
|
||||
- Coverage: **>90%** on new/modified code (test-only task — the bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_git_source_branch.py -v --no-cov` green in isolation
|
||||
- [ ] the branch's content is what gets indexed (the sentinel docs on the right branch present, the wrong branch's doc absent — the feature's core promise)
|
||||
- [ ] the `branch=None` path imports the default branch exactly as pre-phase (the unaffected-source test)
|
||||
- [ ] the mismatch is LOUD (the failed sync names the remove-and-re-add fix — the owner is never left with a silently-wrong checkout)
|
||||
- [ ] every 422 detail is the fixed string (the test asserts on the exact detail, not a substring of the input)
|
||||
@@ -0,0 +1,39 @@
|
||||
# Phase 136 — A force push no longer breaks the sync: the checkout resets to the head of the tracked branch
|
||||
|
||||
**Source:** `TODO.md` L5 — "Force pushes break brain of reese git pull. Brain of reese should always have the head of the branch specified, so if pull fails due to force pushes it should just reset origin/<branch>"
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `scripts/git_sync.py::clone_or_pull` — today's existing-checkout path: the shallow probe + one-time `git fetch --unshallow` self-heal (phase 107 D11), then `git pull --ff-only` — a force push is a NON-fast-forward, so `--ff-only` refuses → `GitSyncError` → the whole sync lands `failed` (the owner's reported break; this is the bug). Phase 135 (the immediately preceding todo phase) lands the `branch` parameter: for a branched row the sequence is already `git fetch origin <b>` + `git merge --ff-only FETCH_HEAD` behind the HEAD guard; for a `NULL`-branch row it is still the legacy `git pull --ff-only`. `run_git` is the A11 single git invocation site; the two call sites (`app/api/sync.py::_run_sync`, `scripts/import_docs.py::_resolve_sources`) are untouched in shape — `clone_or_pull`'s signature is theirs to satisfy; `file_commit_dates` walks the post-update checkout (a reset checkout yields the NEW head's per-file dates — exactly the head-of-branch semantics the owner asked for). The sync's failure path (`_run_sync`'s `except Exception` → the `failed` state + the sanitized error) is what a force push hits today — after this phase it no longer does.
|
||||
|
||||
## Objective
|
||||
A force push (or any divergent remote head) never fails the sync: the existing-checkout update tries the fast-forward and, when that is impossible, **resets the checkout to the remote head of the tracked ref** — for a branched row `origin/<branch>` (the owner's verbatim "reset origin/<branch>"), for a `NULL`-branch row the head of the branch the checkout is on — with one loud log line. The checkout is a cache of the remote head (the app owns it, the KB is rebuilt from it on the same run), so "always the head of the branch" is the invariant this phase enforces.
|
||||
|
||||
## Dependencies
|
||||
- `135_git_source_branch` (todo) — the `branch` parameter of `clone_or_pull` (the reset's target ref for branched rows) and the phase-135 HEAD guard, which stays in front of the update sequence.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The existing-checkout update sequence** (replaces BOTH today's `git pull --ff-only` path and phase 135's branch path — one unified sequence):
|
||||
1. ref resolution: `branch` given → `ref = branch` (the HEAD guard from phase 135 runs FIRST, unchanged: checkout on a different branch → the remove-and-re-add `GitSyncError`); `branch` None → `ref = git rev-parse --abbrev-ref HEAD` (the checkout's own branch — the pre-phase default-branch behavior; if the result is `"HEAD"` (detached) → `GitSyncError` naming the checkout: the app never CREATES a detached checkout — a fresh clone always lands on a branch — so this is a genuinely broken checkout and fails loud per D12, not a force-push case).
|
||||
2. the shallow probe + one-time `git fetch --unshallow` self-heal (UNCHANGED — phase 107 D11).
|
||||
3. `git fetch origin <ref>` — a fetch FAILURE (network/auth, the branch gone from the remote, git missing) STILL raises `GitSyncError` (a loud failure — a force push is NOT a fetch failure; the owner's "if pull fails due to force pushes" scopes the reset to the merge step only).
|
||||
4. try `git merge --ff-only FETCH_HEAD`.
|
||||
5. **on merge failure — the reset fallback (this phase, ASSUMPTION C1):** capture `old = git rev-parse HEAD` and `new = git rev-parse FETCH_HEAD`, run `git reset --hard FETCH_HEAD`, log one WARNING line (PLAN §9): `git sync: divergent remote head on <dest> (force push) — reset <old[:7]> → <new[:7]> to origin/<ref>`, and RETURN the dest normally (the sync proceeds — a force push never fails a run). The merge's stderr is included in the log (diagnosability) but never in a user-facing error (there is no error — the run continues).
|
||||
- **Unchanged:** the fresh-clone path (with and without `--branch`); the `run_git` discipline; the two call sites; `file_commit_dates`; the sync's status/progress surface. The `branch=None` argv CHANGES by design (the legacy `git pull --ff-only` becomes the explicit fetch + ff-or-reset sequence — that IS the fix, and it applies to every row, branched or not).
|
||||
- **ASSUMPTION C2 — the dirty checkout:** the checkout is app-managed (a cache of the remote head; the KB is rebuilt from it by the same run's import), so local modifications are not expected and are not defended against: the `reset --hard` discards any by definition. No `git status` precheck (one fewer git call per source per sync; the house minimal-invocation discipline) — a dirty checkout simply resets, which is the desired end state either way.
|
||||
|
||||
## Tasks
|
||||
1. `01_ff_or_reset.md` — the unified update sequence in `clone_or_pull` (ff-or-reset, the detached-HEAD fail-loud, the loud log) + the unit/integration matrix.
|
||||
2. `02_e2e_force_push_resync.md` — the dedicated Playwright E2E (a real force push of the fixture repo between two UI sync runs — the pre-phase behavior, a failed sync, is the regression pin).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the mocked-`subprocess` argv matrix in `tests/unit/test_git_sync.py` (ff success — no reset argv; ff failure — the rev-parse pair + `reset --hard FETCH_HEAD` + return, no raise; fetch failure — still raises; detached `NULL`-branch — raises; the phase-135 mismatch guard still fires before the update sequence).
|
||||
- Integration: real-`git` `file://` scenarios in `tests/integration/test_import_docs_git.py` (normal ff; a force push — the origin's history rewritten; an UNRELATED-history force push — different root commit; a branched row's force push — the reset targets `origin/<branch>`; `file_commit_dates` on the post-reset checkout reflects the new head).
|
||||
- E2E: one dedicated Playwright file, run in isolation (the owner's exact reproduction: sync → force push the origin → sync again → success + the new head's content).
|
||||
- Coverage: **>90%** on new/modified code (`app/` — the change lives in `scripts/`, exercised through the `app/` sync pipeline tests; the coverage bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a force push of a source's branch no longer fails the sync — the next sync succeeds and the checkout (and therefore the KB) holds the NEW head (pinned by the integration + E2E)
|
||||
- [ ] an unrelated-history force push (a rewritten root) is handled by the same reset (the E2E or integration covers it — pick the integration, the E2E stays on the owner's exact repro)
|
||||
- [ ] a genuinely broken fetch (git missing / unreachable origin / branch deleted upstream) STILL fails loud (the `GitSyncError` pins are green — the reset is scoped to the merge step only)
|
||||
- [ ] the WARNING log line lands on a reset (one line, the SHAs, the checkout path, `origin/<ref>` — no URL in it)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` green in isolation
|
||||
- [ ] the phase-135 behavior is preserved (the `--branch` clone, the mismatch guard, the `branch=None` fresh-clone argv — the pins from phase 135 stay green)
|
||||
@@ -0,0 +1,71 @@
|
||||
# Task 01 — `clone_or_pull`: ff-or-reset (the force push no longer fails the sync)
|
||||
|
||||
**Phase:** `136_force_push_reset` · **Source:** `TODO.md:5` — "Force pushes break brain of reese git pull. Brain of reese should always have the head of the branch specified, so if pull fails due to force pushes it should just reset origin/<branch>"
|
||||
|
||||
## Objective
|
||||
The existing-checkout update becomes: fetch the tracked ref → try the fast-forward → on a non-ff failure, reset the checkout hard to that ref's head and continue — with one loud WARNING line. A force push is now a logged, self-healing event, never a failed sync; genuinely broken fetches still fail loud.
|
||||
|
||||
## Work
|
||||
1. `scripts/git_sync.py::clone_or_pull` — replace the existing-checkout tail (today: probe → unshallow-if-shallow → `git pull --ff-only`; phase 135's branch tail: HEAD guard → probe → unshallow-if-shallow → `fetch origin <b>` → `merge --ff-only FETCH_HEAD`) with the unified sequence from the phase overview:
|
||||
```python
|
||||
# existing checkout (the fresh-clone path above is untouched):
|
||||
if branch is not None:
|
||||
# phase-135 HEAD guard, UNCHANGED (runs first, before any update):
|
||||
current = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip()
|
||||
if current != branch:
|
||||
raise GitSyncError(
|
||||
f"checkout {dest} is on branch '{current}' but the source specifies "
|
||||
f"branch '{branch}' — remove and re-add the source to switch branches"
|
||||
)
|
||||
ref = branch
|
||||
else:
|
||||
ref = run_git(["git", "rev-parse", "--abbrev-ref", "HEAD"], cwd=dest).strip()
|
||||
if ref == "HEAD": # detached — the app never creates one (D12 fail-loud)
|
||||
raise GitSyncError(
|
||||
f"checkout {dest} is in a detached-HEAD state — remove and re-add the "
|
||||
f"source to rebuild it"
|
||||
)
|
||||
shallow = run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest).strip() == "true"
|
||||
if shallow:
|
||||
run_git(["git", "fetch", "--unshallow"], cwd=dest)
|
||||
run_git(["git", "fetch", "origin", ref], cwd=dest) # fetch failure STILL raises
|
||||
try:
|
||||
run_git(["git", "merge", "--ff-only", "FETCH_HEAD"], cwd=dest)
|
||||
except GitSyncError as ff:
|
||||
old = run_git(["git", "rev-parse", "HEAD"], cwd=dest).strip()
|
||||
new = run_git(["git", "rev-parse", "FETCH_HEAD"], cwd=dest).strip()
|
||||
run_git(["git", "reset", "--hard", "FETCH_HEAD"], cwd=dest)
|
||||
logger.warning(
|
||||
"git sync: divergent remote head on %s (force push) — reset %s → %s to "
|
||||
"origin/%s (%s)", dest, old[:7], new[:7], ref, ff,
|
||||
)
|
||||
return dest
|
||||
```
|
||||
Notes: the `branch=None` path deliberately LOSES the legacy `git pull --ff-only` (that is the fix — every row gets the ff-or-reset update); the `dest` in the log is the checkout path (NEVER the URL — credential-safety, the phase-121 discipline); the merge stderr rides the log only (the run continues — there is no user-facing error). Update the module + function docstrings (the D11/D12 paragraphs gain the phase-136 force-push contract: "a non-ff merge resets to the ref's head instead of failing — the checkout is a cache of the remote head").
|
||||
2. Unit tests — `tests/unit/test_git_sync.py` (extend — the mocked-`subprocess` argv pattern; the mock needs per-argv responses: `rev-parse --abbrev-ref HEAD` → the branch, `rev-parse --is-shallow-repository` → `false`, the ff merge → non-zero on the failure case, the `rev-parse HEAD` / `FETCH_HEAD` → SHAs, the reset → zero):
|
||||
- `branch=None`, ff SUCCESS: argv sequence `rev-parse --abbrev-ref HEAD` → `rev-parse --is-shallow-repository` → `fetch origin <branch-from-HEAD>` → `merge --ff-only FETCH_HEAD` — and NO `reset` argv, NO `rev-parse HEAD/FETCH_HEAD` pair (the capture is only on failure).
|
||||
- `branch=None`, ff FAILURE (merge exit ≠ 0): the sequence gains `rev-parse HEAD` → `rev-parse FETCH_HEAD` → `reset --hard FETCH_HEAD` — and the function RETURNS the dest (no raise).
|
||||
- `branch="release"` (matching HEAD), ff failure: same reset tail, `fetch origin release` — the reset targets the branch's ref (the owner's verbatim "reset origin/<branch>").
|
||||
- the phase-135 mismatch (`branch="release"`, HEAD `main`): STILL raises before any fetch (the guard is first — the phase-135 pin stays green).
|
||||
- `branch=None` + detached HEAD (`rev-parse --abbrev-ref HEAD` → `"HEAD"`): raises the detached `GitSyncError` (no fetch, no reset).
|
||||
- fetch FAILURE (non-zero): raises `GitSyncError` (NOT the reset path — the reset is scoped to the merge step; a deleted upstream branch / dead network still fails loud).
|
||||
- the shallow self-heal still fires in front of the fetch (the `--unshallow` argv pin, unchanged).
|
||||
3. Integration tests — `tests/integration/test_import_docs_git.py` (extend — the real-`git` `file://` pattern):
|
||||
- **Normal ff:** a fixture repo, `clone_or_pull` ×2 with a normal second commit → ff-merged, no reset (assert via `git -C dest log` — the second commit's message is the HEAD, no merge commit).
|
||||
- **Force push (the owner's repro):** after the first clone, rewrite the origin's history (a NEW root commit — `git checkout --orphan` or `git reset --soft` + a fresh commit with different content — the origin's branch now diverges); `clone_or_pull` again → NO exception, the checkout's HEAD == the origin's new head (`git -C origin rev-parse <ref>` == `git -C dest rev-parse HEAD`), the working tree holds the NEW content.
|
||||
- **Unrelated-history force push:** the rewritten origin has a DIFFERENT root commit (a fully unrelated tree) → the same reset path handles it (the ff merge of an unrelated history is exactly what `--ff-only` refuses).
|
||||
- **Branched force push:** a two-branch fixture (the phase-135 shape), a row-shaped call `clone_or_pull(url, dest, "release")` after a force push of ONLY `release` → the reset lands `origin/release`'s head and `main`'s content is not involved.
|
||||
- **`file_commit_dates` after a reset:** the dates map reflects the NEW head (the new root commit's date is what the walk returns for its files — the head-of-branch dates the owner asked for).
|
||||
- **The sync pipeline end-to-end (the seam `tests/integration/test_sync_api.py` already exercises):** a forced-push fixture + a sync → `state == "success"` (the pre-phase code lands `failed` — this test is the regression pin at the API seam; if the file's existing fixtures make the seam awkward, the E2E, task 02, carries the API-seam pin and this step stays at the `clone_or_pull` level — document the choice in the test's docstring).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the argv matrix (step 2) — the ff-success pin (no reset argv) and the fetch-failure pin (still raises) are the two regression gates.
|
||||
- Integration: the real-git matrix (step 3).
|
||||
- Coverage: **>90%** on new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_import_docs_git.py -v` green
|
||||
- [ ] a force push (normal divergence AND unrelated history) resets instead of failing — pinned by real-git integration tests
|
||||
- [ ] the phase-135 pins (the mismatch guard, the `--branch` clone argv, the `branch=None` fresh-clone argv) stay green untouched
|
||||
- [ ] the WARNING line is the ONLY observable side effect of a reset besides the checkout state (no user-facing error, no status change — the run continues)
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,42 @@
|
||||
# Task 02 — E2E: `tests/e2e/test_force_push_resync.py` (the dedicated Playwright suite, run in isolation)
|
||||
|
||||
**Phase:** `136_force_push_reset` · **Source:** `TODO.md:5` — "Force pushes break brain of reese git pull. Brain of reese should always have the head of the branch specified, so if pull fails due to force pushes it should just reset origin/<branch>"
|
||||
|
||||
## Objective
|
||||
The owner's exact reproduction is pinned end to end: sync (clone + import) → force-push the origin (its history rewritten) → sync again → SUCCESS, and the KB holds the NEW head's content. Pre-phase, that second sync landed `failed` — this test is the regression pin for the fix.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_force_push_resync.py` (new) — the `test_sync_button.py` module-app pattern:
|
||||
- **Ports/env:** `APP_PORT = int(os.environ.get("E2E_APP_PORT_FPPUSH", "8135"))`; the standard module env (`BOR_STATIC_DIR`, the mock LLM, `BOR_RELEVANCE_THRESHOLD=0.30`, the admin/session secrets, `BOR_GIT_SOURCES=file://<fixture repo>`, a dedicated `BOR_SOURCES_DIR`).
|
||||
- **The fixture repo** (module-scoped, `tmp_path_factory`, real `git` — the `sync_git_repo` pattern): `notes/doc.md` with `Marker: RESE-FPPUSH-V1-5e8a`, one commit, one branch.
|
||||
- **The force-push helper:**
|
||||
```python
|
||||
def _force_push_new_head(repo: Path, content: str) -> None:
|
||||
"""Rewrite the origin's history to a NEW root commit (a full
|
||||
divergence — not just a new commit on top): delete the branch
|
||||
ref and recreate it at a new orphan commit with `content`."""
|
||||
```
|
||||
(Implementation at execution time: `git checkout --orphan newroot` → replace `notes/doc.md`'s content → commit → `git branch -M <branch>` / `git push --force` is not applicable to a local fixture — the test rewrites the repo IN PLACE: `git update-ref -d <branch>`, commit on the orphan, `git update-ref <branch> HEAD`. The app's next `git fetch` sees the divergent head. The exact ref mechanics are the executor's call — the OBSERVABLE contract is: the origin's single branch now points at a commit with UNRELATED history and the new content.)
|
||||
- **`_wait_sync_state`** — the `test_sync_button._wait_sync_done` pattern (terminal `{"success", "failed"}`), plus a UI-level settle wait on `#sync-label`.
|
||||
2. Tests:
|
||||
- `test_sync_survives_a_force_push` — the owner's exact flow:
|
||||
1. login (lands on `/sources.html` — the button's home);
|
||||
2. **run 1:** click `#sync-btn` → "Syncing…" → "Synced HH:MM" + "1 added"; the catalog shows `doc.md` (the drill-down assertion, the `_expect_fixture_row` pattern);
|
||||
3. **the force push** (server-side, via the fixture helper — the test owns the origin repo): `_force_push_new_head(repo, "Marker: RESE-FPPUSH-V2-2b7f …")`;
|
||||
4. **run 2:** click `#sync-btn` again → the status endpoint reaches `state == "success"` (NOT `failed` — the pre-phase regression: the `--ff-only` pull refused the divergent head and the run died) within the timeout; the UI settles to "Synced HH:MM" (the error banner is HIDDEN, no error modal);
|
||||
5. **the KB holds the NEW head:** the document modal / `GET /api/docs` content for `notes/doc.md` contains `RESE-FPPUSH-V2-2b7f` (the reset's content flowed through the import — the digest delta re-indexed the file as `updated`; `#sync-result` shows "1 updated" or "0 added · 1 updated" — assert on the content, the counts are the secondary signal);
|
||||
6. **run 3 (idempotent):** "0 added · 1 unchanged" (the checkout is stably at the new head — the reset was not a one-off fluke).
|
||||
- `test_failed_pull_is_not_the_outcome` (the negative pin, cheap while the fixture exists): assert on the APP LOG or skip if the seam is awkward — PREFERRED: a direct API-level assertion instead — after the force push, `POST /api/sync` + poll → `state == "success"` (the same contract as run 2 without the UI; keeps the UI test focused). If the module app's fixture state makes two full UI+API passes heavy, fold it into the main test as step 4b (the API trigger of run 2) — document the choice in the docstring.
|
||||
- Wall-time guard: the whole module < 120 s (three small syncs against the mock LLM — no `slow_llm` proxy needed: the point is the TERMINAL state, not a running window).
|
||||
3. Header docstring: the story (the TODO item, L5 — the owner's verbatim repro), the run command (`uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov`), the isolation note (per-module app, dedicated port), and the pre-phase behavior it pins against (run 2 used to land `failed` — the test fails on the pre-phase code, which is the point).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite; it runs in ISOLATION (`--no-cov`, the AGENTS.md rule 9 gate) with the DB up (`podman compose up -d db`) and git on PATH.
|
||||
- Coverage: **>90%** on new/modified code (test-only task — the bar is carried by the suite run).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` green in isolation
|
||||
- [ ] the test FAILS on the pre-phase code (run 2 lands `failed` — verified by temporarily reverting the `clone_or_pull` change in a scratch branch, then restoring — the pin has teeth)
|
||||
- [ ] the NEW head's content is what the KB serves (the sentinel V2 in the document — the reset's observable end-to-end effect)
|
||||
- [ ] the third run is idempotent (the checkout is stably at the new head — no drift, no second reset)
|
||||
- [ ] no error banner / modal on a force-push resync (a reset is a logged event, not a user-facing failure)
|
||||
@@ -0,0 +1,48 @@
|
||||
# Phase 137 — PDF-embedded images always carry the document context in their summary
|
||||
|
||||
**Source:** `TODO.md` L3 — "Images extracted from PDFs still don't have any relevant context in their summaries provided by the PDF. They're still just getting summarized 'as-is'. … Image summaries should always include the document context if the images come from a document (this includes markdown files) so the embedder can more effectively retrieve image sources."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/pdf.py` (phase 126 — `extract_pdf_text` (L87, per-page `get_text("text")` with `---PAGE-N---` separators, fail-soft `""`), `extract_pdf_images` (L145, one `PdfImage` per occurrence: `page` 1-indexed, `index` 0-based, `data`, `mime`)); `app/rag/importer.py` — `_index_pdf_embedded_images` (L1091, the per-PDF embedded-image pass after its document commit), `_index_one_embedded_image` (L1182, the phase-122 pipeline inline: mime gate → synthetic path `<pdf>#page=N#img=I.<ext>` → byte-digest skip → `_markdown_image_context` (L1403, the phase-125 reference scan) → `_describe_or_skip` (L1742, the single describe seam) → copy write → chunk + embed → `_store_summary`), `_markdown_image_context` (L1403) + `_md_ref_block` (L1436, the `Referenced in <source>/<md>:` block writer), `_enrich_pending_images` (L1627, the phase-125 task-04 post-index pass — the shape of the new backfill pass), `ImportSummary` (L212, stats counters + `log()` — stats are log-only, the sync `detail` API shape never changes); `app/rag/summarizer.py` — `describe_image` (L208, ONE chat-model vision call, `context` appended after the locked `DESCRIBE_PROMPT` as a `\n\nThis image is referenced in documentation with the following context:\n` prefix — the wording phase 137 generalizes), `DESCRIBE_PROMPT` / `IMAGE_DESCRIPTION_MODE` (L78/L57 — the mock's key; byte-identity criteria below); `app/rag/prompts.py` — `_extract_context_line` (L470, renders only `Referenced in …:` blocks, everything else degrades to `None`), `_image_context_line` (L536), `_document_body` (L548, the phase-125 context-line lead); `app/models.py` — `Document.image_context` (L170, TEXT nullable, migration 0023); `tests/e2e/mock_llm.py` — the `IMAGE_DESCRIPTION_MODE` canned vision reply (L2472 branch, `IMAGE_DESCRIPTION_ANSWER` L865) and the phase-125 `CONTEXT_LINE_TRIGGER` pattern (the model for this phase's mock trigger); `tests/e2e/test_pdf_support.py` (the synthetic PDF fixture recipe: PyMuPDF build, `page2.insert_image(…, stream=PNG_1X1)`, images-on/off module apps); `test_sources/players_handbook.pdf` (293 pages, 586 embedded images — the real multi-page fixture), `test_sources/hnb_2026_guide.tar.gz` (the owner's example — stock-photo descriptions with no document context).
|
||||
|
||||
## Objective
|
||||
Every image that comes from a document carries the document context in the text that gets embedded: the vision prompt for a PDF-embedded image includes WHERE it came from (source + PDF path + page) and the surrounding page text, so the model writes a description that names the document and what the image illustrates; the context is stored on `image_context` (idempotent, re-scanned on re-index) and rendered as a short locator line in the chat suggestion block; and every image document already in the KB is backfilled on the next sync. Markdown-referenced images (phase 125) keep working byte-for-byte; PDFs are the only importable format that embeds images, so this closes the TODO's gap completely.
|
||||
|
||||
## Dependencies
|
||||
- `136_force_push_reset` (todo) — pipeline order only.
|
||||
- Code: the completed phase-126 PDF pipeline + the phase-122 vision seam + the phase-125 context machinery (all on `main`); and pending `133_pdf_screenshot_fallback` — its `is_usable_pdf_text` gate (task 02 of that phase) runs BEFORE this phase in the pipeline and is reused to suppress garbage page text in the context (ASSUMPTION A7).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The context block (task 02, ASSUMPTIONS A2/A3):** one block per embedded image, built from the PDF's own text —
|
||||
```
|
||||
Extracted from <source>/<pdf path> (page <N>):
|
||||
<page text, stripped, capped at PDF_IMAGE_CONTEXT_MAX_CHARS = 6000>
|
||||
```
|
||||
The origin line is ALWAYS present (even when the page has no text — the owner's core complaint is "no mention of the document it came from"); the page-text line is omitted for a text-less page. If phase 133's gate exists (`app.rag.pdf.is_usable_pdf_text`) and says the PDF's text layer is unusable, the page text is dropped (origin line only — ASSUMPTION A7). When the image ALSO has phase-125 markdown reference blocks, the stored/passed context is the document block FIRST, then the markdown blocks (provenance → purpose — ASSUMPTION A2); the combining helper is one shared function used by the forward path AND the backfill (task 03) so the two can never drift.
|
||||
- **The seam (task 02, ASSUMPTION A5):** `describe_image`'s `context` prefix wording generalizes from "This image is referenced in documentation with the following context:" to neutral text that covers both provenance and reference (e.g. "This image is part of a document. The following context describes where it comes from and what it is used for:"). `DESCRIBE_PROMPT` (marker + locked instruction) stays byte-identical; `context=None` (standalone image) sends the bare prompt — byte-identical to today, so the phase-122 standalone path and every pre-existing E2E stay green.
|
||||
- **The forward path (task 02):** `_index_pdf_embedded_images` calls the new `extract_pdf_page_texts` ONCE per PDF (a `dict[int, str]` page → stripped text — task 01) and passes it to `_index_one_embedded_image`, which builds the combined context, passes it as `context` to `_describe_or_skip`, and stores the SAME value on `image_context` (the phase-125 task-03 rule: context follows the current source state on re-index — a re-indexed image re-scans; the digest-skip for unchanged images is UNCHANGED: an unchanged image keeps its (now context-carrying) content).
|
||||
- **The backfill (task 03, ASSUMPTION A4):** a post-index pass in `import_sources` next to `_enrich_pending_images` — the phase-118 A2 "best-effort until it sticks" shape. Candidates: `is_image=True` docs whose `path` matches the embedded-image pattern (`*.pdf#page=<N>#img=<I>.<ext>`) AND whose `image_context` does NOT start with `Extracted from ` (NULL or markdown-only) AND whose `image_path` copy still exists on disk. Per candidate: bytes from the copy, page text re-extracted from the parent PDF if it is still in the source (parent gone → origin line only, parsed from the doc's own synthetic path — no disk dependency), re-describe through the SAME seam, re-chunk + re-embed (`_replace_content_chunks`, one transaction per image), store the combined context. Idempotent by the stored-context gate: a success writes a context starting with `Extracted from ` → the doc never qualifies again. Fail-soft per image (a `None` description leaves the row untouched, counts `images_failed`, sync continues). New `ImportSummary.image_context_backfilled: int = 0` counter + `log()` field ONLY (the stats log-only convention — the sync `detail` API shape is untouched). One loud log line per backfilled image (PLAN §9).
|
||||
- **The display line (task 04, ASSUMPTION A6):** `_extract_context_line` learns the `Extracted from <source>/<pdf path> (page <N>):` header (regex; the page-text body is NEVER rendered — same rule as the `Context:` body) and renders it as `From <source>/<pdf path> (page <N>)`. A value carrying BOTH block kinds renders the clauses in stored order (`From …; Referenced in …`). A value with no parsable block degrades to `None` exactly as today (pre-phase rows, markdown `IMAGE_CONTEXT:` records on text docs — the `is_image` guard already excludes those).
|
||||
- **The E2E (task 05):** the mock gains ONE new trigger, following the phase-125 `CONTEXT_LINE_TRIGGER` pattern exactly: when the `IMAGE_DESCRIPTION_MODE` message's text part contains `Extracted from ` (the new provenance header), the canned reply becomes a context-aware description (fixed string, e.g. naming the document + page) instead of `IMAGE_DESCRIPTION_ANSWER` — the deterministic observable that the context REACHED the vision model. The suite uploads a synthetic PDF (text on page 1, one embedded PNG on page 2 — the `test_pdf_support.py` recipe) through the images-on module app, syncs, and asserts: the image doc's content/summary is the context-aware canned reply; its `image_context` starts with `Extracted from <source>/report.pdf (page 2)`; a chat question that retrieves the image shows the `From <source>/report.pdf (page 2)` line in the suggestion. A markdown-referencing case (a `.md` in the same archive referencing the synthetic path) asserts BOTH blocks, document first.
|
||||
|
||||
## Tasks
|
||||
1. `01_pdf_page_texts.md` — `extract_pdf_page_texts` in `app/rag/pdf.py` (the shared per-page iteration, fail-soft) + unit tests on the real fixture PDFs.
|
||||
2. `02_embedded_image_document_context.md` — the document-context block builder + the forward path (build once per PDF, combine with markdown blocks, pass through the seam, store on `image_context`) + the generalized seam prefix + unit tests with a capturing fake LLM.
|
||||
3. `03_backfill_image_context.md` — the per-sync idempotent backfill pass + the `image_context_backfilled` counter + integration tests (existing row backfilled, idempotent no-op, parent-gone origin-only, standalone never touched).
|
||||
4. `04_retrieval_display_line.md` — `_extract_context_line` renders the `Extracted from` block as the one-line `From … (page N)` locator + unit tests (both-blocks ordering, degradation, pre-phase byte-identity).
|
||||
5. `05_e2e_pdf_image_context.md` — the mock's `Extracted from` trigger + the dedicated Playwright suite `tests/e2e/test_pdf_image_context.py`, run in isolation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: task 01 — `extract_pdf_page_texts` on `test_sources/players_handbook.pdf` (293 pages, spot-check known page text, page count, 1-indexed keys) and `pdf_horse_test.pdf`; an unreadable path → `{}` (fail-soft); `extract_pdf_text` output byte-identical after the shared-iteration refactor (the phase-126 regression pin). Task 02 — the block builder (page text capped at 6000, text-less page → origin line only, garbage gate drops the page text when 133's gate exists); the combining order (document first, markdown second; markdown-only and document-only cases); the seam message (context present → neutral prefix + combined text AFTER the byte-identical `DESCRIBE_PROMPT`; context=None → byte-identical pre-137 message); the forward path via a capturing fake LLM (the stored `image_context` == the context passed; the title/unchanged-skip rules untouched). Task 04 — the renderer (`Extracted from` alone, both blocks in stored order, unparseable → `None`, the `is_image` guard). Task 05 — the mock trigger (context-aware reply only when `Extracted from ` is in the text part; the bare `IMAGE_DESCRIPTION_ANSWER` otherwise — the phase-122/126 suites stay green without edits).
|
||||
- Integration: task 03 — a seeded PDF-embedded image row without context is backfilled through the real `import_sources` (the fake LLM records the context-bearing describe call; content/chunks/`image_context` updated); a second sync is a NO-OP (zero describe calls — the idempotency gate); a row whose parent PDF was deleted from the source gets the origin-line-only context; a standalone image (no `#page=`) is NEVER touched; a markdown-enriched row (context `Referenced in …`) gets the combined context (document block prepended).
|
||||
- E2E: one dedicated Playwright file, run in isolation (AGENTS.md §4/§9), mock-only.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a (re)indexed PDF's embedded images are described with the document context: the vision message carries `Extracted from <source>/<pdf> (page N):` + the capped page text (the unit/integration proofs above), and the stored `image_context`/summary carry it too
|
||||
- [ ] a markdown-referenced PDF image carries BOTH blocks, document first — and a standalone image's describe message is byte-identical to pre-phase (the phase-122 regression pin)
|
||||
- [ ] an existing context-free image doc (e.g. one of the HNB-guide stock photos) is re-described WITH context on the next sync, and the following sync burns zero vision calls for it (the idempotency test)
|
||||
- [ ] the chat suggestion block for a backfilled/forward image leads with `From <source>/<pdf> (page N)`
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` green in isolation; the pre-existing `test_pdf_support.py`, `test_image_anchor_context.py`, `test_image_documents.py` suites stay green UNEDITED
|
||||
- [ ] no behavior change in completed phases (the phase-122 standalone path, the phase-125 markdown enrichment, the phase-126 PDF text path, the sync API shape)
|
||||
- [ ] `uv run ruff check . && uv run pyright` green
|
||||
- [ ] one atomic Conventional Commits commit for the phase (`feat(rag): …`), `--no-gpg-sign`
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 01 — `extract_pdf_page_texts`: per-page text map for image context
|
||||
|
||||
**Source:** `TODO.md` L3 — "Image summaries should always include the document context if the images come from a document … so the embedder can more effectively retrieve image sources." (This task builds the per-page text source the context block is made from; the item itself is implemented across tasks 01–05.)
|
||||
|
||||
## Objective
|
||||
`app/rag/pdf.py` gains `extract_pdf_page_texts(pdf_path: Path) -> dict[int, str]` — one entry per page that has text, key = 1-indexed page number, value = the page's stripped text (the same `page.get_text("text")` read `extract_pdf_text` uses). Pages without text contribute nothing (no key). This is the raw material task 02 feeds the vision prompt: for one embedded image on page N, the document context is page N's text. The existing `extract_pdf_text` output stays **byte-identical** (the phase-126 contract — the `---PAGE-N---` blocks, blank-line join, fail-soft `""`).
|
||||
|
||||
## Work
|
||||
1. Read `app/rag/pdf.py` in full (phase 126). Note `extract_pdf_text` (L87): open → per-page `get_text("text")` → strip → skip empties → `PAGE_SEPARATOR` prefix → `"\n\n"` join; fail-soft per page and per file.
|
||||
2. Refactor the shared page iteration into one private helper, e.g. `_iter_page_texts(doc) -> Iterator[tuple[int, str]]` yielding `(page_1_indexed, stripped_text)` for pages with non-empty text (the per-page try/except + warning logging moves into the helper, unchanged in behavior). `extract_pdf_text` becomes: open (fail-soft `""`) → consume the helper → build the separator-joined string. Its signature, output bytes, and log lines are UNCHANGED.
|
||||
3. Add the public function:
|
||||
```python
|
||||
def extract_pdf_page_texts(pdf_path: Path) -> dict[int, str]:
|
||||
"""One entry per page of *pdf_path* that has text (1-indexed page → stripped text). …"""
|
||||
```
|
||||
Same open/fail-soft contract as `extract_pdf_text` (unreadable file → `{}`, one warning — the `pdf:` logger prefix house style).
|
||||
4. Update the module docstring's function list (the file documents every public function at the top).
|
||||
5. Unit tests in `tests/unit/test_pdf.py` (the file that owns `app/rag/pdf` — extend it):
|
||||
- `test_sources/players_handbook.pdf` (293 pages): 293 or fewer entries, keys 1-indexed, first key `1`, every value non-empty and stripped; spot-check one page's text equals the corresponding `extract_pdf_text` block body (parse a `---PAGE-N---` block and compare — the two functions must agree page-for-page).
|
||||
- `test_sources/pdf_horse_test.pdf`: 1 page → exactly one entry (or `{}` if that page is text-less — assert against `extract_pdf_text`'s output for the same file, whatever it is).
|
||||
- A missing path → `{}` (fail-soft, no raise).
|
||||
- **Regression pin:** `extract_pdf_text` on both fixture files is byte-identical before/after the refactor (capture the current output NOW, in the test, by asserting the separator structure + a known substring from page 1 — do not hardcode a 300-page string).
|
||||
6. Run the phase-126 pdf unit tests + `uv run ruff check . && uv run pyright` — green before moving on.
|
||||
|
||||
## Testing & Quality
|
||||
- The tests above ARE the task's tests; all in the existing `app/rag/pdf` unit-test file (no new file unless the house layout says otherwise — check `tests/unit/` first).
|
||||
- Coverage: `app/rag/pdf.py` stays >90% (the helper's per-page failure branch: reuse the existing unreadable-page test pattern if one exists; otherwise a temp file that fitz opens but fails mid-read is not worth fabricating — the existing coverage of that branch via `extract_pdf_text`'s tests carries over since both call sites share the helper).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `extract_pdf_page_texts` exists, is documented, fail-soft, and returns 1-indexed page → stripped text
|
||||
- [ ] `extract_pdf_text` output is byte-identical (the regression pin green on both fixture PDFs)
|
||||
- [ ] `uv run pytest tests/unit -k pdf -v --no-cov` green; `uv run ruff check . && uv run pyright` green
|
||||
@@ -0,0 +1,72 @@
|
||||
# Task 02 — The forward path: document context into the vision prompt + `image_context`
|
||||
|
||||
**Source:** `TODO.md` L3 — "Images extracted from PDFs still don't have any relevant context in their summaries … there are lots of stock photos in test_sources/hnb_2026_guide.tar.gz and they get summaries like 'A smiling woman with dark curly hair sits holding a grey laptop on her lap...' which has no mention of the document it came from or what it might have been helping to illustrate. … Image summaries should always include the document context if the images come from a document."
|
||||
|
||||
## Objective
|
||||
When a PDF is (re)indexed, every embedded image's vision call and stored `image_context` carry a document-context block — `Extracted from <source>/<pdf path> (page <N>):` plus the page's text (capped) — combined with any phase-125 markdown reference blocks (document block first). The vision model therefore writes a description that names the document and the page, and the stored `content`/`summary`/`image_context` carry the same context the embedder needs. Standalone images and the phase-125 markdown-only shape are untouched.
|
||||
|
||||
## Work
|
||||
1. Read, in full: `app/rag/importer.py` — `_index_pdf_embedded_images` (L1091), `_index_one_embedded_image` (L1182), `_markdown_image_context` (L1403), `_md_ref_block` (L1436), `_describe_or_skip` (L1742); `app/rag/summarizer.py` — `describe_image` (L208, esp. the `context_prefix` build near the message construction).
|
||||
2. **The block builder** in `app/rag/importer.py` (next to `_markdown_image_context`):
|
||||
```python
|
||||
PDF_IMAGE_CONTEXT_MAX_CHARS = 6000 # ASSUMPTION A3 — module constant, no env var
|
||||
|
||||
def _pdf_image_context_block(
|
||||
source: str, pdf_rel: str, page: int, page_texts: dict[int, str],
|
||||
text_usable: bool = True,
|
||||
) -> str:
|
||||
"""The document-context block for one embedded image (phase 137, A2).
|
||||
|
||||
Always non-empty: the origin line is present even for a text-less
|
||||
page. The page text is stripped, capped at
|
||||
:data:`PDF_IMAGE_CONTEXT_MAX_CHARS` (overflow cut at the cap,
|
||||
then ``TRUNCATION_MARKER`` on its own line — the summarizer's
|
||||
existing truncation convention), and OMITTED entirely when the
|
||||
page has no text OR when the phase-133 gate (see step 3) marks
|
||||
the PDF's text layer unusable.
|
||||
"""
|
||||
```
|
||||
Shape (exactly):
|
||||
```
|
||||
Extracted from <source>/<pdf_rel> (page <N>):
|
||||
<page text or — for a text-less page — nothing after the header line>
|
||||
```
|
||||
`TRUNCATION_MARKER` lives in `app.rag.retriever` (import it there-from, the same way `app/rag/summarizer.py` does — L49).
|
||||
(header line `Extracted from {source}/{pdf_rel} (page {page}):` — `page` is the 1-indexed page number (the forward path passes `img.page`; the task-03 backfill passes the page parsed from the synthetic path — taking a plain `int`, NOT a `PdfImage`, is deliberate so the backfill needs no fake `PdfImage`); NO "of M" total — the total page count is not on `PdfImage` and the origin line must stay a stable single line for the task-04 renderer's regex.)
|
||||
3. **The garbage-text guard (ASSUMPTION A7):** the phase-133 gate `is_usable_pdf_text` is added by pending `133_pdf_screenshot_fallback` (runs BEFORE this phase in the pipeline). In `_index_pdf_embedded_images`, import it defensively — module-level `from app.rag.pdf import extract_pdf_images, extract_pdf_page_texts` and, for the gate, `is_usable_pdf_text = getattr(_pdf_module, "is_usable_pdf_text", None)` (a small import-time probe or a try/except ImportError fallback) — and pass a `text_usable: bool` (default `True` when the gate is absent) down to `_index_one_embedded_image` → `_pdf_image_context_block`: `text_usable=False` → the block is the origin line only. Never import the gate unconditionally (the phase must run in isolation if 133 is skipped).
|
||||
4. **`_index_pdf_embedded_images`:** after the `settings.images` gate, call `extract_pdf_page_texts(full_path)` ONCE (task 01's function) and pass the resulting `page_texts` + `text_usable` to `_index_one_embedded_image` (add keyword params — the function signature is internal, no other callers: verify with `rg "_index_one_embedded_image"`).
|
||||
5. **`_index_one_embedded_image`:** where it currently computes `image_context = _markdown_image_context(...)` (the phase-125 task-03 scan), build the combined context with a NEW shared helper (task 03 reuses it — define it once, near the block builder):
|
||||
```python
|
||||
def _combine_image_contexts(doc_block: str, md_blocks: str | None) -> str:
|
||||
"""Document block first, markdown reference block(s) second (A2).
|
||||
|
||||
``md_blocks`` is ``None``/empty → the doc block alone; the doc
|
||||
block is ALWAYS present for embedded images (it is the provenance
|
||||
— never empty). Markdown-only values (no doc block) are produced
|
||||
by the standalone path only and never pass through this helper.
|
||||
"""
|
||||
```
|
||||
Set `image_context = _combine_image_contexts(block, md)` — everything downstream is unchanged (the `context=` arg of `_describe_or_skip`, the `content = f"{image_context}\n\n{description}"` enrichment, the row write, `_store_summary`).
|
||||
6. **The seam prefix (ASSUMPTION A5):** in `app/rag/summarizer.py`'s `describe_image`, replace the context prefix
|
||||
`"\n\nThis image is referenced in documentation with the following context:\n{context}"`
|
||||
with neutral wording covering both provenance and reference, e.g.
|
||||
`"\n\nThis image is part of a document. The following context describes where it comes from and what it is used for:\n{context}"`.
|
||||
`DESCRIBE_PROMPT` (marker + locked instruction) is UNTOUCHED; `context=None` still yields the bare prompt (byte-identical — assert it). Update `describe_image`'s docstring (`*context*` no longer means "markdown reference context" only — it means "the document/reference context built by the importer") and `_describe_or_skip`'s docstring likewise.
|
||||
7. Unit tests (extend the importer's existing image tests — `rg "_markdown_image_context|_index_one_embedded_image" tests/unit`):
|
||||
- **Block builder:** page with text → header + text; text capped at exactly 6000 + `TRUNCATION_MARKER` (build a page of ~7000 chars); text-less page → header line only; `text_usable=False` → header line only even with text present.
|
||||
- **Combining:** doc-only; doc + markdown (doc block first, blank-line join — assert the exact joined string); markdown shape never lost (the `Referenced in …` block byte-identical inside the combined value).
|
||||
- **Forward path with a capturing fake LLM** (the importer's existing fake pattern — `chat` records the message list): a PDF with page text + one embedded image → the `describe_image` message's text part == `DESCRIBE_PROMPT` + neutral prefix + combined context (assert the prefix bytes explicitly); the stored `Document.image_context` == the same combined value; `content`/`summary` == `context + "\n\n" + description`.
|
||||
- **Regression pins:** standalone image (no markdown, no PDF) → message byte-identical to pre-phase (capture the current message in a test by asserting `DESCRIBE_PROMPT` is the entire text part — `context=None`); markdown-only reference for a STANDALONE image (the phase-125 path) → still the markdown block with the (now neutral) prefix — the stored value is byte-identical (only the prompt prefix wording moved, not the stored data).
|
||||
- The phase-126 unchanged-image digest skip is untouched (an unchanged embedded image still re-describes nothing — add/keep the existing test).
|
||||
8. `uv run pytest tests/unit -k "pdf or image" -v --no-cov`, then the full unit suite; `uv run ruff check . && uv run pyright` — green before moving on.
|
||||
|
||||
## Testing & Quality
|
||||
- The tests in step 7; every one deterministic (fake LLM, fixture PDFs — no network, no real model).
|
||||
- Coverage: new/modified lines in `app/rag/importer.py` + `app/rag/summarizer.py` covered by these tests (the cap boundary, both guard branches, both combining branches).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a (re)indexed PDF's embedded images reach the vision model with `Extracted from <source>/<pdf> (page N):` + the (capped) page text, and the stored `image_context`/`content`/`summary` carry the same value
|
||||
- [ ] a markdown-referenced embedded image carries BOTH blocks, document first (exact-string test)
|
||||
- [ ] the standalone image's describe message is byte-identical to pre-phase; `DESCRIBE_PROMPT` unchanged
|
||||
- [ ] garbage-text guard: when the phase-133 gate exists and fails, the block is the origin line only; when the gate is absent, the block behaves normally (the phase runs in isolation)
|
||||
- [ ] `uv run pytest --no-cov` green (full suite); `uv run ruff check . && uv run pyright` green
|
||||
@@ -0,0 +1,45 @@
|
||||
# Task 03 — Backfill: existing PDF-embedded image docs gain the document context on the next sync
|
||||
|
||||
**Source:** `TODO.md` L3 — "Images extracted from PDFs still don't have any relevant context in their summaries … That context is critical in the image summaries. Image summaries should always include the document context if the images come from a document …" (The forward path of task 02 only fixes images that get (re)indexed; this task fixes the images ALREADY in the KB — e.g. the HNB-guide stock photos the owner cited — without waiting for a byte change.)
|
||||
|
||||
## Objective
|
||||
A per-sync, idempotent, best-effort pass (the phase-118 A2 "until it sticks" shape, next to the phase-125 `_enrich_pending_images` post-index pass) re-describes every PDF-embedded image document that is missing the document-context block: bytes come from the `image_dir` copy, the page text is re-extracted from the parent PDF when it is still in the source (parent gone → origin line only), the new context-aware description goes through the SAME seam, and the row's `content`/`summary`/`image_context`/chunks are replaced. After a success the doc never qualifies again; a failure leaves the row untouched and retries next sync.
|
||||
|
||||
## Work
|
||||
1. Read, in full: `_enrich_pending_images` (L1627 — the shape to mirror: driven by committed doc rows, idempotent, fail-soft, one transaction per image, deletes nothing), the per-source loop in `import_sources` (L525–590: `source = root.name`, the `_enrich_pending_images` call site at L587), `ImportSummary` (L212 + its `log()` method — the stats log-only convention: new field + log field, the sync `detail` API shape never changes), `_store_summary` (the image-aware summary store — the description IS the summary, stored verbatim), `_replace_content_chunks` (the per-image re-chunk + re-embed), `_pdf_image_context_block` / `_combine_image_contexts` / `extract_pdf_page_texts` (tasks 01–02).
|
||||
2. **The candidate query + gate** in a new function `async def _backfill_image_contexts(session, *, source: str, root: Path, llm: Embedder, summary: ImportSummary) -> None` in `app/rag/importer.py`:
|
||||
- Select this source's `Document` rows with `is_image.is_(True)`.
|
||||
- A doc is a CANDIDATE iff ALL hold:
|
||||
1. its `path` parses as the phase-126 embedded-image synthetic path `<pdf_rel>#page=<N>#img=<I>.<ext>` (module-level compiled regex, e.g. `^.+\.pdf#page=(?P<page>\d+)#img=(?P<img>\d+)(?P<ext>\.\w+)$` — parse out `pdf_rel`, page, index, ext; non-matches — standalone images, screenshot-fallback PDFs are `is_image=False` anyway — are NEVER touched);
|
||||
2. its `image_context` is `NULL` OR does not start with `"Extracted from "` (the idempotency gate — a markdown-only `"Referenced in …"` value STILL qualifies: the doc block is missing);
|
||||
3. its `image_path` names a file that exists on disk (the `image_dir` copy — the bytes source; missing → debug log, skip, the doc stays as-is).
|
||||
- If no candidates → return (the common case after the one-time backfill — zero cost per sync).
|
||||
3. **Per candidate** (fail-soft, one transaction per image, mirroring `_enrich_pending_images`'s error handling):
|
||||
- `page_texts`: call `extract_pdf_page_texts(root / pdf_rel)` ONCE per unique `pdf_rel` across the candidates (a local cache dict in the pass — one parent may own many images); parent missing/unreadable → `page_texts = {}` AND `text_usable = True` (origin line only — the origin is still valuable and needs no disk); parent present → apply the same phase-133 gate probe as task 02 (shared helper if task 02 factored one out — reuse it, do not duplicate the probe).
|
||||
- Build `doc_block = _pdf_image_context_block(source, pdf_rel, page, page_texts, text_usable=text_usable)` (`page` from the regex — the builder takes a plain `int`, task 02), then `combined = _combine_image_contexts(doc_block, existing_md_blocks)` — where `existing_md_blocks` = the doc's current `image_context` if it starts with `"Referenced in "` (the markdown blocks are kept verbatim — re-scanning the disk for them is NOT the pass's job: the phase-125 pass owns the markdown context and ran first this sync), else `None`.
|
||||
- `data = Path(doc.image_path).read_bytes()` (OSError → skip, debug log).
|
||||
- `description = await _describe_or_skip(llm, data=data, source=source, rel=doc.path, full_path=Path(doc.image_path), context=combined)` — the SAME seam (the mime comes from the copy's extension, exactly like the forward path). `None` → the row is UNTOUCHED (no rollback of anything — nothing was written), `summary.images_failed += 1`, warning, move on (the next sync retries — the gate still matches).
|
||||
- Write: `content = f"{combined}\n\n{description}"`; `image_context = combined`; `indexed_at = now`; then `_replace_content_chunks(...)` (same call shape as the forward path, L1311) in the image's own transaction (an embed failure rolls back THIS image only — `summary.errors += 1`, error line, the copy stays, move on); then `_store_summary(...)` (the image convention — the description is the summary, verbatim); commit; `summary.image_context_backfilled += 1`; one loud log line (PLAN §9): `import: backfilled image context source=<source> path=<doc.path> chunks=<n>`.
|
||||
- Deliberately UNCHANGED on the row: `content_hash` (the byte digest — the image identity is untouched), `created_at`/`created_at_manual` (the SOURCE did not change — the phase-106 date rules do not apply), `title`, `image_path`.
|
||||
4. **The counter:** `ImportSummary` gains `image_context_backfilled: int = 0` (docstring: "PDF-embedded image docs whose stored `image_context` was backfilled with the document-context block (phase 137, task 03) — the one-time best-effort pass; a success never repeats, a failure retries next sync") + its field in `ImportSummary.log()` (the per-turn/per-sync log line — append it next to `images_failed`; the sync `detail` API response shape is NOT changed — the stats log-only convention, phase 133).
|
||||
5. **The call site:** in `import_sources`'s per-source loop, immediately AFTER the existing `await _enrich_pending_images(...)` (L587 — enrichment first: it owns the markdown-FIRST case and may set `image_context`; the backfill then prepends the document block to whatever is there). The pass is driven by committed rows (like the enrichment pass), so it is safe under `limit`.
|
||||
6. **Integration tests** (extend the importer's existing integration file — `rg "_enrich_pending_images" tests/integration`):
|
||||
- **Backfill happy path:** seed the DB with a PDF-embedded image row (the phase-126 synthetic path, `image_context=NULL`, `content`/`summary` = a bare description, an `image_path` pointing at a temp file with the image bytes) + the parent PDF on disk with page text; run the REAL `import_sources` over the source (fake LLM that records describe calls and returns a fixed context-aware reply); assert: exactly ONE describe call whose context carries `Extracted from <source>/<pdf> (page N):` + the page text; the row's `image_context` starts with `Extracted from `; `content`/`summary` = combined context + new description; chunks replaced; `image_context_backfilled == 1`.
|
||||
- **Idempotency:** run `import_sources` a second time over the same source → ZERO describe calls, the row byte-identical (the gate no longer matches), `image_context_backfilled == 0`.
|
||||
- **Parent gone:** the parent PDF is deleted from the source dir → the describe call's context is the origin line ONLY (no page-text line); the row is still backfilled (the gate matches again next sync iff it failed — here it succeeds once).
|
||||
- **Markdown-only row:** `image_context = "Referenced in <source>/doc.md:\n- Alt text: \"x\""` (the phase-125 shape) → the backfilled `image_context` = doc block + blank line + the markdown block VERBATIM (document first).
|
||||
- **Standalone image never touched:** an `is_image` row whose `path` has no `#page=` (a plain `pic.png`) with `image_context=NULL` → zero describe calls for it, across two runs.
|
||||
- **Fail-soft:** a candidate whose describe call returns `None` (the fake fails once) → the row untouched, `images_failed == 1`; a second run with a healthy fake → backfilled.
|
||||
7. `uv run pytest tests/integration -k "image or import" -v --no-cov`, then the full unit + integration suites; `uv run ruff check . && uv run pyright` — green before moving on.
|
||||
|
||||
## Testing & Quality
|
||||
- The tests in step 6 (real `import_sources`, fake LLM, temp dirs + seeded session — the house integration pattern; no network, no real model).
|
||||
- Coverage: every gate branch (pattern non-match, gate match, missing copy, parent gone), the happy path, idempotency, the fail-soft paths — all new lines covered.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] an existing context-free PDF-embedded image doc is re-described WITH the document context on the next sync (integration test), counted in the new log-only `image_context_backfilled` counter
|
||||
- [ ] the pass is idempotent: the following sync burns zero vision calls for a backfilled doc (integration test)
|
||||
- [ ] a parent-gone image gets the origin-line-only context; a markdown-enriched row keeps its markdown block verbatim, document block prepended
|
||||
- [ ] standalone images and non-embedded rows are never touched; a describe failure leaves the row untouched and retries next sync
|
||||
- [ ] the sync `detail` API shape is unchanged (the new counter is log-only — verify the sync response schema test still passes unedited)
|
||||
- [ ] `uv run pytest --no-cov` green; `uv run ruff check . && uv run pyright` green
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 04 — The suggestion-block display line: `From <source>/<pdf> (page N)`
|
||||
|
||||
**Source:** `TODO.md` L3 — "…so the embedder can more effectively retrieve image sources." (The summary itself now carries the context via tasks 02/03; this task makes the context VISIBLE in the chat suggestion block the way phase 125 did for markdown references — a short locator line, never the page-text body.)
|
||||
|
||||
## Objective
|
||||
`_extract_context_line` in `app/rag/prompts.py` renders the phase-137 document-context block — `Extracted from <source>/<pdf path> (page <N>):` — as the one-line clause `From <source>/<pdf path> (page <N>)`, so a PDF-embedded image's `<document>` suggestion body leads with where the image came from (before its description), exactly the way an enriched markdown image leads with `Referenced in …`. A context value carrying BOTH block kinds renders both clauses in stored order. Everything that parses to nothing degrades to `None` exactly as today (pre-phase rows, the `IMAGE_CONTEXT:` pending record, unparseable text).
|
||||
|
||||
## Work
|
||||
1. Read, in full: `_extract_context_line` (L470), `_image_ctx_clause` (L459), `_image_context_line` (L536), `_document_body` (L548) in `app/rag/prompts.py` — the phase-125 task-05 renderer: header regex `_IMAGE_CTX_HEADER_RE` (`^Referenced in (?P<ref>.+):$`), the alt-bullet regex, the clause join (`"Referenced in " + "; ".join(clauses)`), and the `expecting_alt` state machine.
|
||||
2. Add the header regex for the new block (next to `_IMAGE_CTX_HEADER_RE`): `_PDF_CTX_HEADER_RE = re.compile(r"^Extracted from (?P<loc>.+) \(page (?P<page>\d+)\):$")` — `loc` is the `<source>/<pdf path>` locator (greedy to the last ` (page N):` — the pdf path may itself contain spaces).
|
||||
3. Extend the state machine in `_extract_context_line` to handle the new header:
|
||||
- A `Extracted from …` header STARTS a pdf clause: it needs NO following bullet (unlike the `Referenced in` header, whose next line MUST be the alt bullet or the block is dropped — the pdf block's next line is page-text body or the next header, and none of it changes the clause).
|
||||
- Its clause text: `From {loc} (page {page})`.
|
||||
- Clause JOINING: the current code prefixes `"Referenced in "` to the joined clauses — that prefix belongs to the REFERENCED clauses only. Restructure so each clause carries its own prefix: referenced clauses stay `Referenced in <ref>: "<alt>"` (byte-identical output for markdown-only values — the phase-125 regression pin), pdf clauses are `From <loc> (page N)`; the join is still `"; "` between clauses IN STORED ORDER (document block first per ASSUMPTION A2 → a both-blocks value renders `From a.pdf (page 2); Referenced in docs/a.md: "alt"`).
|
||||
- A pdf clause followed by page-text body lines: the body lines are skipped by the existing "any other line is context body" fall-through — verify the state machine does not mis-read a page-text line that LOOKS like a header (e.g. a line of page text starting with `Referenced in ` — the existing spurious-header rule already handles the referenced case by demanding the alt bullet; a spurious `Extracted from … (page N):` line inside page text would emit a clause — accept that: it is the stored context the importer wrote, not model text; document it in the docstring).
|
||||
- A value with NO parsable header of either kind → `None` (unchanged).
|
||||
- Update `_extract_context_line`'s docstring (the stored value can now carry the `Extracted from` block first — tasks 02/03 write it — plus the markdown blocks) and the module-level comment block above the regexes (the phase-137 note).
|
||||
4. `_image_context_line` / `_document_body` need NO changes (the `is_image` guard + the renderer call already do the right thing) — verify with a read, and note it in the commit message if so.
|
||||
5. Unit tests (extend the prompts test file — `rg "_extract_context_line|_document_body" tests/unit`):
|
||||
- pdf block alone → `"From <source>/guide.pdf (page 2)"` exactly.
|
||||
- pdf block + page-text body (multi-line, including a blank line and a line starting with `Referenced in ` without the alt bullet) → the same single clause (the spurious referenced header is dropped by the existing rule).
|
||||
- markdown block alone → byte-identical to the pre-task output (the phase-125 pin: `Referenced in docs/a.md: "alt a"; docs/b.md` from the existing test's shape).
|
||||
- BOTH blocks in stored order (pdf first, markdown second — the tasks 02/03 combined shape) → `From <source>/guide.pdf (page 2); Referenced in docs/a.md: "alt"`.
|
||||
- garbage/unparseable value → `None` (unchanged); the `IMAGE_CONTEXT:` pending record → `None` (unchanged).
|
||||
- `_document_body` end-to-end: an `is_image` doc with the combined context → body leads with the one-line rendering, then the summary; a non-image doc with a non-NULL `image_context` → byte-identical body (the `is_image` guard).
|
||||
6. `uv run pytest tests/unit -k prompt -v --no-cov`, then the full unit suite; `uv run ruff check . && uv run pyright` — green before moving on.
|
||||
|
||||
## Testing & Quality
|
||||
- The tests in step 5; the markdown-only byte-identity pin is the load-bearing one (phase 125's display contract must not move).
|
||||
- Coverage: the new header branch, both orderings, the degradation paths — all covered by step 5.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a PDF-embedded image doc's suggestion `<document>` body leads with `From <source>/<pdf> (page N)` (unit test), never rendering the page-text body
|
||||
- [ ] a both-blocks context renders `From …; Referenced in …` in stored order; a markdown-only context renders byte-identically to pre-phase (the phase-125 pin green UNEDITED)
|
||||
- [ ] unparseable values and the `IMAGE_CONTEXT:` record degrade to `None` as before; non-image docs are byte-identical
|
||||
- [ ] `uv run pytest --no-cov` green; `uv run ruff check . && uv run pyright` green
|
||||
@@ -0,0 +1,58 @@
|
||||
# Task 05 — The dedicated E2E: `tests/e2e/test_pdf_image_context.py` (mock trigger + the story suite)
|
||||
|
||||
**Source:** `TODO.md` L3 — the whole item, end to end: "Images extracted from PDFs still don't have any relevant context in their summaries … Image summaries should always include the document context if the images come from a document (this includes markdown files) so the embedder can more effectively retrieve image sources." (AGENTS.md §4/§9: every phase carries its own Playwright E2E file, run in isolation; this is this phase's gate.)
|
||||
|
||||
## Objective
|
||||
One new Playwright suite proving the story through the REAL UI + REAL pipeline with the deterministic mock LLM: (a) an uploaded PDF's embedded image is described WITH the document context (the context-aware canned reply is the witness that the context reached the vision model), stored on `image_context`/`content`/`summary`; (b) the chat suggestion for that image leads with the `From <source>/<pdf> (page N)` line; (c) a markdown reference in the same source COMBINES with the document context, document block first. Plus the one mock-LLM change that makes (a) observable.
|
||||
|
||||
## Work
|
||||
1. Read, in full: `tests/e2e/test_pdf_support.py` (the module-app pattern: `_app_env` with the port/env overrides, `zip_path` fixture — the PyMuPDF PDF build: `doc.new_page().insert_text((72, 72), PAGE_ONE)`, `page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)`, `doc.set_metadata({"title": PDF_TITLE})`, zip to `<source>.zip`; the `_truncate_all`, `_upload_zip` (admin login → git-sources page → `#archive-upload-file`/`#archive-upload-btn` → result line), `_sync_via_ui` (RAG page `#sync-btn` → `#sync-label` "Syncing…" → synced label + `#sync-result`), and `seeded` module fixture), `tests/e2e/test_image_anchor_context.py` (the `ANCHOR_QUESTION = "quote the image reference — what is shown in the homelab network diagram?"` pattern + the mock-calibrated gate env values + the standalone-doc "no reference line" witness), `tests/e2e/mock_llm.py` — the `IMAGE_DESCRIPTION_MODE` branch (L2472), `IMAGE_DESCRIPTION_ANSWER` (L865), `CONTEXT_LINE_TRIGGER` (L1886) + its branch (L2532, `_seeded_image_doc` / `_seed_context_line`), and the docstring's behavior list at the top (every new mock behavior is documented there).
|
||||
2. **The mock change** (the ONLY edit to `tests/e2e/mock_llm.py`):
|
||||
- New constant next to `IMAGE_DESCRIPTION_ANSWER`:
|
||||
```python
|
||||
#: The phase-137 witness — the mock's vision reply when the
|
||||
#: ``IMAGE_DESCRIPTION_MODE`` message's text part carries the
|
||||
#: document-context block (its ``Extracted from …`` header). It
|
||||
#: names the document + page (the context reached the model — the
|
||||
#: deterministic proof) and keeps the diagram wording of
|
||||
#: :data:`IMAGE_DESCRIPTION_ANSWER` so the story question's
|
||||
#: token-overlap retrieval still grounds on the image doc.
|
||||
IMAGE_DESCRIPTION_CONTEXT_ANSWER = (
|
||||
"From report.pdf, page 2: a network diagram of the homelab "
|
||||
"server room: a core router on top, a core switch in the "
|
||||
"middle, and three labeled subnets at the bottom — VLAN 10 "
|
||||
"office, VLAN 20 lab, and VLAN 30 storage — with a legend of "
|
||||
"cable runs. Title: Homelab Network Map."
|
||||
)
|
||||
```
|
||||
- In the `IMAGE_DESCRIPTION_MODE` branch: `answer = IMAGE_DESCRIPTION_CONTEXT_ANSWER if "Extracted from " in user else IMAGE_DESCRIPTION_ANSWER` — the `Extracted from ` probe is the phase-137 provenance header (the importer's block builder — task 02); the bare path (standalone images, the pre-137 embedded shape, every existing suite's PDFs) stays on `IMAGE_DESCRIPTION_ANSWER` — byte-identical.
|
||||
- Document the new branch behavior in the mock's top docstring (the house rule — every behavior is listed).
|
||||
- NO other mock change: the `CONTEXT_LINE_TRIGGER` branch already echoes the seeded image doc's block FIRST line — which is now the task-04 `From …` rendering for PDF images (zero mock work for the display assertion).
|
||||
3. **The suite** `tests/e2e/test_pdf_image_context.py` (run in isolation: `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov`):
|
||||
- Module app: ONE images-ON app, the `test_pdf_support.py` pattern — new env port (`E2E_APP_PORT_PDF_IMG_CTX`, default 8154 — check `8152/8153` are taken by test_pdf_support and pick the next free), scratch dirs via `tmp_path_factory`, the mock's port from the session `mock_llm` fixture; assert `/api/config` `images` is True in the seed (the suite is void otherwise).
|
||||
- **Fixture archive 1** (`zip_path`): ONE PDF at the root, source `e2e-pdf-imgctx`, `report.pdf` — page 1 = `PAGE_ONE` text, page 2 = `PAGE_TWO` text + the 1×1 PNG embedded (the phase-126 recipe; define `PAGE_ONE`/`PAGE_TWO`/`PDF_TITLE`/`PNG_1X1` locally — do NOT import from test_pdf_support; the suites must stay standalone).
|
||||
- **Seed 1** (module): `_truncate_all` → `_upload_zip` → `_sync_via_ui` (expected result line: the 2-added counts — the PDF doc + the image doc).
|
||||
- **Test 1 — the forward path (the TODO's core complaint):** DB assertions (`SessionLocal`, like test_pdf_support's `seeded`):
|
||||
- the image doc exists at `e2e-pdf-imgctx/report.pdf#page=2#img=0.png`, `is_image=True`;
|
||||
- its `image_context` STARTS WITH exactly `Extracted from e2e-pdf-imgctx/report.pdf (page 2):` and CONTAINS the `PAGE_TWO` text (the document context reached the store — the owner's "no mention of the document it came from" is gone);
|
||||
- its `content` == `image_context + "\n\n" + IMAGE_DESCRIPTION_CONTEXT_ANSWER` and `summary` == `content` (the image convention) — the CONTEXT-AWARE reply is the witness the describe call carried the context (the bare `IMAGE_DESCRIPTION_ANSWER` string must NOT appear in `content`).
|
||||
- **Test 2 — the display line in chat:** the `ANCHOR_QUESTION` pattern (`"quote the image reference — what is shown in the homelab network diagram?"` — the diagram wording is in `IMAGE_DESCRIPTION_CONTEXT_ANSWER`, so the same retrieval grounds), the mock-calibrated gate env (the test_image_anchor_context values); assert the final answer contains `From e2e-pdf-imgctx/report.pdf (page 2)` (the mock's `CONTEXT_LINE_TRIGGER` echo of the block's first line — the task-04 rendering) and does NOT contain `stands alone` (the standalone witness is absent).
|
||||
- **Fixture archive 2** (`zip_md_path`): the SAME PDF + `notes.md` at the root containing `` plus a paragraph of surrounding context (the phase-125 reference shape) — source `e2e-pdf-imgctx-md`.
|
||||
- **Seed 2** (module): `_truncate_all` → upload archive 2 → sync (expected counts: PDF + image + the markdown doc).
|
||||
- **Test 3 — markdown + document combined, document first:** the image doc's `image_context` == `Extracted from e2e-pdf-imgctx-md/report.pdf (page 2):` + `PAGE_TWO` text + `"\n\n"` + the markdown block starting `Referenced in e2e-pdf-imgctx-md/notes.md:` (assert the STARTS-WITH document block AND the `Referenced in` block's presence AND the document block's index < the `Referenced in` block's index — document first, ASSUMPTION A2); the chat display answer (the same anchor question) contains BOTH `From e2e-pdf-imgctx-md/report.pdf (page 2)` and `Referenced in e2e-pdf-imgctx-md/notes.md`.
|
||||
- **Test 4 — standalone parity witness:** a plain standalone image upload in the SAME seeded source is overkill; instead assert against seed 1's KB: NO other `is_image` doc exists (the suite's KB is exactly the PDF + its image), and the mock's bare branch is pinned by test_pdf_support running green unedited (do not duplicate it).
|
||||
4. **Regression run (the load-bearing gate):** `uv run pytest tests/e2e/test_pdf_support.py tests/e2e/test_image_anchor_context.py tests/e2e/test_image_documents.py -v --no-cov` — all three green UNEDITED (the mock change must not move their canned replies — the bare describe path is byte-identical; the `CONTEXT_LINE_TRIGGER` echo of a markdown-enriched doc is byte-identical).
|
||||
5. `uv run ruff check . && uv run pyright` — green. (The suite itself is linted like its siblings; no type-check burden on the mock constants.)
|
||||
|
||||
## Testing & Quality
|
||||
- The suite in step 3 IS the phase's E2E deliverable; deterministic end to end (mock LLM, fixture PDFs built in tmp dirs — `test_sources/` is never touched, the house rule from phase 133's fixture handling).
|
||||
- The mock edit is covered by the suite's tests 1–3 (the trigger) + the regression run step 4 (the non-trigger).
|
||||
- Coverage gate (app/) is unaffected by test files; the phase's >90% comes from tasks 01–04.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/e2e/test_pdf_image_context.py` exists and passes IN ISOLATION: `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov`
|
||||
- [ ] the embedded image's stored summary names the document + page (the context-aware canned reply — test 1); `image_context` leads with `Extracted from <source>/report.pdf (page 2):` + the page text
|
||||
- [ ] the chat suggestion for the image leads with `From <source>/report.pdf (page 2)` (test 2, via the existing `CONTEXT_LINE_TRIGGER` echo — no new display mock branch)
|
||||
- [ ] a markdown-referenced PDF image shows BOTH the document and the reference context, document first (test 3)
|
||||
- [ ] `test_pdf_support.py`, `test_image_anchor_context.py`, `test_image_documents.py` green UNEDITED after the mock change
|
||||
- [ ] `uv run ruff check . && uv run pyright` green
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
All verification complete. Final report:
|
||||
|
||||
**Phase 127 — final verification pass: all green** (all 4 tasks were already in `complete/`; no code changes were needed this pass)
|
||||
|
||||
- **Verified implementation:** `ImportSummary.changed_paths` recorded at all 5 added/updated sites + `_prune` (before delete, guards intact); `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys = candidates ∩ affected, prune pass both modes, hook total = affected count); sync's changed-KB branch passes `changed_docs=summary.changed_paths` with the `changed=N` log line; CLI (`scripts/import_docs.py`) passes no change set — full-regen contract preserved
|
||||
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, 0 errors
|
||||
- **Completion criteria:** ① one-file change rewrites only the folder chain (E2E pins sibling rows byte-identical, text AND `updated_at`) ✅ ② coverage >90% ✅ ③ E2E isolated green ✅ ④ no completed-phase behavior change (full suite + first-sync full-regen E2E) ✅ ⑤ atomic commit — delegated to the harness per this pass's rules (I make no git changes)
|
||||
- **Docs:** README.md — fixed 7 pre-existing inconsistencies: missing Theme row in the pages table; stale `ls` (now drill-down tree, phase 94) and `read` (now capped at `BOR_READ_MAX_CHARS`, phase 95) tool descriptions; missing phase-124 answer images (new section + `show_image` tool row + `BOR_ANSWER_IMAGES` env row); "Document summaries (non-markdown)" → all docs incl. markdown (phase 118); theme palette 8 → 17 colors (phases 92/93); `BOR_LLM_SUMMARY_MODEL` now names folder summaries; CLI section notes overview + folder-summary regen. AGENTS.md — verified consistent, no changes needed
|
||||
- **Deviations:** none. Untracked pre-existing `NOTES.md` left untouched (owner's notes, will ride in the harness commit)
|
||||
- **Next pending phase:** `128_sync_label_mobile`
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.
|
||||
|
||||
**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)
|
||||
|
||||
- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
|
||||
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
|
||||
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
|
||||
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
|
||||
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
|
||||
- **Next pending phase:** `128_sync_label_mobile`
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
Everything is green. Final state verified:
|
||||
|
||||
- **`app/rag/importer.py`**: `ImportSummary.changed_paths` field (task-specified docstring, `field(default_factory=list)`); `changed_paths.append` beside **every** `added`/`updated` increment — all 9 sites across the 5 verb blocks (text, PDF, PDF-embedded image, image-context enrichment, standalone image) — and in `_prune` (new `summary: ImportSummary | None = None` param, appended before `session.delete`, only for docs actually deleted past both guards); only the `import_sources` call site updated; `log()` untouched.
|
||||
- **`tests/unit/test_importer.py`**: 8 new tests — mixed 2-added/1-updated/1-pruned run with exact order + `len == added+updated+pruned`, unchanged re-run empty, backfill re-run empty, date-only re-run empty, changed image (added+updated sites), changed PDF (both sites), `_prune` with/without summary, and default-construction `== []`.
|
||||
|
||||
**Results:**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 3005 passed, exit 0, TOTAL 99% (gate >90% ✅; `app/rag/importer.py` 98%, remaining misses pre-existing)
|
||||
- `uv run ruff check .` → All checks passed
|
||||
- `uv run pyright` → 0 errors (fixed two `# pyright: ignore[reportCallIssue]` placements)
|
||||
- Grep audit: 9 increment sites ↔ 9 matching appends ✅
|
||||
|
||||
**Docs:** no doc impact (internal record-tracking only — no commands, config, env vars, or API surface changed; README/AGENTS.md untouched).
|
||||
|
||||
**Decisions:** appended right after each verb if/else (single line, same encounter order as the counter); enrichment path records `(source, image_doc.path)` since its docstring already states it fires the KB-change gate; PNG/PDF fixtures generated in-test with PyMuPDF (house no-binary-fixtures convention).
|
||||
|
||||
**Next pending task:** `02_folder_summaries_affected_mode.md`
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 98%]
|
||||
..................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 123 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5122 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
Everything is green. The unit run shows all dots through [100%] with exit 0.
|
||||
|
||||
**Task 02 complete — report**
|
||||
|
||||
- **Implemented** in `app/rag/folder_summaries.py`:
|
||||
- `affected_folder_keys(changed)` pure helper (next to `folder_of`/`group_by_folder`) — per changed doc: source-root key + `folder_of` prefix walk; fragment paths fold in naturally; empty input → empty set
|
||||
- `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg: `None` = byte-identical full regen (CLI stays on it, verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys)`; manual skip, fail-soft, `on_progress` (total = affected count), and the prune pass are untouched code; docstrings (module + function + progress) updated
|
||||
- **Tests** (`tests/unit/test_folder_summaries.py`, +380 lines): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts (call log + fresh rows), unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs
|
||||
- **Results:** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed**, exit 0; `app/` coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit -q` → exit 0, all green; `uv run ruff check .` → passed; `uv run pyright` → 0 errors
|
||||
- **Docs:** no doc impact (README sync description remains accurate; no new env/API/commands)
|
||||
- **Decisions:** intersection with `candidates` (per task ASSUMPTION) — vanished affected folders are the prune pass's job; no caller combines `changed_docs` + `only_missing` (documented in the docstring)
|
||||
- **Next pending task:** `03_sync_wiring.md` (sync's changed-KB branch passes `summary.changed_paths`)
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
All gates are green. The previous rejection was an infrastructural one (the child run ended without emitting its final report) — no test failures existed; the working tree already contained the completed task. I re-verified everything end-to-end this run: full suite with coverage, the target unit files, and lint + types.
|
||||
|
||||
**Task 02 complete — report**
|
||||
|
||||
- **Implemented** in `app/rag/folder_summaries.py` (unchanged from prior run, verified intact):
|
||||
- `affected_folder_keys(changed)` pure helper — per changed doc: source-root key `(source, "")` + `folder_of` prefix walk (same concept as `group_by_folder` pass 1); fragment paths (`x.pdf#page=…`) fold into parent chain; empty input → empty set
|
||||
- `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg — `None` = byte-identical full regen (CLI verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys(changed_docs))`; manual skip / fail-soft / `on_progress` (total = affected count) / prune pass untouched; module + function docstrings updated
|
||||
- **Tests** (`tests/unit/test_folder_summaries.py`, 14 new): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts, unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs
|
||||
- **Results (this run):** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed, exit 0**; `app/` TOTAL coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit/test_folder_summaries.py tests/unit/test_importer.py -q` → exit 0; `uv run ruff check .` → all passed; `uv run pyright` → 0 errors, 0 warnings
|
||||
- **Docs:** no doc impact (README sync section stays accurate; no new env/API/commands)
|
||||
- **Decisions:** intersection with `candidates` per task ASSUMPTION — vanished affected folders are the prune pass's job; `changed_docs` + `only_missing` documented as never combined by any caller
|
||||
- **Next pending task:** `03_sync_wiring.md` — the sync's changed-KB branch passes `summary.changed_paths`; docstrings + logging
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
Everything is verified green. The previous run completed all of task 03's work but its validation was interrupted by a KeyboardInterrupt; I re-verified every gate from scratch and confirmed the implementation matches the task spec exactly.
|
||||
|
||||
**Task 03 — Sync wiring: complete (verified)**
|
||||
|
||||
- `app/api/sync.py`: changed-KB branch calls `generate_folder_summaries(fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook)`; else-arm (gap probe, `only_missing=True`) byte-identical; log line now `sync: folder_summaries changed=%d stats=%s`
|
||||
- Docstrings updated to the incremental contract: module docstring step-5 + phase-machine paragraph (incl. stale "at least 2 docs" → "at least 1 doc" fix) and `SyncStatus` `summaries_done/total` note (total = affected count on changed KB, missing count on gap-fill)
|
||||
- `detail` dict (12 keys) and `GET /api/sync/status` shape unchanged — asserted in tests
|
||||
- Fakes extended to accept/record `changed_docs=`: `tests/integration/test_sync_api.py` (`FakeFolderSummaries`, `_GatedFolderSummaries`, `_HookThenFail`) + `tests/unit/test_sync_button.py`
|
||||
- 3 new tests in `tests/integration/test_sync_api.py`: change-set wiring + `changed=N` log + detail shape; unchanged-walk gap probe passes `only_missing=True` with `changed_docs=None`; mid-span status `summaries_total == len(affected_folder_keys(...))` — all 3 pass individually (3 passed, 22 deselected)
|
||||
|
||||
**Results:**
|
||||
- `uv run pytest tests/unit -q` → 2326 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, app/ **99%** (sync.py 100%, folder_summaries.py 100%, importer.py 98%) — gate >90% ✅
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
|
||||
|
||||
**Docs:** no doc impact (no setup/commands/config/env/API changes; README never described the regeneration mode, AGENTS.md untouched).
|
||||
|
||||
**Note:** prune-only syncs still take the gap-probe branch (pre-existing `added+updated>0` gate untouched, per task scope). `NOTES.md` (owner's untracked note) left in place.
|
||||
|
||||
**Next pending task:** `04_e2e_incremental_sync.md` (dedicated Playwright E2E in isolation)
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
All green. Final report:
|
||||
|
||||
**Implemented**
|
||||
- New `tests/e2e/test_incremental_folder_summaries.py` — dedicated mock-only Playwright suite (phase 127 task 04), house pattern from `test_sync_summary_visibility.py`: module app on port 8147 (env-overridable), host temp-dir `kind=local` source `incsum/` (4 folders × 2 docs + 1 root doc = 9 files, 5 candidates), autouse truncate + settle-wait cleanup, stale-terminal freshness gate in `_run_sync`.
|
||||
- 3 tests: (1) first full sync → exactly 5 canned rows, terminal keeps 5/5; (2) one `alpha/` file changed → only `alpha` + source root re-stamped (strictly newer `updated_at`), `bravo`/`charlie`/`delta` byte-identical (text AND `updated_at`), terminal keeps `summaries_done == summaries_total == 2`; (3) `bravo/` vanishes → its row pruned by the prune pass, `charlie` + root re-stamped, `alpha`/`delta` byte-identical, terminal 2/2.
|
||||
|
||||
**Gates (all 100% green)**
|
||||
- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed, in isolation (run twice; DB up)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%)
|
||||
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
|
||||
|
||||
**Docs**
|
||||
- README.md: added step 4 to the "Sync from the UI" pipeline (folder summaries: incremental on changed KB, gap-fill on unchanged) — it predated folder summaries entirely (stale since phase 94). AGENTS.md: no impact.
|
||||
|
||||
**Decisions**
|
||||
- "bravo's last doc" read as *the docs whose deletion makes the folder vanish* (0 docs → not a candidate); both `bravo/` files deleted, since a 1-doc folder is still a candidate and would not be pruned. All other assertions match the task verbatim.
|
||||
|
||||
**Next pending task:** none — task 04 is the last task of phase 127.
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
All checks are green and consistent. The documentation is accurate:
|
||||
|
||||
- **README.md** — the step-4 addition from the prior run is intact and matches the implemented behavior (incremental on changed KB via the changed set's folder chains, gap-fill on unchanged walks). The "Updating the documents" CLI section makes no regeneration-scope claims (CLI keeps full regen — consistent).
|
||||
- **AGENTS.md** — no references to the sync pipeline internals; operating rules, test gates, and conventions all still accurate. No changes needed.
|
||||
|
||||
All gates re-verified from scratch this run (the prior a2 attempt was an infrastructural interruption — it died mid-run without a report, leaving no port or DB residue):
|
||||
|
||||
**What was verified/fixed this run**
|
||||
- Confirmed working tree intact from the successful a1 run: `tests/e2e/test_incremental_folder_summaries.py` (3 tests: first-full-sync 5 canned rows @ terminal 5/5; one `alpha/` file → only `alpha`+root re-stamped, siblings byte-identical text AND `updated_at`, terminal 2/2; `bravo/` vanishes → row pruned, `charlie`+root re-stamped, terminal 2/2) plus tasks 01–03 code/tests.
|
||||
- No code changes required — re-ran every gate fresh; found no defects, no leftovers (ports clean, DB truncated).
|
||||
- Doc consistency check: README step-4 accurate; AGENTS.md untouched.
|
||||
|
||||
**Results (this run)**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%)
|
||||
- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed in isolation (run twice)
|
||||
- `uv run ruff check .` → all passed; `uv run pyright` → 0 errors
|
||||
|
||||
**Notable decision** (unchanged from a1): "bravo's last doc" = the docs whose deletion makes the folder vanish (both deleted; a 1-doc folder is still a candidate and wouldn't prune).
|
||||
|
||||
**Next pending task:** none — task 04 is the last task of phase 127.
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
...................................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 262 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 139 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 374 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 244 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 400 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5133 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -65,6 +65,7 @@ client-side router. Direct URLs deep-link to the matching view.
|
||||
| **Tuning** | `/tuning.html` | Admin only |
|
||||
| **Saved chats** | `/history.html` | Admin only |
|
||||
| **Access tokens** | `/tokens.html` | Admin only |
|
||||
| **Theme** | `/theme.html` | Admin only |
|
||||
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
|
||||
| **Login** | `/login.html` | Everyone |
|
||||
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
|
||||
@@ -207,6 +208,10 @@ reflect the new files.
|
||||
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
|
||||
bmp`) are indexed as documents too — see
|
||||
[Image support](#image-support).
|
||||
- A run that changed the KB also regenerates the **KB overview** and the
|
||||
**folder summaries** (the drill-down catalog's stored per-folder
|
||||
descriptions) with the `lite` model — best-effort: a summary failure
|
||||
never fails the import.
|
||||
|
||||
### Git-based sources
|
||||
|
||||
@@ -262,6 +267,12 @@ whole git-source refresh in one click:
|
||||
2. **re-import with prune** — files deleted upstream leave the index.
|
||||
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
|
||||
chat turn injects) — only when the import changed the knowledge base.
|
||||
4. **refresh the folder summaries** (the drill-down catalog's stored
|
||||
per-folder descriptions) — a changed knowledge base rewrites ONLY the
|
||||
folders the changed files touch (each file's folder chain up to its
|
||||
source root — phase 127: one changed file re-stamps a handful of
|
||||
rows, not the whole table); an unchanged re-sync instead gap-fills
|
||||
any missing rows (`only_missing`).
|
||||
|
||||
- States: the button shows **Syncing…** with a spinner while polling
|
||||
`GET /api/sync/status` every 2 s. No client-side timeout — a clone +
|
||||
@@ -288,13 +299,16 @@ To hide it, set `BOR_STREAM_THINKING=0`.
|
||||
Retrieval only puts the top documents in context. When an answer depends on
|
||||
a file a note *references* ("the exact JSON shape is in
|
||||
`example-record-file.json`"), the model can extend its own context with
|
||||
three server-side tools — on **grounded** (high-relevance) turns only:
|
||||
server-side tools — on **grounded** (high-relevance) turns only:
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source |
|
||||
| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string |
|
||||
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
|
||||
| **`read(path)`** | Appends one indexed document's text to context by its combined `source/path`; very large documents are truncated at `BOR_READ_MAX_CHARS` with a notice that points at `grep` to find the rest |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
|
||||
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
|
||||
|
||||
Each call is executed against Postgres only (no extra LLM round trip) and
|
||||
streamed as an SSE `tool` frame. In the chat, each call shows a transient
|
||||
@@ -361,12 +375,14 @@ docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
|
||||
|
||||
---
|
||||
|
||||
## Document summaries (non-markdown)
|
||||
## Document summaries
|
||||
|
||||
Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
|
||||
retrieval can miss exactly the documents that are all configuration. At
|
||||
import time, every **non-markdown** A9 document is summarized by the aipi
|
||||
`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`):
|
||||
import time, **every** new or changed document gets a `lite`-model summary —
|
||||
**markdown included**: the summary is a natural-language retrieval target
|
||||
and the seed text for the onboarding suggestion blocks (phase 118 retired
|
||||
the old "markdown is already natural language" exclusion):
|
||||
|
||||
* The summary is stored on `documents.summary` **and** indexed as one extra
|
||||
embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
|
||||
@@ -452,6 +468,30 @@ image:
|
||||
A question image is **never indexed as a document** — it is turn-local
|
||||
storage, not a source.
|
||||
|
||||
### Answer images (phase 124)
|
||||
|
||||
A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
|
||||
on, the grounded chat model may show a standalone image document as part
|
||||
of its answer:
|
||||
|
||||
- **The model decides.** The agent is offered a fourth tool,
|
||||
`show_image(path)`, and calls it only when the image itself contains
|
||||
the answer (a diagram, screenshot, or photo of the thing asked about) —
|
||||
never for mere topical relevance. At most **one** image per answer
|
||||
(enforced server-side); the text answer is always given in full — the
|
||||
image supplements, never replaces.
|
||||
- **Rendered as an answer figure.** A shown image appears as a
|
||||
first-class figure below the answer text and above the source chips —
|
||||
on the live turn, after a page reload, and on the shared chat page; a
|
||||
failed load collapses the figure (never a broken-image icon).
|
||||
- **Image chips on zero-read turns.** A retrieved image document in the
|
||||
suggested tier is auto-cited: its source chip (with figure) appears
|
||||
even when the model never `read`s it — the model's seed block already
|
||||
carries the image's full description, so a real model has no reason to
|
||||
read it (this applies with `BOR_IMAGES=1`, any `BOR_ANSWER_IMAGES`).
|
||||
- **Off = byte-identical.** No fourth tool offered, no figure, no
|
||||
`answer_image` key in any SSE frame or saved record.
|
||||
|
||||
---
|
||||
|
||||
## Caching & deploys
|
||||
@@ -515,7 +555,7 @@ to `.env` and adjust.
|
||||
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
|
||||
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
|
||||
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
|
||||
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview |
|
||||
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
|
||||
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
|
||||
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
|
||||
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
|
||||
@@ -578,7 +618,7 @@ to `.env` and adjust.
|
||||
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
||||
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
||||
|
||||
### Image support (phases 122/123)
|
||||
### Image support (phases 122/123/124)
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
@@ -587,6 +627,7 @@ to `.env` and adjust.
|
||||
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
|
||||
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
|
||||
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
|
||||
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |
|
||||
|
||||
### Docs push
|
||||
|
||||
@@ -615,10 +656,11 @@ Every identity string is an env var: `BOR_APP_NAME` (display name),
|
||||
applied by `assets/brand.js` at boot.
|
||||
|
||||
The colors are set from the admin **Theme tab** (`/theme.html`,
|
||||
admin-only — phase 91): the 8 identity colors plus the three strings
|
||||
above are edited with text fields and color pickers, persisted in the
|
||||
`ui_settings` table, and injected into every served page as an inline
|
||||
`<style>` BEFORE first paint (no red flash, no pop-in). The old
|
||||
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
|
||||
the 8 semantic state colors — success/error/accent families) and the
|
||||
three strings above are edited with text fields and color pickers,
|
||||
persisted in the `ui_settings` table, and injected into every served page
|
||||
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The old
|
||||
CSS-file theming is retired — a leftover `BOR_THEME` line in a
|
||||
deployment's `.env` is simply ignored. Leave everything unset and the
|
||||
app renders the built-in dark-tech palette byte-identically.
|
||||
|
||||
@@ -1,9 +1 @@
|
||||
# TODO
|
||||
|
||||
- [ ] On sync brain of reese rewrites all the summaries for every directory. So even if just one file changed this can result in 700+ directory summary rewrites in production which can take a very long time.
|
||||
- [ ] The `Syncing… name/of/file/here (2/3)` doesn't show up on mobile, it needs to
|
||||
- [ ] The `Syncing… name/of/file/here (2/3)` needs to show what chunk we're on out of how many
|
||||
- [ ] The chat output is getting messy. I think it's because the `source-chip`, `source-image`, `tune-btn`, `retry-btn`, and `save-as-doc-btn` end up in the same div which causes them to push each other in weird ways. Those three buttons should be in a button cluster in the bottom right. We don't need to display the `source-chip` for images, just the image itself is fine. `source-chips` should show up before images in their own cluster, then images in their own cluster to keep things organized.
|
||||
- [ ] The agent is struggling with huge documents. It can search for the text it wants and it finds the text, including line number, but then it gets stuck saying "However, read takes a path, not a line number." and can't figure out how to read that part of the document. Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck. I have a huge test pdf (the DND player's handbook) in test_sources called "pdf_tests.tar" that you can use for testing. It's a massive pdf. The question the agent got stuck on was "What are the components of the find traps spell?" which is toward the end of the players handbook so it will need to search and read specific lines. I've been setting BOR_READ_MAX_CHARS to 24000 to keep things relatively quick so the agent realizes quickly its answer isn't in the first few pages.
|
||||
- [ ] Uploading an image in the chat provides no feedback to the user. The user clicks the paperclick, selects and image, presses "send" and then nothing happens. No loading bar, no "uploading image", no indication. This results in the user spamming "send" over and over because they think it's frozen. We need to visually show the user's image is uploading.
|
||||
- [ ] test_sources/hnb_2026_guide.tar.gz contains a pdf that is being read as binary. Figure out why. It contains images, tables, and text but none of that is being extracted correctly. Instead, the raw xml is being embedded and the preview shows the entire document as lines like `<illustrator:StartupProfile>Web</illustrator:StartupProfile>`. This project should support this kind of PDF and should be able to extract it correctly, even if that means screenshotting the pages and reading each one as an image to preserve formatting when text/image/table extraction isn't working.
|
||||
+52
-23
@@ -56,9 +56,16 @@ decisions):
|
||||
(phase 31 trigger, best-effort inside) — and ``generate_
|
||||
folder_summaries`` (phase 94, task 02) regenerates the stored folder
|
||||
summaries (the drill-down ``ls``'s per-level descriptions): its gate
|
||||
is the same change trigger (full regeneration) **plus**, on an
|
||||
unchanged re-sync, a GAP probe — a candidate folder (at least 2
|
||||
docs) with no stored row (``missing_folder_summaries``, phase 96,
|
||||
is the same change trigger — INCREMENTAL since phase 127 (task 03):
|
||||
the run's changed documents (``ImportSummary.changed_paths``, the
|
||||
import's added/updated/pruned ``(source, path)`` set) are passed as
|
||||
``changed_docs=`` and ONLY their folder chains (each document's
|
||||
folder up to its source root — ``affected_folder_keys``) are
|
||||
rewritten, so a one-file change burns a handful of rows, not the
|
||||
whole table (the ``scripts/import_docs.py`` CLI keeps the
|
||||
byte-identical full regeneration — it passes no change set) **plus**,
|
||||
on an unchanged re-sync, a GAP probe — a candidate folder (at least
|
||||
1 doc) with no stored row (``missing_folder_summaries``, phase 96,
|
||||
task 03): a gap fills ONLY the missing rows (``only_missing=True``;
|
||||
the old table-empty first-sync trigger is subsumed exactly — an
|
||||
empty table leaves every candidate missing), a complete table burns
|
||||
@@ -91,13 +98,16 @@ reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
|
||||
(set immediately before ``import_sources``), ``"overview"`` (set
|
||||
before ``regenerate_overview`` — a changed KB only), and
|
||||
``"summaries"`` (set before a folder-summary generation — the
|
||||
changed-KB full regeneration or the unchanged-walk gap-fill; the
|
||||
no-gap skip sets NO phase and the run stays ``"import"``). While in
|
||||
changed-KB incremental regeneration (phase 127, task 03 — the changed
|
||||
set's folder keys only) or the unchanged-walk gap-fill; the no-gap
|
||||
skip sets NO phase and the run stays ``"import"``). While in
|
||||
``"summaries"`` the generator's ``on_progress`` hook fills
|
||||
``current_summary`` (the ``source`` / ``source/folder_path`` being
|
||||
summarized — the bare source name for the source-root row) plus
|
||||
``summaries_done`` / ``summaries_total`` — the per-folder position
|
||||
through the long summary span where the file count sits still.
|
||||
through the long summary span where the file count sits still (the
|
||||
hook's ``total`` is the run's key count — the AFFECTED count on a
|
||||
changed KB, the missing count on a gap-fill).
|
||||
Terminal states (success AND failed) clear ``phase`` +
|
||||
``current_summary`` but keep the run's final ``summaries_done`` /
|
||||
``summaries_total`` (the phase-64 keep-final-counts convention).
|
||||
@@ -161,9 +171,13 @@ class SyncStatus:
|
||||
``"summaries"`` phase, ``current_summary`` is the source /
|
||||
``source/folder_path`` the generator is summarizing right now and
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
progress hook's position. Terminal states clear ``phase`` +
|
||||
``current_summary`` but keep the run's final summary counts (the
|
||||
phase-64 keep-final-counts convention).
|
||||
progress hook's position — the hook's ``total`` is the run's key
|
||||
count: the AFFECTED count on a changed KB (phase 127, task 03 —
|
||||
the changed set's folder keys, NOT the table's candidate count) or
|
||||
the missing count on an unchanged-walk gap-fill (as before).
|
||||
Terminal states clear ``phase`` + ``current_summary`` but keep the
|
||||
run's final summary counts (the phase-64 keep-final-counts
|
||||
convention).
|
||||
"""
|
||||
|
||||
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||
@@ -358,13 +372,19 @@ async def _run_sync() -> None:
|
||||
if summary.added + summary.updated > 0:
|
||||
_status.phase = "overview" # phase 98 (task 01)
|
||||
overview = await regenerate_overview(llm)
|
||||
# Phase 94 (task 02), phase 96 (task 03): the folder
|
||||
# summaries — the drill-down ls's per-level descriptions. A
|
||||
# changed KB (added + updated > 0) is a full regeneration
|
||||
# (today's behavior, byte-identical); an unchanged re-sync
|
||||
# takes the GAP probe instead of the old table-empty one — a
|
||||
# candidate folder (at least 1 doc — every existing folder)
|
||||
# with no stored row: a gap
|
||||
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
|
||||
# the folder summaries — the drill-down ls's per-level
|
||||
# descriptions. A changed KB (added + updated > 0) is an
|
||||
# INCREMENTAL regeneration (phase 127, task 03): the run's
|
||||
# changed documents (summary.changed_paths — the import's
|
||||
# added/updated/pruned set) are passed as changed_docs= and
|
||||
# ONLY their folder chains (each document's folder up to its
|
||||
# source root) are rewritten — a one-file change burns a
|
||||
# handful of rows, not the whole table (the CLI keeps the
|
||||
# byte-identical full regeneration — it passes no change set);
|
||||
# an unchanged re-sync takes the GAP probe instead of the old
|
||||
# table-empty one — a candidate folder (at least 1 doc — every
|
||||
# existing folder) with no stored row: a gap
|
||||
# fills ONLY the missing rows (only_missing=True — the
|
||||
# subsumed table-empty first-sync trigger included, where
|
||||
# every candidate is missing), a complete table burns zero
|
||||
@@ -378,8 +398,9 @@ async def _run_sync() -> None:
|
||||
# Phase 98 (task 01): the folder-summary progress hook — the
|
||||
# closure-captures-``_status`` convention the file's ``_hook``
|
||||
# above already uses; the generator's ``total`` (the loop-start
|
||||
# candidate count) and the bare source name for the root row
|
||||
# come straight from the hook's arguments (D5's shape).
|
||||
# key count — the affected count on a changed KB, the missing
|
||||
# count on a gap-fill) and the bare source name for the root
|
||||
# row come straight from the hook's arguments (D5's shape).
|
||||
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
|
||||
_status.current_summary = (
|
||||
source if folder_path == "" else f"{source}/{folder_path}"
|
||||
@@ -390,15 +411,23 @@ async def _run_sync() -> None:
|
||||
fs_db = SessionLocal()
|
||||
try:
|
||||
if summary.added + summary.updated > 0:
|
||||
# Changed KB: full regeneration — the summaries phase
|
||||
# starts now, and the hook reports the per-folder
|
||||
# position through the (long) span.
|
||||
# Changed KB: incremental regeneration (phase 127,
|
||||
# task 03) — ONLY the changed set's folder keys
|
||||
# (changed_docs=summary.changed_paths: each document's
|
||||
# folder chain up to its source root); the summaries
|
||||
# phase starts now, and the hook reports the per-folder
|
||||
# position through the (long) span (its total is the
|
||||
# affected count).
|
||||
_status.phase = "summaries"
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, on_progress=_summary_hook
|
||||
fs_db, llm, changed_docs=summary.changed_paths,
|
||||
on_progress=_summary_hook,
|
||||
)
|
||||
fs_db.commit()
|
||||
logger.info("sync: folder_summaries stats=%s", folder_stats)
|
||||
logger.info(
|
||||
"sync: folder_summaries changed=%d stats=%s",
|
||||
len(summary.changed_paths), folder_stats,
|
||||
)
|
||||
else:
|
||||
missing = missing_folder_summaries(fs_db)
|
||||
if missing:
|
||||
|
||||
@@ -59,11 +59,22 @@ the stored keys — one concept, as with :func:`group_by_folder`), and
|
||||
:func:`generate_folder_summaries`'s ``only_missing=True`` fills
|
||||
EXACTLY those on the next sync — existing rows stay
|
||||
byte-identical (text AND ``updated_at``), the prune pass still runs.
|
||||
|
||||
Restricted regeneration (phase 127, task 02): a changed-KB sync no
|
||||
longer rewrites EVERY folder — the sync path passes the run's changed
|
||||
documents (``ImportSummary.changed_paths``) as ``changed_docs=`` and
|
||||
the generator then rewrites ONLY the candidate folders the change
|
||||
touches — each document's folder chain up to the source root
|
||||
(:func:`affected_folder_keys`, the SAME prefix walk
|
||||
:func:`group_by_folder` pass 1 uses — one concept); every other
|
||||
stored row keeps its text AND ``updated_at``. The
|
||||
``scripts/import_docs.py`` CLI keeps the byte-identical full
|
||||
regeneration (``changed_docs=None``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Callable, Sequence
|
||||
from collections.abc import Callable, Iterable, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from typing import Protocol
|
||||
|
||||
@@ -209,6 +220,32 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
|
||||
return groups
|
||||
|
||||
|
||||
def affected_folder_keys(
|
||||
changed: Iterable[tuple[str, str]],
|
||||
) -> set[tuple[str, str]]:
|
||||
"""The candidate folder keys a changed-document set invalidates.
|
||||
|
||||
For each ``(source, path)``: the source-root key ``(source, "")``
|
||||
plus every folder prefix of *path* — :func:`folder_of` walked from
|
||||
the document's nearest folder up to ``""`` — the SAME prefix walk
|
||||
:func:`group_by_folder` pass 1 uses (one concept): a doc under
|
||||
``a/b/`` affects ``a/b``, ``a``, and the source root. A
|
||||
root-level file (``folder_of(path) == ""``) affects ONLY the
|
||||
source root. Embedded-image fragment paths (``x.pdf#page=…#img=…``,
|
||||
phase 126) fold into their parent's folder chain with no special
|
||||
case — ``folder_of`` never sees a ``/`` inside the fragment.
|
||||
Returns a set; the generator sorts it (deterministic order).
|
||||
"""
|
||||
keys: set[tuple[str, str]] = set()
|
||||
for source, path in changed:
|
||||
keys.add((source, ""))
|
||||
folder = folder_of(path)
|
||||
while folder:
|
||||
keys.add((source, folder))
|
||||
folder = folder_of(folder)
|
||||
return keys
|
||||
|
||||
|
||||
def _first_summary_line(summary: str | None) -> str:
|
||||
"""First line of a stored summary, stripped; ``''`` when absent.
|
||||
|
||||
@@ -406,13 +443,34 @@ async def generate_folder_summaries(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
"""Regenerate the stored folder summaries for the current catalogue.
|
||||
|
||||
The sync-path orchestrator (phase 94, task 02 calls it
|
||||
change-gated, like the KB overview; ``--limit`` debug runs pass
|
||||
``skip=True``). Steps, in order:
|
||||
``skip=True``).
|
||||
|
||||
Modes (phase 127, task 02 — the *changed_docs* kwarg):
|
||||
``changed_docs=None`` (the DEFAULT) is the FULL regeneration —
|
||||
every candidate folder, byte-identical to the pre-127 behavior;
|
||||
the ``scripts/import_docs.py`` CLI stays on it. A GIVEN change set
|
||||
(the sync path passes ``ImportSummary.changed_paths``) restricts
|
||||
the generation to the CANDIDATE folders the changed documents
|
||||
touch — :func:`affected_folder_keys`: a document's change
|
||||
invalidates exactly its folder chain + source root, the folders
|
||||
whose subtree summary text CAN change; every other stored row
|
||||
keeps its summary text AND ``updated_at`` (an unaffected folder's
|
||||
documents — and hence its summary — are untouched). An affected
|
||||
folder with no stored row (a gap in the chain) is a candidate and
|
||||
is generated; an affected folder that dropped below the minimum
|
||||
is the prune pass's job (step 4), not a generation's. The two
|
||||
restriction kwargs are ORTHOGONAL and no caller combines them:
|
||||
``only_missing=True`` (the unchanged-sync gap fill, phase 96)
|
||||
keeps its exact pre-127 semantics.
|
||||
|
||||
Steps, in order:
|
||||
|
||||
1. ``skip=True`` → a no-op: the zero stats dict is returned, the
|
||||
LLM is never called, and no rows are touched.
|
||||
@@ -461,10 +519,12 @@ async def generate_folder_summaries(
|
||||
BEFORE the attempt — so an instant manual skip and a failed folder
|
||||
BOTH advance the counter (the UI's position moves on either), and
|
||||
``total`` is ``len(keys)`` at loop start (under ``only_missing
|
||||
=True`` that is the MISSING count, not the full candidate count).
|
||||
``None`` — the ``scripts/import_docs.py`` CLI path — is a
|
||||
zero-cost no-op (guarded at the call site; nothing observable
|
||||
changes without it).
|
||||
=True`` that is the MISSING count, not the full candidate count;
|
||||
under a ``changed_docs`` restriction that is the AFFECTED count —
|
||||
the sync status's ``summaries_total`` reports it through this
|
||||
existing hook). ``None`` — the ``scripts/import_docs.py`` CLI
|
||||
path — is a zero-cost no-op (guarded at the call site; nothing
|
||||
observable changes without it).
|
||||
|
||||
Only flushes — the CALLER commits (the phase-53
|
||||
``bump_sources_version`` convention: the sync path owns the
|
||||
@@ -490,6 +550,15 @@ async def generate_folder_summaries(
|
||||
for row in db.execute(select(FolderSummary)).scalars()
|
||||
}
|
||||
keys = sorted(candidates)
|
||||
if changed_docs is not None:
|
||||
# Phase 127 (task 02): restricted regeneration — exactly the
|
||||
# CANDIDATE folders the change set touches (each document's
|
||||
# folder chain + source root). Unaffected candidates keep
|
||||
# their rows byte-identical (text AND ``updated_at``); a gap
|
||||
# in an affected folder is a candidate and gets generated;
|
||||
# an affected folder that lost its last document is no
|
||||
# candidate — the prune pass below drops its row.
|
||||
keys = sorted(set(candidates) & affected_folder_keys(changed_docs))
|
||||
if only_missing:
|
||||
keys = [key for key in keys if key not in existing]
|
||||
|
||||
|
||||
+23
-1
@@ -244,6 +244,12 @@ class ImportSummary:
|
||||
#: Fail-soft: the sync continues, this counter + the warning line
|
||||
#: are the signal.
|
||||
images_failed: int = 0
|
||||
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
|
||||
#: tuples in encounter order (added/updated in walk order, pruned in
|
||||
#: _prune order). Drives the sync's incremental folder-summary
|
||||
#: regeneration (phase 127): len(changed_paths) == added + updated +
|
||||
#: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
|
||||
changed_paths: list[tuple[str, str]] = field(default_factory=list)
|
||||
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
|
||||
formats: dict[str, int] = field(default_factory=dict)
|
||||
|
||||
@@ -590,7 +596,8 @@ async def import_sources(
|
||||
logger.warning("import: --prune ignored because --limit was given")
|
||||
else:
|
||||
summary.pruned = _prune(
|
||||
session, source_names, seen, images=llm.settings.images
|
||||
session, source_names, seen, images=llm.settings.images,
|
||||
summary=summary,
|
||||
)
|
||||
summary.embed_batches = llm.embed_batches
|
||||
summary.log()
|
||||
@@ -794,6 +801,7 @@ async def _index_file(
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.changed_paths.append((source, rel))
|
||||
summary.chunks += chunks_n
|
||||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||||
|
||||
@@ -1068,6 +1076,7 @@ async def _index_pdf_file(
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.changed_paths.append((source, rel))
|
||||
summary.chunks += chunks_n
|
||||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||||
|
||||
@@ -1328,6 +1337,7 @@ async def _index_one_embedded_image(
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.changed_paths.append((source, image_rel))
|
||||
summary.chunks += chunks_n
|
||||
logger.info(
|
||||
"import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
|
||||
@@ -1728,6 +1738,7 @@ async def _enrich_pending_images(
|
||||
)
|
||||
continue
|
||||
summary.updated += 1
|
||||
summary.changed_paths.append((source, image_doc.path))
|
||||
summary.chunks += chunks_n
|
||||
logger.info(
|
||||
"import: image-context-enriched source=%s path=%s chunks=%d",
|
||||
@@ -1994,6 +2005,7 @@ async def _index_image_file(
|
||||
summary.added += 1
|
||||
else:
|
||||
summary.updated += 1
|
||||
summary.changed_paths.append((source, rel))
|
||||
summary.chunks += chunks_n
|
||||
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
|
||||
|
||||
@@ -2017,6 +2029,7 @@ def _prune(
|
||||
source_names: set[str],
|
||||
seen: set[tuple[str, str]],
|
||||
images: bool = False,
|
||||
summary: ImportSummary | None = None,
|
||||
) -> int:
|
||||
"""Delete documents of *source_names* whose file is no longer in *seen*.
|
||||
|
||||
@@ -2040,6 +2053,13 @@ def _prune(
|
||||
toggle states (toggle off, the phase-122 guard already skips every
|
||||
``is_image`` doc; this one is the toggle-on half of the same
|
||||
"invisible, not deleted" contract).
|
||||
|
||||
*summary* (phase 127, task 01): when given, every document that is
|
||||
ACTUALLY deleted (i.e. not skipped by either guard above) records
|
||||
its ``(doc.source, doc.path)`` on ``summary.changed_paths`` —
|
||||
appended BEFORE the ``session.delete`` — so the sync's incremental
|
||||
folder-summary regeneration (phase 127) sees the pruned docs' folders
|
||||
too. Callers passing nothing keep the pre-phase count-only contract.
|
||||
"""
|
||||
if not source_names:
|
||||
return 0
|
||||
@@ -2059,6 +2079,8 @@ def _prune(
|
||||
# invisible to the walk, not deleted.
|
||||
continue
|
||||
_delete_image_copy(doc.image_path)
|
||||
if summary is not None:
|
||||
summary.changed_paths.append((doc.source, doc.path))
|
||||
session.delete(doc)
|
||||
pruned += 1
|
||||
logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
|
||||
|
||||
@@ -0,0 +1,619 @@
|
||||
"""Phase 127 task 04 E2E (Playwright, mock-only): a one-file change
|
||||
rewrites ONLY that folder's summary chain.
|
||||
|
||||
The dedicated story suite for ``127_sync_incremental_folder_summaries``
|
||||
(A16 — one Playwright file per phase, run in isolation): the owner's
|
||||
report was that "on sync brain of reese rewrites all the summaries for
|
||||
every directory — so even if just one file changed this can result in
|
||||
700+ directory summary rewrites in production which can take a very
|
||||
long time" (``TODO.md`` L3). The fix under test: a KB-changing sync
|
||||
passes the import's changed-document set
|
||||
(``ImportSummary.changed_paths``) into
|
||||
``generate_folder_summaries(changed_docs=…)``, so ONLY the folders a
|
||||
changed document touches — each document's folder chain up to its
|
||||
source root (``affected_folder_keys``) — are re-stamped; every other
|
||||
stored row keeps its summary text AND ``updated_at`` byte-identical.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic mock (the phase-94 ``FOLDER_SUMMARY_MODE`` canned
|
||||
``Fixture folder summary for <folder>.`` lines): the stored rows are a
|
||||
pure function of the request, so the byte-identical assertions are
|
||||
exact only against the mock.
|
||||
|
||||
No slow-LLM proxy: the assertions are on the POST-terminal DB state
|
||||
(the rows' ``summary`` AND ``updated_at``) plus the terminal status
|
||||
(the keep-final-counts convention — the terminal keeps
|
||||
``summaries_done == summaries_total == <affected count>``), never on
|
||||
in-flight sampling, so the fast mock is dense enough by construction.
|
||||
|
||||
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
|
||||
the same host) registered as a ``kind=local`` source (the
|
||||
``test_local_directory_sources.py`` / phase-94 API-registration
|
||||
pattern; no git anywhere): ``incsum/`` with FOUR ≥ 2-doc folders —
|
||||
``alpha/``, ``bravo/``, ``charlie/``, ``delta/`` (2 docs each) — plus
|
||||
ONE root-level doc (9 documents total). The generator's candidate set
|
||||
is therefore exactly 5: the source root (``folder_path ""``) + the
|
||||
four folders (the recursive-subtree ≥ 1 rule).
|
||||
|
||||
Autouse cleanup (the phase-96 pattern): before each test wait for no
|
||||
running sync on the module app, then truncate the shared registry + KB
|
||||
tables (documents/chunks/query_log/steering_notes/kb_overview/
|
||||
git_sources/folder_summaries) — each test seeds its own source and
|
||||
runs its own sync(s); after each test the same truncate. (The temp
|
||||
tree itself persists for the module; the tests' disk mutations are
|
||||
ordered by test definition and each test re-syncs the state it finds.)
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_first_sync_generates_all_rows`` — a first (full) sync settles
|
||||
``success``; the ``folder_summaries`` table holds EXACTLY the 5
|
||||
rows (source root + the four folders) with the mock's canned text
|
||||
naming each folder, each ``updated_at`` non-null, and the terminal
|
||||
keeps ``summaries_done == summaries_total == 5`` (the full
|
||||
regeneration — a changed KB with no prior rows, the pre-127
|
||||
behavior preserved end to end).
|
||||
2. ``test_one_file_change_restamps_only_that_folders_chain`` — re-seed
|
||||
(autouse truncated) + sync #1, capture every row's
|
||||
``(summary, updated_at)``; change ONE doc inside ``alpha/`` and
|
||||
sync #2 settles ``success``: the ``alpha`` AND the source-root rows
|
||||
carry a NEW ``updated_at`` (strictly after the captured stamp) with
|
||||
the canned text, while the ``bravo``/``charlie``/``delta`` rows are
|
||||
BYTE-IDENTICAL (same ``summary`` AND same ``updated_at``); the
|
||||
terminal keeps ``summaries_done == summaries_total == 2`` (the
|
||||
AFFECTED count — alpha + root, NOT the table's candidate count —
|
||||
the keep-final-counts convention) and the detail shows the
|
||||
single-file change (1 updated, 8 unchanged, 0 added/pruned).
|
||||
3. ``test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_
|
||||
folder`` — re-seed + sync #1, capture; delete ``bravo/``'s last
|
||||
docs (the folder VANISHES — 0 documents, no longer a candidate) and
|
||||
change one doc in ``charlie/``; the sync settles ``success``: the
|
||||
``bravo`` row is PRUNED (gone — the prune pass owns vanished
|
||||
folders' rows), the source-root AND ``charlie`` rows are re-stamped
|
||||
(new ``updated_at`` + canned text), the ``alpha``/``delta`` rows
|
||||
are byte-identical, and the terminal keeps
|
||||
``summaries_done == summaries_total == 2`` (the affected CANDIDATE
|
||||
count — charlie + root; bravo is no longer a candidate).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module's app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_INCSUM", "8147"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture constants (deterministic, token-controlled)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "incsum"
|
||||
|
||||
#: The four ≥ 2-doc folders (path order).
|
||||
FOLDER_A = "alpha"
|
||||
FOLDER_B = "bravo"
|
||||
FOLDER_C = "charlie"
|
||||
FOLDER_D = "delta"
|
||||
FOLDERS = (FOLDER_A, FOLDER_B, FOLDER_C, FOLDER_D)
|
||||
|
||||
ROOT_NOTE = "root-note.md" # the one root-level doc
|
||||
|
||||
N_FILES = 9 # four folders × 2 docs + the root-level doc
|
||||
N_CANDIDATES = 5 # the source root + the four folders (the ≥ 1-doc rule)
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
|
||||
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
|
||||
#: tail: ``<source>`` for the root, ``<source>/<folder>`` for a folder).
|
||||
SUMMARY_FOR = "Fixture folder summary for {}."
|
||||
EXPECTED: dict[str, str] = {
|
||||
"": SUMMARY_FOR.format(SOURCE),
|
||||
FOLDER_A: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_A}"),
|
||||
FOLDER_B: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_B}"),
|
||||
FOLDER_C: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_C}"),
|
||||
FOLDER_D: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_D}"),
|
||||
}
|
||||
|
||||
#: The single-file change (test 2) — a revised body whose hash differs
|
||||
#: from the fixture's (the importer's sha256 delta reports ``updated``)
|
||||
#: and which no other fixture text contains.
|
||||
ALPHA_A_REVISED = (
|
||||
"Incsum alpha fixture note A, REVISED: the only file test 2 changed "
|
||||
"— the incremental regeneration must touch exactly this file's chain "
|
||||
"(alpha + the source root), and nothing else. (RESE- INCSUM-01)"
|
||||
)
|
||||
|
||||
#: The charlie change (test 3) — the same shape on the other side.
|
||||
CHARLIE_A_REVISED = (
|
||||
"Incsum charlie fixture note A, REVISED: the one changed file of "
|
||||
"test 3's sync, while bravo vanishes entirely. (RESE- INCSUM-02)"
|
||||
)
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def incsum_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
server runs on the same host, so the paths are visible to it. The
|
||||
directory NAME is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_incsum")
|
||||
src = root / SOURCE
|
||||
for folder in FOLDERS:
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter in ("a", "b"):
|
||||
(src / folder / f"{folder}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Incsum {folder.title()} {letter.upper()}",
|
||||
f"Incsum {folder} fixture note {letter.upper()}: covers "
|
||||
f"topic {letter.upper()} of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(src / ROOT_NOTE).write_text(
|
||||
_md(
|
||||
"Incsum Root Note",
|
||||
f"This file sits directly under the {SOURCE} source, not in "
|
||||
"any folder.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == N_FILES
|
||||
return src
|
||||
|
||||
|
||||
def _app_env(
|
||||
base_url: str, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern, cf.
|
||||
``test_kb_tree.py`` / ``test_sync_summary_visibility.py``): the
|
||||
``base_url`` is the mock LLM directly (the fast leg is the only
|
||||
leg — see the module docstring), ``BOR_GIT_SOURCES`` forced empty
|
||||
(the sync source is this suite's own DB-registered local row), and
|
||||
the leak-guarded code defaults."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): this suite never
|
||||
# asks the chat model anything — the gate is never on a path.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int,
|
||||
incsum_dir: Path,
|
||||
tmp_path_factory: pytest.TempPathFactory,
|
||||
) -> Iterator[str]:
|
||||
"""The app under test (all three tests): ``BOR_LLM_BASE_URL`` is
|
||||
the mock directly — the canned ``FOLDER_SUMMARY_MODE`` lines are
|
||||
byte-stable there, and the byte-identity assertions key on that
|
||||
exact text (no slow proxy: the assertions are post-terminal, see
|
||||
the module docstring). ``incsum_dir`` is a dependency only for the
|
||||
fixture ordering (the temp tree exists before the app boots — the
|
||||
sync reads it)."""
|
||||
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the candidate count and the counts the syncs and the
|
||||
folder-rows assertions pin exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to
|
||||
be non-running BEFORE the truncate. Own admin session (the
|
||||
endpoint is admin-only) — usually a no-op: the tests settle only
|
||||
after their run's terminal state."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation (the phase-96 cleanup pattern): the
|
||||
shared E2E Postgres is truncated before AND after every test —
|
||||
each test seeds its own source and runs its own sync(s), so the
|
||||
KB rows + folder_summaries rows of the temp source are exactly
|
||||
this test's own. The pre-wait keeps a leaked run from the
|
||||
previous test (the module app is shared) from racing the
|
||||
truncate."""
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_source(base_url: str, path: str) -> None:
|
||||
"""Register the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` / phase-94 pattern — a plain
|
||||
directory is an API/DB-only operation)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": path}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
|
||||
|
||||
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
|
||||
run reaches a terminal state (the ``test_local_directory_sources``
|
||||
pattern, over plain httpx — with the stale-terminal freshness gate:
|
||||
the module app is shared, so the PREVIOUS run's terminal lives in
|
||||
the app's memory until this run supersedes it; a terminal is
|
||||
accepted only after a running tick was observed or its
|
||||
``started_at`` post-dates this POST)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
t0 = time.time()
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
saw_running = False
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed") and (
|
||||
saw_running
|
||||
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
|
||||
):
|
||||
return body
|
||||
time.sleep(0.1)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the
|
||||
server stamps it in UTC; this process runs on the same host —
|
||||
same clock, the phase-64 recorder's freshness convention)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, bool, datetime]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, manually_edited, updated_at)}``
|
||||
(``""`` = the source root) — the test process's direct DB access
|
||||
(the E2E's other established lens)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.manually_edited,
|
||||
FolderSummary.updated_at,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {
|
||||
folder: (summary, manually, updated)
|
||||
for folder, summary, manually, updated in rows
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The first sync (a changed KB with no prior rows) is the FULL
|
||||
# regeneration: exactly the 5 candidate rows, canned text, the
|
||||
# terminal keeps the full candidate count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_first_sync_generates_all_rows(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""A first sync of the temp source settles ``success`` with all 9
|
||||
docs added; the ``folder_summaries`` table holds EXACTLY the 5
|
||||
candidate rows (the source root + the four folders — the
|
||||
recursive-subtree ≥ 1 rule) with the mock's canned text naming
|
||||
each folder, every row AI-written and freshly stamped, and the
|
||||
terminal keeps ``summaries_done == summaries_total == 5`` (the
|
||||
full-regeneration count — a first sync passes no prior rows, so
|
||||
this is the pre-127 behavior preserved end to end)."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the FULL candidate
|
||||
# count — the pre-127 shape a first sync still reports.
|
||||
assert body["summaries_done"] == N_CANDIDATES, body
|
||||
assert body["summaries_total"] == N_CANDIDATES, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
# The DB pins the full regeneration: exactly one AI row per
|
||||
# candidate, the deterministic canned line, a fresh stamp.
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {""} | set(FOLDERS), rows
|
||||
for folder, (summary, manually, updated_at) in rows.items():
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at is not None, (folder, updated_at)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. ONE changed file in alpha/ → exactly alpha's chain (alpha + the
|
||||
# source root) is re-stamped; every other row is byte-identical
|
||||
# (text AND updated_at); the terminal keeps the AFFECTED count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_one_file_change_restamps_only_that_folders_chain(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Re-seed (the autouse truncate) + sync #1 (full), capturing every
|
||||
row's ``(summary, updated_at)``; then ONE doc inside ``alpha/``
|
||||
changes content and sync #2 settles ``success``: the ``alpha`` AND
|
||||
the source-root rows carry a NEW ``updated_at`` (strictly after
|
||||
the captured stamp) with the canned text, while the
|
||||
``bravo``/``charlie``/``delta`` rows are byte-identical — same
|
||||
``summary`` AND same ``updated_at`` (an unaffected folder's
|
||||
documents, and hence its summary, are untouched — no re-stamp).
|
||||
The terminal keeps ``summaries_done == summaries_total == 2`` (the
|
||||
AFFECTED count — alpha + root, NOT the table's candidate count —
|
||||
the keep-final-counts convention), and the detail shows the
|
||||
single-file change."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == N_FILES, body["detail"]
|
||||
before = _folder_rows()
|
||||
assert set(before) == {""} | set(FOLDERS), before
|
||||
|
||||
# The single-file change: ONE alpha doc gets new content (the
|
||||
# importer's sha256 delta reports ``updated`` — no add, no prune).
|
||||
target = incsum_dir / FOLDER_A / f"{FOLDER_A}-a.md"
|
||||
target.write_text(
|
||||
_md("Incsum Alpha A (revised)", ALPHA_A_REVISED), encoding="utf-8"
|
||||
)
|
||||
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 1, detail
|
||||
assert detail["unchanged"] == N_FILES - 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the AFFECTED count —
|
||||
# alpha + the source root — the phase-127 observable at the wire.
|
||||
assert body["summaries_done"] == 2, body
|
||||
assert body["summaries_total"] == 2, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
|
||||
after = _folder_rows()
|
||||
assert set(after) == {""} | set(FOLDERS), after
|
||||
# The chain (alpha + the source root): RE-STAMPED — a new stamp
|
||||
# strictly after the captured one, the canned text, still
|
||||
# AI-written (the mock labels from the folder header only, so the
|
||||
# text is the same deterministic line).
|
||||
for folder in ("", FOLDER_A):
|
||||
summary, manually, updated_at = after[folder]
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at > before[folder][2], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
# The siblings: BYTE-IDENTICAL — same summary text AND same
|
||||
# updated_at (never re-stamped, never even re-fetched for
|
||||
# generation).
|
||||
for folder in FOLDERS:
|
||||
if folder == FOLDER_A:
|
||||
continue
|
||||
assert after[folder] == before[folder], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. A vanished folder (bravo/ loses its last docs) has its row PRUNED
|
||||
# by the prune pass; the changed sibling's chain (charlie + the
|
||||
# source root) is re-stamped; the other rows are byte-identical
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_folder(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Re-seed + sync #1 (full), capturing every row; then ``bravo/``
|
||||
loses its last docs (the folder VANISHES — 0 documents, no longer
|
||||
a candidate) and ONE doc in ``charlie/`` changes content; the sync
|
||||
settles ``success``: the ``bravo`` row is PRUNED (gone — the
|
||||
prune pass owns vanished folders' rows, independent of the
|
||||
generation's affected set), the source-root AND ``charlie`` rows
|
||||
are re-stamped (new ``updated_at`` + canned text), the
|
||||
``alpha``/``delta`` rows are byte-identical (text AND
|
||||
``updated_at``), and the terminal keeps
|
||||
``summaries_done == summaries_total == 2`` (the affected
|
||||
CANDIDATE count — charlie + root; bravo is no longer a candidate,
|
||||
so it is neither generated nor counted)."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == N_FILES, body["detail"]
|
||||
before = _folder_rows()
|
||||
assert set(before) == {""} | set(FOLDERS), before
|
||||
|
||||
# bravo/ loses its last docs — the folder vanishes from the
|
||||
# catalogue (0 documents → below the ≥ 1 minimum → not a
|
||||
# candidate → the prune pass owns its row).
|
||||
for letter in ("a", "b"):
|
||||
(incsum_dir / FOLDER_B / f"{FOLDER_B}-{letter}.md").unlink()
|
||||
# One charlie doc gets new content (``updated``).
|
||||
target = incsum_dir / FOLDER_C / f"{FOLDER_C}-a.md"
|
||||
target.write_text(
|
||||
_md("Incsum Charlie A (revised)", CHARLIE_A_REVISED), encoding="utf-8"
|
||||
)
|
||||
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 1, detail
|
||||
assert detail["unchanged"] == N_FILES - 3, detail
|
||||
assert detail["pruned"] == 2, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the affected
|
||||
# CANDIDATE count — charlie + the source root (bravo is gone from
|
||||
# the candidates, so it is not in the affected set's intersection).
|
||||
assert body["summaries_done"] == 2, body
|
||||
assert body["summaries_total"] == 2, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
|
||||
after = _folder_rows()
|
||||
# The vanished folder's row is PRUNED (the prune pass — it runs in
|
||||
# both modes, manual rows exempt — bravo's row is AI-written).
|
||||
assert FOLDER_B not in after, after
|
||||
assert set(after) == ({""} | set(FOLDERS)) - {FOLDER_B}, after
|
||||
# charlie + the source root: RE-STAMPED — a new stamp strictly
|
||||
# after the captured one, the canned text, still AI-written.
|
||||
for folder in ("", FOLDER_C):
|
||||
summary, manually, updated_at = after[folder]
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at > before[folder][2], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
# alpha + delta: BYTE-IDENTICAL — same summary text AND same
|
||||
# updated_at (neither folder is in the changed set's chains).
|
||||
for folder in (FOLDER_A, FOLDER_D):
|
||||
assert after[folder] == before[folder], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
@@ -64,6 +64,17 @@ background sync task lives on the app's event loop, so the loop must
|
||||
survive across requests — exactly how the app runs under uvicorn.
|
||||
(A TestClient without the context manager starts a fresh loop per
|
||||
request and would cancel the task on request exit.)
|
||||
|
||||
Phase 127 (task 03): the changed-KB branch passes the import's change
|
||||
set to the generator (``changed_docs=summary.changed_paths`` —
|
||||
incremental regeneration over the changed set's folder keys only),
|
||||
the ``changed=N`` log line names the change count, and the status's
|
||||
``summaries_total`` is the affected count (the hook's total) rather
|
||||
than the table's candidate count. The unchanged-walk gap probe is
|
||||
orthogonal: it still calls the generator exactly as before
|
||||
(``only_missing=True``, no change set). The fakes above record the
|
||||
``changed_docs=`` kwarg (``FakeFolderSummaries.changed_docs_sets``) so
|
||||
the wiring is asserted, not inferred.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -71,7 +82,7 @@ import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from collections.abc import Callable, Iterator, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -85,8 +96,9 @@ from app.api import sync as sync_api
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import GitSource
|
||||
from app.models import Document, GitSource
|
||||
from app.rag import git_sources as git_sources_resolver
|
||||
from app.rag.folder_summaries import affected_folder_keys
|
||||
from app.rag.importer import ImportSummary
|
||||
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
|
||||
from app.rag.sources_meta import current_sources_version
|
||||
@@ -330,6 +342,11 @@ class FakeFolderSummaries:
|
||||
hook when given — the same way the import's file counter is driven
|
||||
through its hook, so the status's summary counters are testable
|
||||
deterministically.
|
||||
Phase 127 (task 03): records the ``changed_docs=`` change set the
|
||||
runner passes on the CHANGED-KB branch (the import's
|
||||
``changed_paths`` — None when the runner regresses to the
|
||||
change-set-less full regeneration; the unchanged-walk gap-fill
|
||||
branch never passes it).
|
||||
"""
|
||||
|
||||
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
||||
@@ -346,6 +363,7 @@ class FakeFolderSummaries:
|
||||
self.skip_flags: list[bool] = []
|
||||
self.only_missing_flags: list[bool] = []
|
||||
self.progress_hooks: list[Callable[[int, int, str, str], None] | None] = []
|
||||
self.changed_docs_sets: list[Sequence[tuple[str, str]] | None] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -354,10 +372,12 @@ class FakeFolderSummaries:
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.skip_flags.append(skip)
|
||||
self.only_missing_flags.append(only_missing)
|
||||
self.changed_docs_sets.append(changed_docs)
|
||||
self.progress_hooks.append(on_progress)
|
||||
if skip:
|
||||
return dict(self.ZERO)
|
||||
@@ -663,6 +683,7 @@ class _GatedFolderSummaries:
|
||||
self.steps_before = steps_before
|
||||
self.steps_after = steps_after
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
self.changed_docs: Sequence[tuple[str, str]] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -671,10 +692,12 @@ class _GatedFolderSummaries:
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
assert not skip, "these runs take a generation branch, never --limit"
|
||||
self.hook = on_progress
|
||||
self.changed_docs = changed_docs
|
||||
for done, total, source, folder_path in self.steps_before:
|
||||
if on_progress is not None:
|
||||
on_progress(done, total, source, folder_path)
|
||||
@@ -822,6 +845,7 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.hook = on_progress
|
||||
@@ -845,6 +869,255 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
|
||||
assert failing.hook is not None # the wiring held
|
||||
|
||||
|
||||
# --- phase 127 (task 03): the changed-KB branch's incremental wiring ----
|
||||
|
||||
|
||||
def test_changed_sync_passes_the_change_set_to_the_generator(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): a changed-KB sync passes the import's
|
||||
change set into the generator (``changed_docs=summary.
|
||||
changed_paths`` — the incremental-regression restriction), NOT the
|
||||
gap-fill flag, the success ``detail`` keeps its exact pre-phase key
|
||||
set (the stats stay log-only), and the log line names the change
|
||||
count (``changed=N``)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
# One updated document — the import's change set carries exactly
|
||||
# its (source, path) identity (task 01's contract).
|
||||
change_set = [("repo", "notes/deep.md")]
|
||||
summary = ImportSummary(
|
||||
files=4, added=0, updated=1, unchanged=3, changed_paths=change_set
|
||||
)
|
||||
fake_import = FakeImportSources(summary)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
fake_folders = FakeFolderSummaries()
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||
|
||||
_login(sync_client)
|
||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# The CHANGED-KB branch fired the generator exactly once ... with
|
||||
# the change set, NOT the gap-fill flag (the two restriction
|
||||
# kwargs are orthogonal — no caller combines them).
|
||||
assert len(fake_folders.changed_docs_sets) == 1
|
||||
assert fake_folders.changed_docs_sets == [change_set]
|
||||
assert fake_folders.only_missing_flags == [False]
|
||||
assert fake_folders.skip_flags == [False]
|
||||
# The success detail is byte-identical in shape to pre-phase 127 —
|
||||
# no new key (the stats stay log-only, the phase-98 convention).
|
||||
assert set(body["detail"]) == {
|
||||
"files", "added", "updated", "unchanged", "pruned", "errors",
|
||||
"chunks", "summaries", "summary_errors", "dates_updated",
|
||||
"overview", "sources_version",
|
||||
}
|
||||
assert body["detail"]["added"] == 0 and body["detail"]["updated"] == 1
|
||||
assert body["detail"]["overview"] is True
|
||||
assert body["detail"]["sources_version"] == 1 # changed KB → one bump
|
||||
# The ample-logging line names the change count (PLAN §9).
|
||||
assert any(
|
||||
r.getMessage() == (
|
||||
"sync: folder_summaries changed=1 "
|
||||
"stats={'generated': 0, 'failed': 0, 'pruned': 0}"
|
||||
)
|
||||
for r in caplog.records
|
||||
)
|
||||
|
||||
|
||||
def test_unchanged_walk_gap_probe_passes_no_change_set(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): the unchanged-walk gap probe is
|
||||
ORTHOGONAL to the change set — a walk that changed nothing with a
|
||||
real gap (seeded catalogue, empty ``folder_summaries`` table) still
|
||||
calls the generator exactly as before: ``only_missing=True`` and
|
||||
NO ``changed_docs`` (the gap probe's total is the missing count,
|
||||
as before)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
# The gap: two docs under notes/ (candidates root + notes), the
|
||||
# folder_summaries table empty (the autouse fixture truncated it).
|
||||
db.add_all([
|
||||
Document(
|
||||
source="repo", path="notes/one.md", full_path="/tmp/one.md",
|
||||
title="One", content="# One\nfirst\n", content_hash="a" * 64,
|
||||
),
|
||||
Document(
|
||||
source="repo", path="notes/two.md", full_path="/tmp/two.md",
|
||||
title="Two", content="# Two\nsecond\n", content_hash="b" * 64,
|
||||
),
|
||||
])
|
||||
db.commit()
|
||||
summary = ImportSummary(files=2, added=0, updated=0, unchanged=2)
|
||||
fake_import = FakeImportSources(summary)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
fake_folders = FakeFolderSummaries()
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# The gap probe fired the generator exactly once — the gap-fill
|
||||
# shape, byte-identical to pre-phase 127: only_missing=True, and
|
||||
# NO change set (the gap is a stored-rows gap, not a document
|
||||
# change — passing changed_paths here would restrict the fill to
|
||||
# nothing).
|
||||
assert len(fake_folders.changed_docs_sets) == 1
|
||||
assert fake_folders.changed_docs_sets == [None]
|
||||
assert fake_folders.only_missing_flags == [True]
|
||||
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||
assert body["detail"]["sources_version"] == 0 # unchanged → no bump
|
||||
|
||||
|
||||
class _AffectedKeysFolderSummaries:
|
||||
"""A folder-summary step that mirrors the REAL generator's hook
|
||||
contract on a changed set (phase 127, task 02): the loop runs over
|
||||
the sorted AFFECTED keys (``affected_folder_keys(changed_docs)``),
|
||||
the hook fires once per key with ``total == len(keys)``, and the
|
||||
step parks mid-span after the first key — the (long) phase the
|
||||
user reported. The test reads the status mid-span and asserts the
|
||||
status's ``summaries_total`` IS the affected count (the hook's
|
||||
total), not the table's candidate count."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
started: threading.Event,
|
||||
release: threading.Event,
|
||||
) -> None:
|
||||
self.started = started
|
||||
self.release = release
|
||||
self.changed_docs: Sequence[tuple[str, str]] | None = None
|
||||
self.keys: list[tuple[str, str]] = []
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
db: Session,
|
||||
llm: LLMClient,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
assert not skip, "these runs take a generation branch, never --limit"
|
||||
self.hook = on_progress
|
||||
self.changed_docs = changed_docs
|
||||
assert on_progress is not None, "the runner must wire the hook"
|
||||
# The real generator's step 2: sorted(candidates & affected) —
|
||||
# the fake takes the affected side (the candidates are a
|
||||
# superset here: every affected folder holds a doc).
|
||||
self.keys = sorted(affected_folder_keys(changed_docs or []))
|
||||
assert self.keys, "the change set must affect at least the root"
|
||||
on_progress(1, len(self.keys), self.keys[0][0], self.keys[0][1])
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
for i, (source, folder_path) in enumerate(self.keys[1:], start=2):
|
||||
on_progress(i, len(self.keys), source, folder_path)
|
||||
return {"generated": len(self.keys), "failed": 0, "pruned": 0}
|
||||
|
||||
|
||||
def test_summaries_phase_reports_the_affected_count_as_total(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): mid-span, the status's ``summaries_total``
|
||||
is the AFFECTED count — ``len(affected_folder_keys(changed set))``
|
||||
(the hook's total), not the table's candidate count — and the
|
||||
terminal keeps that final count (the phase-64 convention, now over
|
||||
the affected span). The change set: one nested-folder doc (a
|
||||
three-key chain) + one root-level doc (shares the root key) →
|
||||
exactly 3 affected keys."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
change_set = [("repo", "notes/a/one.md"), ("repo", "top.md")]
|
||||
fake_import = FakeImportSources(
|
||||
ImportSummary(
|
||||
files=6, added=1, updated=2, unchanged=3, changed_paths=change_set
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
gated_folders = _AffectedKeysFolderSummaries(
|
||||
threading.Event(), threading.Event()
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
|
||||
s = sync_client.get("/api/sync/status").json()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "summaries"
|
||||
# The hook's first key: the source root (the bare source name —
|
||||
# folder_path "" never gets a slash) ...
|
||||
assert s["current_summary"] == "repo"
|
||||
assert s["summaries_done"] == 1
|
||||
# ... and the total IS the affected count (the task's pin):
|
||||
# notes/a/one.md → ("repo", ""), ("repo", "notes"),
|
||||
# ("repo", "notes/a"); top.md → ("repo", "") — 3 keys.
|
||||
expected = sorted(affected_folder_keys(change_set))
|
||||
assert expected == [("repo", ""), ("repo", "notes"), ("repo", "notes/a")]
|
||||
assert s["summaries_total"] == len(expected)
|
||||
assert s["summaries_total"] == 3
|
||||
gated_folders.release.set()
|
||||
finally:
|
||||
gated_folders.release.set()
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
# Terminal: phase + current_summary cleared, the hook's FINAL
|
||||
# (affected) counts kept (the phase-64 keep-final-counts rule, now
|
||||
# over the affected span).
|
||||
assert body["phase"] is None and body["current_summary"] is None
|
||||
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
|
||||
# Wiring: the runner passed the change set, not the gap-fill flag.
|
||||
assert gated_folders.changed_docs == change_set
|
||||
assert gated_folders.hook is not None
|
||||
|
||||
|
||||
# --- admin: failures -------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ from app.rag.folder_summaries import (
|
||||
FOLDER_SUMMARY_MODE,
|
||||
MIN_DOCS_PER_FOLDER,
|
||||
SYSTEM_PROMPT,
|
||||
affected_folder_keys,
|
||||
build_folder_summary_prompt,
|
||||
folder_of,
|
||||
generate_folder_summaries,
|
||||
@@ -202,6 +203,56 @@ def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
|
||||
assert "top.md" not in [folder for _source, folder in groups]
|
||||
|
||||
|
||||
# ---------- affected_folder_keys (phase 127, task 02) ----------
|
||||
|
||||
|
||||
def test_affected_folder_keys_root_level_file_is_only_the_root() -> None:
|
||||
"""A root-level file (``folder_of == ""``) affects ONLY the source
|
||||
root — there is no folder prefix to walk."""
|
||||
assert affected_folder_keys([("S", "a.md")]) == {("S", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_nested_file_walks_the_folder_chain() -> None:
|
||||
"""A doc under ``a/b/`` affects ``a/b``, ``a``, and the source
|
||||
root — the SAME prefix walk ``group_by_folder`` pass 1 uses (one
|
||||
concept)."""
|
||||
assert affected_folder_keys([("S", "a/b/c.md")]) == {
|
||||
("S", ""),
|
||||
("S", "a"),
|
||||
("S", "a/b"),
|
||||
}
|
||||
|
||||
|
||||
def test_affected_folder_keys_multi_source_set_semantics() -> None:
|
||||
"""Keys are per source (the PK is ``(source, folder_path)``), and
|
||||
the result is a SET — a repeated chain adds no new keys, and an
|
||||
unrelated source keeps its own root."""
|
||||
keys = affected_folder_keys(
|
||||
[
|
||||
("S", "a/b/c.md"),
|
||||
("T", "x.md"),
|
||||
("S", "a/d.md"), # the same chain as the first entry — no new keys
|
||||
]
|
||||
)
|
||||
assert keys == {("S", ""), ("S", "a"), ("S", "a/b"), ("T", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_fragment_paths_fold_into_their_parent_folder() -> None:
|
||||
"""Embedded-image fragment paths (phase 126: ``x.pdf#page=…#img=…``)
|
||||
get NO special case: the fragment never holds a ``/``, so
|
||||
``folder_of`` folds them into the parent folder's chain (a
|
||||
root-level fragment affects only the source root)."""
|
||||
assert affected_folder_keys([("S", "dir/img.pdf#page=2#img=0")]) == {
|
||||
("S", ""),
|
||||
("S", "dir"),
|
||||
}
|
||||
assert affected_folder_keys([("S", "img.pdf#page=1#img=0")]) == {("S", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_empty_input_is_an_empty_set() -> None:
|
||||
assert affected_folder_keys([]) == set()
|
||||
|
||||
|
||||
# ---------- build_folder_summary_prompt: system ----------
|
||||
|
||||
|
||||
@@ -1093,3 +1144,332 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
|
||||
("FSU", "a/b"): REPLY,
|
||||
("FSU-solo", ""): REPLY,
|
||||
}
|
||||
|
||||
|
||||
# ---------- generate_folder_summaries(changed_docs=…) (phase 127, task 02) ----------
|
||||
|
||||
|
||||
def _seed_three_sources(db: Session) -> None:
|
||||
"""The restricted-mode catalogue (three sources): A is nested
|
||||
(``a/`` and ``a/b/``), B is a sibling single-folder source, and C
|
||||
is a single-doc root — six candidate keys total: A ×
|
||||
{``""``, ``a``, ``a/b``}, B × {``""``, ``b``}, C × {``""``}.
|
||||
"""
|
||||
_add_doc(db, "A", "a/b/one.md", "A One", "A one lead.")
|
||||
_add_doc(db, "A", "a/b/two.md", "A Two")
|
||||
_add_doc(db, "A", "a/three.md", "A Three")
|
||||
_add_doc(db, "B", "b/one.md", "B One")
|
||||
_add_doc(db, "C", "solo.md", "C Solo")
|
||||
|
||||
|
||||
def _stamps(
|
||||
db: Session, keys: set[tuple[str, str]]
|
||||
) -> dict[tuple[str, str], object]:
|
||||
"""The stored rows' ``updated_at`` for *keys* (raw SQL — the
|
||||
byte-identity comparisons stay honest)."""
|
||||
return {
|
||||
(source, folder): _updated_at(db, source, folder) for source, folder in keys
|
||||
}
|
||||
|
||||
|
||||
def _freeze(
|
||||
db: Session, values: dict[tuple[str, str], str]
|
||||
) -> dict[tuple[str, str], object]:
|
||||
"""Overwrite the stored summaries for *values*' keys with DISTINCT
|
||||
frozen text (a "regenerated" row is then distinguishable from an
|
||||
"untouched" one), and return the pre-run stamps."""
|
||||
for (source, folder_path), value in values.items():
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE folder_summaries SET summary = :v"
|
||||
" WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"v": value, "s": source, "f": folder_path},
|
||||
)
|
||||
db.commit()
|
||||
return _stamps(db, set(values))
|
||||
|
||||
|
||||
def test_changed_docs_restricted_regeneration_only_affected_keys(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""One changed file in A/``a/b/`` → exactly that file's folder
|
||||
chain (``a/b``, ``a``, and the A root) regenerates: one lite call
|
||||
per affected non-manual key, in sorted order, fresh rows. Every
|
||||
OTHER stored row — same-source siblings and other sources — keeps
|
||||
its summary text AND ``updated_at`` byte-identical (zero lite
|
||||
calls for them)."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
frozen = {
|
||||
("A", ""): "A root frozen",
|
||||
("A", "a"): "A/a frozen",
|
||||
("A", "a/b"): "A/a/b frozen",
|
||||
("B", ""): "B root frozen",
|
||||
("B", "b"): "B/b frozen",
|
||||
("C", ""): "C root frozen",
|
||||
}
|
||||
stamps = _freeze(db, frozen)
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3, "one lite call per AFFECTED non-manual key — nothing else"
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a",
|
||||
"Folder: A/a/b",
|
||||
], "the affected chain in sorted (source, folder_path) order"
|
||||
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "")] == REPLY
|
||||
assert stored[("A", "a")] == REPLY
|
||||
assert stored[("A", "a/b")] == REPLY, "the changed file's own folder regenerates"
|
||||
assert stored[("B", "")] == "B root frozen"
|
||||
assert stored[("B", "b")] == "B/b frozen"
|
||||
assert stored[("C", "")] == "C root frozen", (
|
||||
"the unaffected rows keep their stored text"
|
||||
)
|
||||
for key in (("B", ""), ("B", "b"), ("C", "")):
|
||||
assert _updated_at(db, *key) == stamps[key], (
|
||||
"the unaffected rows are never re-stamped"
|
||||
)
|
||||
|
||||
|
||||
def test_changed_docs_manual_affected_row_is_skipped(db: Session, clean_tables) -> None:
|
||||
"""A manual row INSIDE the affected chain is skipped exactly as in
|
||||
full mode (phase 97): ``kept_manual``, no lite call, text AND
|
||||
``updated_at`` byte-identical — the restriction changes the key
|
||||
set, not the manual-row rule."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
manual_text = "Owner's words about A/a."
|
||||
row = db.get(FolderSummary, ("A", "a"))
|
||||
assert row is not None
|
||||
row.summary = manual_text
|
||||
row.manually_edited = True
|
||||
db.commit()
|
||||
stamp_before = _updated_at(db, "A", "a")
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 2, "failed": 0, "pruned": 0, "kept_manual": 1}
|
||||
assert llm.calls == 2, "the manual key burns zero lite calls"
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a/b",
|
||||
]
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a")] == manual_text, "the owner's text is never rewritten"
|
||||
assert _updated_at(db, "A", "a") == stamp_before, ("never re-stamped")
|
||||
assert stored[("A", "")] == REPLY and stored[("A", "a/b")] == REPLY
|
||||
assert stored[("B", "")] == REPLY and stored[("B", "b")] == REPLY
|
||||
assert stored[("C", "")] == REPLY, "the unaffected rows are left as they were"
|
||||
|
||||
|
||||
def test_changed_docs_failing_affected_folder_fails_soft(db: Session, clean_tables) -> None:
|
||||
"""Per-folder fail-soft under the restriction: the failing
|
||||
affected key is attempted and counted, its PREVIOUS row is kept
|
||||
(an old summary is better than none), the other affected keys
|
||||
still land, and the unaffected rows are untouched."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
# A DISTINCT previous text on the failing folder, so "kept" is
|
||||
# visible (it would otherwise be the REPLY either way).
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE folder_summaries SET summary = 'old A/a/b'"
|
||||
" WHERE (source, folder_path) = ('A', 'a/b')"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("B", ""), ("B", "b"), ("C", "")})
|
||||
|
||||
llm = _FakeLLM(fail_folders=("A/a/b",))
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3, "the failing key is attempted (the hook's premise)"
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a/b")] == "old A/a/b", "the previous row survives"
|
||||
assert stored[("A", "")] == REPLY and stored[("A", "a")] == REPLY
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
|
||||
|
||||
|
||||
def test_changed_docs_multi_source_change_set(db: Session, clean_tables) -> None:
|
||||
"""A change set spanning SOURCES: the affected keys are the UNION
|
||||
of each document's chain (A's nested chain + B's root and ``b``),
|
||||
sorted across sources; the unaffected C root row keeps its text
|
||||
AND ``updated_at``."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
c_stamp = _updated_at(db, "C", "")
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(
|
||||
db, llm, changed_docs=[("A", "a/b/two.md"), ("B", "b/one.md")]
|
||||
)
|
||||
)
|
||||
assert stats == {"generated": 5, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 5
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a",
|
||||
"Folder: A/a/b",
|
||||
"Folder: B",
|
||||
"Folder: B/b",
|
||||
]
|
||||
stored = _rows(db)
|
||||
assert stored[("C", "")] == REPLY
|
||||
assert _updated_at(db, "C", "") == c_stamp
|
||||
|
||||
|
||||
def test_changed_docs_gap_in_affected_folder_is_generated(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""An affected folder with NO stored row (a gap in the chain) is a
|
||||
candidate and gets generated like any other affected key — while a
|
||||
gap in an UNAFFECTED folder stays the unchanged-walk gap probe's
|
||||
job (phase 96): the restriction never widens beyond the affected
|
||||
set."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries"
|
||||
" WHERE (source, folder_path) IN (('A', 'a/b'), ('B', ''))"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a/b")] == REPLY, "the gap in the affected chain is filled"
|
||||
assert ("B", "") not in stored, (
|
||||
"the gap in the UNAFFECTED source is left for the gap probe"
|
||||
)
|
||||
assert stored[("B", "b")] == REPLY
|
||||
|
||||
|
||||
def test_changed_docs_vanished_affected_folder_is_pruned_not_generated(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The ASSUMPTION pinned: an affected folder that no longer holds
|
||||
any document is a PRUNE-pass job, not a generation's — the pruned
|
||||
document's chain (root + its now-vanished folder) is in the change
|
||||
set, only the surviving candidate (the root) generates, the
|
||||
vanished folder's row is dropped, and the unaffected rows keep
|
||||
their ``updated_at``."""
|
||||
_add_doc(db, "A", "a/b/one.md", "A One")
|
||||
_add_doc(db, "A", "a/b/two.md", "A Two")
|
||||
_add_doc(db, "A", "gone/old.md", "Gone")
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("A", "a"), ("A", "a/b")})
|
||||
|
||||
db.execute(text("DELETE FROM documents WHERE path = 'gone/old.md'"))
|
||||
db.commit()
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "gone/old.md")])
|
||||
)
|
||||
# affected = {("A", ""), ("A", "gone")} — the only surviving
|
||||
# candidate in that set is the source root.
|
||||
assert stats == {"generated": 1, "failed": 0, "pruned": 1, "kept_manual": 0}
|
||||
assert llm.calls == 1
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == ["Folder: A"]
|
||||
stored = _rows(db)
|
||||
assert ("A", "gone") not in stored, "the vanished folder's row is pruned"
|
||||
assert stored[("A", "")] == REPLY, "the root's subtree changed — regenerated"
|
||||
assert stored[("A", "a")] == REPLY and stored[("A", "a/b")] == REPLY
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
|
||||
|
||||
|
||||
def test_changed_docs_hook_total_is_the_affected_count(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The progress hook (phase 98) under the restriction fires exactly
|
||||
``len(affected)`` times with ``total == len(affected)`` — the sync
|
||||
status's ``summaries_total`` then reports the affected count
|
||||
through the existing hook — in sorted key order."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(
|
||||
db, llm, changed_docs=[("A", "a/b/one.md")], on_progress=record
|
||||
)
|
||||
)
|
||||
assert events == [
|
||||
(1, 3, "A", ""),
|
||||
(2, 3, "A", "a"),
|
||||
(3, 3, "A", "a/b"),
|
||||
]
|
||||
assert llm.calls == 3
|
||||
assert stats["generated"] == 3
|
||||
|
||||
|
||||
def test_changed_docs_none_is_the_pre_127_full_regeneration(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The default (no ``changed_docs`` kwarg) is the pre-127 FULL
|
||||
regeneration over the same three-source catalogue: every
|
||||
candidate key, one call each, the 4-field stats shape unchanged —
|
||||
the ``scripts/import_docs.py`` CLI contract."""
|
||||
_seed_three_sources(db)
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||
assert stats == {"generated": 6, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 6, "no kwarg → every candidate burns one call"
|
||||
assert set(_rows(db)) == {
|
||||
("A", ""), ("A", "a"), ("A", "a/b"),
|
||||
("B", ""), ("B", "b"),
|
||||
("C", ""),
|
||||
}
|
||||
|
||||
|
||||
def test_changed_docs_empty_set_generates_nothing_but_still_prunes(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""``changed_docs=[]`` (a change set that IS given but empty)
|
||||
restricts the generation to zero keys — no lite calls, no
|
||||
re-stamps, the hook never fires — while the prune pass still runs
|
||||
in the given mode (the sweep is mode-independent, phase 97)."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.add(FolderSummary(source="A", folder_path="gone/old", summary="stale"))
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("A", ""), ("A", "a"), ("A", "a/b"), ("B", ""), ("C", "")})
|
||||
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[], on_progress=record)
|
||||
)
|
||||
assert stats == {"generated": 0, "failed": 0, "pruned": 1, "kept_manual": 0}
|
||||
assert llm.calls == 0
|
||||
assert events == [], "zero keys → the hook never fires"
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "no row is re-stamped"
|
||||
|
||||
@@ -15,9 +15,11 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf as fitz
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -28,6 +30,7 @@ from app.models import Chunk, Document
|
||||
from app.rag.importer import (
|
||||
EXCLUDED_DIRS,
|
||||
ImportSummary,
|
||||
_prune,
|
||||
_store_summary,
|
||||
import_sources,
|
||||
iter_importable_files,
|
||||
@@ -1194,3 +1197,236 @@ def test_progress_hook_with_limit_keeps_full_total(db, tmp_path: Path) -> None:
|
||||
assert events == [(root.name, "a.md", 1, 3), (root.name, "b.md", 2, 3)]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 127 (task 01): ImportSummary.changed_paths ----------
|
||||
|
||||
|
||||
class _VisionLLM(FakeEmbedder):
|
||||
"""The phase-122 mock VISION client (the test_pdf pattern): the
|
||||
multimodal describe call (``user`` content = a part list) gets a fixed
|
||||
description; text (``lite``) calls keep the ``FakeEmbedder``
|
||||
behaviour."""
|
||||
|
||||
DESCRIPTION = "A test image: a solid swatch of color."
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
return self.DESCRIPTION
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def _png_bytes(width: int, height: int, rgb: tuple[float, float, float]) -> bytes:
|
||||
"""One solid-color PNG built with PyMuPDF (the phase-126 no-binary-
|
||||
fixtures convention)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
|
||||
for y in range(height):
|
||||
for x in range(width):
|
||||
pix.set_pixel(x, y, rgb)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
def _build_one_page_pdf(path: Path, text: str) -> None:
|
||||
"""One temp single-page PDF (the phase-126 convention — no binary
|
||||
fixtures)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), text)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def test_import_summary_changed_paths_defaults_empty() -> None:
|
||||
"""Completion criterion: default construction gives an empty change
|
||||
set — no existing construction site migrates."""
|
||||
assert ImportSummary().changed_paths == []
|
||||
|
||||
|
||||
def test_changed_paths_records_added_updated_and_pruned_in_order(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The contract (phase 127, task 01): one ``(source, path)`` entry per
|
||||
ADDED, UPDATED, or PRUNED document — added/updated in walk order, the
|
||||
pruned pair last (``_prune`` runs after the walk) — and
|
||||
``len(changed_paths) == added + updated + pruned``."""
|
||||
root = tmp_path / "changed"
|
||||
root.mkdir()
|
||||
for name in ("a.md", "b.md", "c.md"):
|
||||
(root / name).write_text(f"# {name}\n\nv1\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 3 and first.pruned == 0
|
||||
assert first.changed_paths == [
|
||||
(source, "a.md"),
|
||||
(source, "b.md"),
|
||||
(source, "c.md"),
|
||||
]
|
||||
# Run 2: update a.md, add d.md + e.md, delete c.md (→ pruned).
|
||||
(root / "a.md").write_text("# a\n\nv2 changed\n")
|
||||
(root / "d.md").write_text("# d\n\nnew d\n")
|
||||
(root / "e.md").write_text("# e\n\nnew e\n")
|
||||
(root / "c.md").unlink()
|
||||
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert (second.added, second.updated, second.unchanged, second.pruned) == (
|
||||
2, 1, 1, 1,
|
||||
)
|
||||
assert second.changed_paths == [
|
||||
(source, "a.md"), # updated — first in walk order
|
||||
(source, "d.md"), # added
|
||||
(source, "e.md"), # added
|
||||
(source, "c.md"), # pruned — the _prune pass runs after the walk
|
||||
]
|
||||
assert len(second.changed_paths) == (
|
||||
second.added + second.updated + second.pruned
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_unchanged_rerun(db, tmp_path: Path) -> None:
|
||||
"""An all-unchanged re-run (prune included) records nothing."""
|
||||
root = tmp_path / "nochange"
|
||||
root.mkdir()
|
||||
(root / "a.md").write_text("# a\n\nstable\n")
|
||||
(root / "b.md").write_text("# b\n\nstable\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert second.unchanged == 2
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_backfill_rerun(db, tmp_path: Path) -> None:
|
||||
"""A backfill (phase 118: unchanged doc, NULL summary) is NOT a content
|
||||
change: the add still records, the backfill re-run stays empty."""
|
||||
root = tmp_path / "backfill"
|
||||
root.mkdir()
|
||||
(root / "a.yaml").write_text("alpha x\n")
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
|
||||
assert first.added == 1 and first.summary_errors == 1
|
||||
assert first.changed_paths == [(root.name, "a.yaml")] # the ADD records
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "a.yaml")
|
||||
)
|
||||
assert doc is not None and doc.summary is None
|
||||
second = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
|
||||
assert second.unchanged == 1 and second.summary_backfilled == 1
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_date_only_rerun(db, tmp_path: Path) -> None:
|
||||
"""A date-only refresh (phase 106, D4: content untouched, mtime moved)
|
||||
is NOT a content change: the re-run records nothing."""
|
||||
root = tmp_path / "dateonly"
|
||||
root.mkdir()
|
||||
path = root / "a.md"
|
||||
path.write_text("# a\n\nstable\n")
|
||||
os.utime(path, (0, datetime(2020, 1, 1, tzinfo=UTC).timestamp()))
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1 and first.dates_updated == 0
|
||||
assert first.changed_paths == [(root.name, "a.md")]
|
||||
# The content is untouched — only the mtime moved (the D4
|
||||
# date-refresh on the unchanged path).
|
||||
os.utime(path, (0, datetime(2021, 6, 15, tzinfo=UTC).timestamp()))
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.unchanged == 1 and second.dates_updated == 1
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_records_a_changed_image(db, tmp_path: Path) -> None:
|
||||
"""The image increment site (phase 122 path): a NEW image and a
|
||||
byte-CHANGED image each record their ``(source, rel)``."""
|
||||
root = tmp_path / "imgsrc"
|
||||
root.mkdir()
|
||||
(root / "pic.png").write_bytes(_png_bytes(1, 1, (1.0, 0.0, 0.0)))
|
||||
llm = _VisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
)
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1 and first.images_failed == 0
|
||||
assert first.changed_paths == [(source, "pic.png")]
|
||||
# The BYTES changed (a new color) — the image digest is over the
|
||||
# bytes, so this is an UPDATE.
|
||||
(root / "pic.png").write_bytes(_png_bytes(1, 1, (0.0, 1.0, 1.0)))
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.updated == 1
|
||||
assert second.changed_paths == [(source, "pic.png")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_records_a_changed_pdf(db, tmp_path: Path) -> None:
|
||||
"""The PDF increment site (phase 126 path): a NEW PDF and a text-
|
||||
CHANGED PDF each record their ``(source, rel)``."""
|
||||
root = tmp_path / "pdfsrc"
|
||||
root.mkdir()
|
||||
pdf_path = root / "doc.pdf"
|
||||
_build_one_page_pdf(pdf_path, "alpha page one")
|
||||
llm = _VisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
import_extensions="md,txt,pdf",
|
||||
)
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1
|
||||
assert first.changed_paths == [(source, "doc.pdf")]
|
||||
# The extracted TEXT changed — the PDF digest is over it, so this
|
||||
# is an UPDATE.
|
||||
_build_one_page_pdf(pdf_path, "bravo page one, revised")
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.updated == 1
|
||||
assert second.changed_paths == [(source, "doc.pdf")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_prune_records_changed_paths_only_when_summary_given(db, tmp_path: Path) -> None:
|
||||
"""The ``_prune`` recording seam: with NO summary (the pre-phase
|
||||
count-only contract) nothing is recorded; with one, each ACTUALLY
|
||||
deleted doc appends its ``(source, path)`` (skipped docs never do)."""
|
||||
root = tmp_path / "prunit"
|
||||
root.mkdir()
|
||||
(root / "gone.md").write_text("# gone\n\nbye\n")
|
||||
(root / "gone2.md").write_text("# gone2\n\nbye\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
source = root.name
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
(root / "gone.md").unlink()
|
||||
(root / "gone2.md").unlink()
|
||||
# Default call (summary=None): count only, no recording.
|
||||
pruned = _prune(db, {source}, {(source, "gone2.md")})
|
||||
assert pruned == 1
|
||||
assert db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == "gone.md")
|
||||
) is None
|
||||
# With a summary: the deleted doc records its pair.
|
||||
counters = ImportSummary()
|
||||
pruned = _prune(db, {source}, set(), summary=counters)
|
||||
assert pruned == 1
|
||||
assert counters.changed_paths == [(source, "gone2.md")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -40,7 +40,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import re
|
||||
import threading
|
||||
from collections.abc import Callable, Iterator
|
||||
from collections.abc import Callable, Iterator, Sequence
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
@@ -819,12 +819,15 @@ def _patch_sync_seams(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
# Phase 98 (task 01): the runner wires the progress hook into
|
||||
# the generation branches — the fake accepts it (the state-
|
||||
# machine tests' canned run never fires it: zero summary
|
||||
# counters, like a no-gap skip).
|
||||
# counters, like a no-gap skip). Phase 127 (task 03): the
|
||||
# changed-KB branch passes the change set as changed_docs= —
|
||||
# the fake accepts it (these canned runs record nothing).
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
||||
|
||||
Reference in New Issue
Block a user