phase: 183_sync_passive_delete_cascade

All gates green. Final state verified.

**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).

**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors

**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).

**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).

**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.

**Next pending phase**: `184_sync_bounded_session`.

.agents/phases/todo/184_sync_bounded_session/00_phase.md

@@ -0,0 +1,59 @@
+# Phase 184 — The sync's session stops growing with the knowledge base
+
+**Source:** owner incident 2026-10-15 (as phase 183) — the import run held ONE session (`app/db.py:30` sets `expire_on_commit=False`) for the whole walk + prune, so every row it touched stayed resident until the run ended.
+**Story:** n/a (incident-derived, owner directive 2026-10-15)
+**Depends on:** `183_sync_passive_delete_cascade` (the batched prune deletes documents with the FK cascade doing the child work — it is only cheap because of `passive_deletes=True`).
+
+**Context:** `app/db.py` — **L30 `SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False, future=True)`** (left EXACTLY as it is — every request path reads attributes after commit); `app/rag/importer.py` — `import_sources` (**L708**, `owns_session = session is None` at **L878**, `session = SessionLocal()` at **L880**, the `total` pre-walk at L884–899, the per-source loop at L901, the per-file loop at **L916** with `progress(source, rel, summary.files, total)` at **L943** and `await _index_file(...)` at **L945**, the `EmbeddingError` → `session.rollback()` handler at L951–958, the per-source post-passes `_enrich_pending_images` (**L965**) and `_backfill_image_contexts` (**L972**), `_prune(...)` (**L979**), `session.close()` in the `finally` at L984–985), `_prune` (**L3860**, `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, `session.delete(doc)` at **L3931**, the single commit at **L3934–3935**), `_delete_image_copy` (takes `image_path: str | None`), `_delete_page_copies(doc, page_dir)` (needs only `doc.id` + `doc.page_count`); `app/api/sync.py` — `import_sources(...)` called with **no `session=`** at **L725** (the importer owns it — this phase never changes that contract); `app/config.py` (the `BOR_`-prefixed `Settings` pattern, `chunk_target_chars` at L228 as the shape), `app/core/settings_catalog.py` — `GROUPS` at **L72** (the `"Import & sources"` group) and the `CATALOG` entry shape at L226, pinned complete by `tests/unit/test_settings_catalog.py::test_catalog_covers_exactly_model_fields` (**`set(CATALOG) == set(Settings.model_fields)`** — a new `Settings` field without a catalogue row FAILS the build); `tests/unit/test_importer.py`, `tests/integration/test_importer_e2e.py`, `tests/integration/test_sync_api.py`, `tests/e2e/test_sync_changed_only.py`, `tests/e2e/test_sync_summary_visibility.py`.
+
+## Measured baseline (2026-10-15, this repo's dev DB, production-shaped rows)
+- A `Document`'s `content` averages **15.2 KB** (17,407 docs, 252.6 MB of markdown in the incident DB), and the identity map keeps every one the walk touched — see phase 185 for the column-level fix.
+- Objects are released by NOTHING during the run: `expire_on_commit=False` means a commit does not drop attributes, and the importer never expunges. The identity map is therefore the run's memory profile, and it is unbounded: it scales with (files walked) + (chunks created) + (chunks hydrateable per prune — phase 183).
+
+## Objective
+Bound the import run's resident set independently of the size of the knowledge base: `session.expunge_all()` on a fixed cadence at the one point where no previously-loaded object is reachable, and `_prune` released in committed batches instead of one transaction at the end of the whole import — with every count, log line, guard, cancel boundary and summary number byte-identical.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+- **D1 — the pin is the identity map, not RSS (task 01).** `len(session.identity_map)` is exact, deterministic and it IS the retention surface. `tests/integration/test_import_session_retention.py` runs an import of N files through `import_sources` (the real entry point, the `test_importer_e2e.py` fixture shapes + the mock-embedder pattern from `tests/unit/test_importer.py`) and asserts the map size **at the end of the run**: PRE-fix it is `≈ docs + chunks + documents-touched` (record the measured number), and the test pins `>= N` to prove accumulation. The post-fix assertion (task 02) is `<= reset_files + slack`. RSS is recorded in the docstring only, never asserted (phase 183 A4's rule, inherited).
+- **D2 — one new setting per new cadence, catalogue-compliant (tasks 02 + 03).** `sync_session_reset_files: int = 200` (`BOR_SYNC_SESSION_RESET_FILES`) and `sync_prune_batch: int = 500` (`BOR_SYNC_PRUNE_BATCH`). **`0` means "never reset / one transaction" = the exact pre-phase behaviour**, the operator's rollback knob without a re-deploy (the `BOR_LOG_LARGE_RESPONSE_BYTES` precedent). Both get `SettingMeta` entries in the `"Import & sources"` group (the completeness pin makes this mandatory, not optional), both documented in `.env.example` and `README.md`, neither `restart_required` (the importer reads `llm.settings` per run) and neither `secret`.
+- **D3 — the reset point is the top of the per-file loop, and nowhere else (task 02).** Inside `import_sources`, at the top of the L916 loop body — after the PREVIOUS file's `_index_file` has returned, before this file's `progress(...)` hook at L943 — call `session.expunge_all()` every `sync_session_reset_files` files (`summary.files % reset == 0`), plus once at each source boundary before the two post-passes (L965/L972) so a source's walk residue never crosses into the next source. Proof of safety that the task must state in the comment: no object from a finished file is reachable afterwards (`_index_file` returns `None`, the progress/chunk/image hooks only assign fields on `_status`, `dates_map`/`ignore` are plain data, and `total` was computed before the loop), and the `EmbeddingError` handler's `session.rollback()` (L951–958) is safe with detached objects because it touches none. The two post-passes select their own rows fresh (`L3241`, `L3391`), so they are unaffected.
+- **D4 — the prune is batched on light rows, not streamed cursors (task 03).** `_prune` replaces `select(Document).where(...).all()` with `select(Document).where(...).options(load_only(Document.id, Document.source, Document.path, Document.is_image, Document.image_path, Document.page_count)).order_by(Document.id)` — the exact column set the guards and the copy cleanup need (`is_image` + `path` for the phase-122 toggle guard and the phase-126 `_EMBEDDED_IMAGE_RE` parent rule, `image_path` for `_delete_image_copy`, `id` + `page_count` for `_delete_page_copies`) — and commits every `sync_prune_batch` deletes instead of once at the end (`sync_session_reset_files`'s sibling knob, `0` = the pre-phase single commit). Deletes stay `session.delete(doc)` (the phase-183 `passive_deletes` makes them child-load-free; a bulk `delete()` is NOT used, so `changed_paths`, the log line and the guards keep their exact per-document shape). `yield_per`/server-side cursors are explicitly rejected: a flush inside an open streaming result is a trap, and the light rows are ~1 KB each.
+- **D5 — batch commits change the prune's atomicity, deliberately (A2).** Pre-phase a crash mid-prune rolled the whole prune back; post-phase a crash leaves a PREFIX pruned. That is safe here and must be stated in the docstring: `seen` is recomputed from the filesystem on every run, so the next sync converges; the import already commits per file throughout the walk (L1176–1190 and friends), so the run was never one transaction. The `summary.pruned` count is the count of documents actually deleted, unchanged in meaning.
+- **D6 — the E2E pins "nothing changed, at scale" (task 04).** `tests/e2e/test_sync_session_reset.py`: a fixture source of **more files than the reset cadence** (the module pins `BOR_SYNC_SESSION_RESET_FILES=3` with 10 files, so the cadence fires repeatedly mid-run), synced from the Sources page → every document present exactly once, the counts line identical to a `BOR_SYNC_SESSION_RESET_FILES=0` run of the same fixture (the byte-identical assertion the phase lives on), a second sync reports all-`unchanged` with `pruned=0`, no 500s and no network-error state on the page, and the KB totals in the Sources stats match the DB. Plus the prune path with the batch pin at 2 (so a multi-batch prune is exercised through the UI).
+- **NOT touched:** `app/db.py:30` (the global `expire_on_commit=False` — changing it would re-fetch rows across every request path for no benefit here), the importer's `owns_session` contract, the per-file index/summary/vision logic, the walk's column set (phase 185), `scripts/git_sync.py` (phase 186), the retrieval path, the frontend.
+
+## Dependencies
+- `183_sync_passive_delete_cascade` — required: batched `session.delete(doc)` without child hydration is the whole point of the batch cadence.
+
+## Tasks
+1. `01_identity_map_probe.md` — the deterministic identity-map probe (an `import_sources` run pinned to accumulate; the measurement recorded) plus the `sync_session_reset_files` / `sync_prune_batch` settings + catalogue + `.env.example` groundwork with the completeness pin green.
+2. `02_session_reset_cadence.md` — the `expunge_all` cadence at the per-file loop top + source boundaries, the probe flipped to the bounded assertion, the safety-argument comment.
+3. `03_batched_prune.md` — `_prune`'s light `load_only` select + per-batch commits + the atomicity note, the prune-side retention and statement-count pins.
+4. `04_e2e_bounded_sync.md` — `tests/e2e/test_sync_session_reset.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
+
+## Testing & Quality
+- **Unit:** the settings defaults + `0`-means-disabled semantics; `tests/unit/test_settings_catalog.py` completeness pin green with the two new fields in `"Import & sources"`; the existing prune guard pins (`images`-off skip, phase-126 parent rule, `changed_paths`) green as written.
+- **Integration:** `tests/integration/test_import_session_retention.py` (the identity-map probe, both cadences), `tests/integration/test_importer*.py`, `test_sync_api.py`, `test_source_removal_api.py` — byte-identical counts and log lines.
+- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_session_reset.py`, run **in isolation**.
+- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
+- **Lint/types:** `uv run ruff check . && uv run pyright`.
+
+## Documentation & Consistency
+- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
+- `README.md` + `.env.example`: the two new settings (default, `0` = pre-phase behaviour, what each bounds); one operations paragraph stating that the sync's memory is now bounded per-run rather than per-KB, and where the operator sets the ceiling.
+
+## Completion Criteria
+- [ ] `sync_session_reset_files=200` and `sync_prune_batch=500` exist on `Settings`, appear in `CATALOG` under `"Import & sources"`, in `.env.example`, in `README.md`; `uv run pytest tests/unit/test_settings_catalog.py -v` green.
+- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green with the POST-fix bound (identity-map size ≤ `reset_files + slack` with the cadence on, unbounded with it `0`) and both measured numbers in the module docstring.
+- [ ] `import_sources` expunges only at the two proven-safe points (documented in the code comment); `_prune` deletes light rows in `sync_prune_batch` commits with the phase-122/126 guards and `changed_paths` intact.
+- [ ] Every existing importer/sync/removal test is green with unchanged expectations (the byte-identical contract).
+- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
+- [ ] `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` green **in isolation**.
+- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
+- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(import): bound the sync session with a reset cadence and batched prune commits`); the phase dir moves to `.agents/phases/complete/`.
+
+## Locked decisions (owner-confirmed 2026-10-15)
+- **A1 — mechanism: periodic `expunge_all` + batched prune commits.** Chosen over "new session per source" (would re-derive the importer's `owns_session`/rollback semantics for a smaller win) and over "flip `expire_on_commit=True` for the sync session" (re-reads rows on every commit across a 27k-file walk and relies on implicit expiry instead of an explicit release).
+- **A2 — the prune's atomicity changes to per-batch, accepted.** `seen` is recomputed from disk each run so a partial prune converges on the next sync; the import already commits per file, so no run-level transaction existed to lose.
+- **A3 — both knobs default ON, `0` restores pre-phase behaviour exactly.** The operator keeps a no-redeploy rollback (the phase-167 `BOR_LOG_LARGE_RESPONSE_BYTES` precedent).
+- **A4 — no server-side cursors / `yield_per`.** A flush inside an open streaming result is a correctness trap; the light prune rows cost ~1 KB each, so materialising them is not the problem this phase is fixing.