phase: 183_sync_passive_delete_cascade

All gates green. Final state verified. **Implemented / verified** - Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after. - Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings. - **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`). **Gates** - `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%) - `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed - `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated) - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors **Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands). **Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit). **Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify. **Next pending phase**: `184_sync_bounded_session`.
2026-10-15 16:00:28 -04:00
parent b34b8ca672
commit 7b5ea7c2ee
52 changed files with 3044 additions and 19 deletions
@@ -0,0 +1,57 @@
+# Phase 183 — Mass prune must not load the embeddings it is deleting
+
+**Source:** owner incident 2026-10-15 — "brain-of-code hit 4 GB of RAM while running the sync" (the sync `started repos=4` at 11:19:23, the run reached `_prune` at 12:14:14, pruned **3,008 litellm docs**, and the VM locked up hard enough that boot −1 has no shutdown record at all).
+**Story:** n/a (incident-derived, owner directive 2026-10-15)
+**Context:** `app/models.py` — `Document.chunks` (**L227–229**, `relationship(back_populates="document", cascade="all, delete-orphan")` — **no `passive_deletes`**), `Chunk` (L232), `Chunk.embedding` (L241, `Vector(EMBEDDING_DIM)` = 768), `Chunk.document` (L247); `app/rag/importer.py` — `_prune` (**L3860**, the one-shot `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, the per-doc `session.delete(doc)` at **L3931**, the single `if pruned: session.commit()` at **L3934–3935**), `_replace_content_chunks` (L992, the `doc.chunks = [...]` ORPHAN-removal path at L1033), `_store_summary` (L1595, the `for old in [c for c in doc.chunks if c.is_summary]` loop at L1648); `app/api/git_sources.py` — `delete_git_source` (L855, the `db.delete(doc)` loop at **L926**, the docstring's "chunks + embeddings via the `all, delete-orphan` cascade" line at **L863**, `docs_pruned` at L919); `app/api/docs.py` — the document-delete endpoint's `db.delete(row)` (L700); `alembic/versions/0001_initial_schema.py` — **L54 `sa.ForeignKey("documents.id", ondelete="CASCADE")`** (the DB already cascades — production confirms: `chunks_document_id_fkey`, `confdeltype='c'`); `tests/unit/test_importer.py` + `tests/integration/test_importer_e2e.py` (the importer families), `tests/integration/test_source_removal_api.py` + `tests/e2e/test_source_removal_cleanup.py` (the removal families), `tests/e2e/test_force_push_resync.py` (the "upstream deleted the file, re-sync prunes it" shape).
+
+## Measured baseline (the numbers this phase is written against; recorded 2026-10-15)
+- A loaded 768-dim `Chunk.embedding` costs **24.8 KiB of Python per chunk** (list header + 768 float objects) against **3 KiB on disk** — the ORM round-trip is an 8× expansion, and pgvector hands back a `list[float]`, never a buffer.
+- `session.delete(doc)` on a relationship **without** `passive_deletes` makes SQLAlchemy load the child collection first (it must know which children to orphan/delete). Measured against the dev DB with production-shaped rows (400 docs × 25 chunks, ~745-char chunk content): **RSS grows linearly at ~0.24 MB per document deleted** = 25 chunks × ~9.7 KiB resident, +241 MB over 400 deletes, and NOTHING is released before the single commit because `app/db.py:30` builds sessions with `expire_on_commit=False`.
+- Applied to the incident run: 3,008 docs × ~24 chunks ≈ **720 MB of float lists materialised to delete 72k rows**, on top of the ~300 MB the same walk already retained — on a 3,904 MB VM with **no swap**, sharing RAM with a 3.1 GB Postgres.
+
+## Objective
+Make a mass prune cost O(documents) instead of O(chunks × 24.8 KiB): `Document.chunks` declares `passive_deletes=True` so the DB's already-present `ON DELETE CASCADE` removes the chunk rows without the ORM ever hydrating them, on both mass-delete surfaces (the importer's prune and the admin's source removal), with every user-visible behaviour (counts, log lines, image/page-copy cleanup, the phase-122/126 prune guards, the `changed_paths` feed into the folder summaries) byte-identical.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+- **D1 — the pin is statement count, not RSS (task 01).** Memory assertions are flaky; the SQLAlchemy `before_cursor_execute` event stream is not. A new `tests/integration/test_prune_cascade_statements.py` counts, for one prune, every SELECT whose SQL targets the `chunks` table. PRE-fix the count is **exactly one per pruned document** (the lazy child collection load) and that test is GREEN before any code change — it IS the demonstration, exactly as phase 179 task 01 pinned the freeze with a latency probe. The measured RSS delta (the ~0.24 MB/doc figure above) is recorded in the module docstring as the corroborating number, asserted only as a loose ceiling (`< 40 MB` for a 200-doc fixture, where pre-fix measures ≈ 124 MB) so a noisy CI box cannot fail the suite on allocator behaviour.
+- **D2 — the fix is one keyword, and it is a semantic claim about the schema (task 02).** `app/models.py:227–229` becomes `relationship(back_populates="document", cascade="all, delete-orphan", passive_deletes=True)`. That is only correct because the FK really cascades — so task 02 also adds a **schema-drift guard test** that reads the live constraint (`pg_constraint` where `conrelid='chunks'::regclass and contype='f'` → `confdeltype == 'c'`) and fails loudly if a future migration ever drops `ON DELETE CASCADE`, because at that point `passive_deletes=True` would silently orphan chunk rows. **No new alembic revision** — 0001 L54 already declares it, and the incident box's live schema matches (A13 stays satisfied because nothing changes).
+- **D3 — both mass-delete surfaces, one keyword's blast radius (task 02).** With `passive_deletes=True` the ORM stops hydrating children for EVERY `session.delete(document)`: the importer's `_prune` (L3931), `delete_git_source`'s `db.delete(doc)` loop (`app/api/git_sources.py:926` — the endpoint that took 47 s for 2,303 immich docs on 2026-10-15 11:18:30), and `app/api/docs.py:700`. Audit-and-note, not change: none of the three reads `doc.chunks` AFTER deleting it (verified: `_prune` only touches `image_path`/`page_count`/`source`/`path`; the git-sources endpoint only counts; the docs endpoint only logs). The `doc.chunks = [...]` reassignment inside `_replace_content_chunks` (L1033) is the **orphan-removal** path, which `passive_deletes` does NOT short-circuit — the collection still loads there (a re-embed needs the old rows gone and the new ones built); it runs only for CHANGED files (217 in the incident run), and it is explicitly OUT OF SCOPE (A2) — the docstring says so in one line so the next reader does not "discover" it.
+- **D4 — the cascade contract is verified at the row level, not assumed (tasks 01 + 03).** Every prune test asserts `select(count()).select_from(Chunk).where(Chunk.document_id.in_(deleted ids)) == 0` — no orphan chunk rows, before and after the change — plus the existing `documents`-level assertions. This is the phase's non-negotiable: a delete that stops hydrating children must still delete the children.
+- **D5 — the E2E pins the owner-visible prune (task 03).** `tests/e2e/test_prune_cascade_cleanup.py` (this phase's dedicated Playwright file, AGENTS rule 4), the `test_force_push_resync.py` + `test_source_removal_cleanup.py` house shapes (dedicated module app, mock LLM, `BOR_SOURCES_DIR`/`BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated before AND after each test): a fixture source of N files is synced to `success` from the Sources page, a subset is deleted from disk upstream, the re-sync's counts line reports `pruned=N-subset` and those documents vanish from the KB (the ls/tree row is gone, `GET /api/docs` 404s the path, the Documents list total drops), while the chunk rows are gone too (the test's own DB query — the same D4 assertion, end to end) and the surviving documents still answer a chat question (a partial prune never damages the RAG). `E2E_REAL_LLM=1` skipped.
+- **NOT touched:** the `_prune` batching (phase 184), the fast-path column set (phase 185), `expire_on_commit` (deliberately left `False` — every request path relies on reading attributes after commit), `Chunk.embedding`'s type, the retrieval query, the frontend, and any LLM/vision path.
+
+## Dependencies
+— (none: the app is fully functional at the head of `complete/`; phases 184 and 185 build on this one, in that order)
+
+## Tasks
+1. `01_prune_cascade_probe.md` — the deterministic statement-count probe (one `chunks` SELECT per pruned doc, PRE-fix, green before the fix) + the row-level orphan assertions + the measured RSS delta recorded in the module docstring.
+2. `02_passive_deletes.md` — the fix (`passive_deletes=True`), the live-FK drift guard, the three delete-site audit, the probe flipped to the post-fix assertion (`0` chunk SELECTs), the git-sources endpoint docstring correction.
+3. `03_e2e_prune_cascade.md` — `tests/e2e/test_prune_cascade_cleanup.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
+
+## Testing & Quality
+- **Integration:** `tests/integration/test_prune_cascade_statements.py` (the statement-count probe, the orphan-row assertions, the loose RSS ceiling + the recorded measurement).
+- **Unit:** the existing importer prune pins stay green as written — the phase-122 images-off prune guard, the phase-126 parent-PDF rule, `changed_paths` population, the `summary.pruned` count (`tests/unit/test_importer.py`, `tests/unit/test_importer_ignore.py`); add a unit pin that `Document.__mapper__.relationships["chunks"].passive_deletes is True`.
+- **Regression:** `tests/integration/test_importer.py`, `test_importer_e2e.py`, `test_source_removal_api.py`, `test_sync_api.py`, `test_git_sources_api.py` — all green, byte-identical counts and log lines.
+- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_prune_cascade_cleanup.py`, run **in isolation**.
+- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
+- **Lint/types:** `uv run ruff check . && uv run pyright`.
+
+## Documentation & Consistency
+- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
+- `README.md`: the operations note that a prune deletes chunk rows in the DATABASE (FK cascade) and that the operator-visible cost of a mass prune is bounded by documents, not chunks (the number the incident was missing).
+- The `_prune` docstring (`app/rag/importer.py:3860`) gains the one-paragraph "why `passive_deletes` + the FK cascade, and why the re-embed path is different" note; `app/models.py:227` gains the one-line "do not remove without re-checking the FK's ON DELETE CASCADE (pinned by tests/integration/test_prune_cascade_statements.py)" warning.
+
+## Completion Criteria
+- [ ] `app/models.py::Document.chunks` declares `passive_deletes=True`; a unit pin asserts it.
+- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with the POST-fix assertion (zero `chunks` SELECTs during a prune of N docs, `0` orphan chunk rows, RSS delta under the pinned ceiling) and the pre/post measurements + the incident reference in the module docstring.
+- [ ] The live-FK drift guard is green and would fail if `ON DELETE CASCADE` were absent (proven by the test's own negative fixture or a documented, executed one-off check).
+- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
+- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB still answers).
+- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
+- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`); the phase dir moves to `.agents/phases/complete/`.
+
+## Locked decisions (owner-confirmed 2026-10-15)
+- **A1 — mechanism: `passive_deletes=True`, not a bulk-`delete()` rewrite.** The relationship keyword keeps `session.delete(doc)` as the one delete shape every caller already uses (importer prune, source removal, doc delete), so there is no new control flow to get wrong; the bulk-rewrite variant belongs to phase 184's batching, which builds on this.
+- **A2 — out of scope: the `doc.chunks = [...]` orphan path** (`_replace_content_chunks` L1033). `passive_deletes` does not cover collection replacement, and a re-embed genuinely needs the loaded collection; it runs for changed files only. Documented as a known, bounded, deliberate non-fix.
+- **A3 — no migration.** The FK's `ON DELETE CASCADE` ships in 0001 L54 and is verified live; this phase adds a drift guard instead of a schema change.
+- **A4 — the probe is statement-count based.** Deterministic, CI-safe, and it names the exact waste; the RSS figure is documentation plus a loose ceiling, never the assertion that can flake a build.
@@ -0,0 +1,29 @@
+# Task 01 — Pin what a prune actually loads (the probe, green PRE-fix)
+
+**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
+**Story:** n/a
+
+## Objective
+A deterministic integration probe that shows — before any code changes — that deleting documents hydrates their chunk rows (and therefore every 768-dim embedding) into Python, one SELECT per document. The probe is GREEN pre-fix; task 02 flips it. The measured cost is recorded in the module docstring so the next reader never has to re-derive it.
+
+## Work
+1. `tests/integration/test_prune_cascade_statements.py` — a NEW file, house patterns:
+   - Session/DB fixtures per `tests/integration/test_importer.py` (the shared test DB, the per-test truncate, `SessionLocal` from `app.db`).
+   - A `seed_docs` helper: **200** `Document` rows, `source="cascade-probe"`, each with **25** `Chunk` rows whose `content` is ~745 chars of fixed text and whose `embedding` is a deterministic 768-float vector (`[i / 768 for i in range(768)]` rotated by the doc index — deterministic, and L2-normalised enough not to matter here). Commit.
+   - **The statement capture:** a `sqlalchemy.event` listener on the engine's `before_cursor_execute` that appends `(statement, params)` tuples for the duration of the test; a helper `chunk_selects(captured)` returns the statements that are `SELECT` and reference `"FROM chunks"` (case-insensitive, `chunks` word-boundary so `documents` never matches). Install in the fixture, remove in teardown (`event.remove`) so no other test in the session sees the listener.
+   - **The probe test** `test_prune_loads_every_chunk_collection`: build the prune inputs the way `_prune` receives them (`source_names={"cascade-probe"}`, `seen=set()` → everything prunes), call `app.rag.importer._prune(session, source_names, seen, images=False, summary=ImportSummary(), page_dir=None)` (the real signature, `app/rag/importer.py:3860`), then assert:
+     - `chunk_selects(captured) == 200` — **exactly one child-collection SELECT per deleted document.** This is the PRE-fix assertion and it is GREEN now. Mark it so the flip is unmissable: a module-level `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC` constant with the comment `# 1 pre-phase-183 (the lazy child load); 0 post-phase-183 — task 02 flips this`.
+     - every `Chunk` row of the 200 documents is gone (`select(count()).select_from(Chunk).where(Chunk.document_id.in_(ids)) == 0`) — the D4 no-orphan contract, asserted before the fix so post-fix it means "unchanged, and not by accident".
+     - the `documents` rows are gone, `_prune`'s return value is 200, and `ImportSummary.changed_paths` holds the 200 `(source, path)` pairs.
+   - **The loose memory ceiling** `test_prune_memory_delta_has_a_ceiling`: the same 200-doc prune with `VmRSS` read from `/proc/self/status` before and after (a tiny local helper inside the test module — no new `app/` code in this task), asserting the delta is **< 40 MB**, with the MEASURED value recorded in the docstring. Pre-fix this measures ≈ **48 MB** for 200 docs (200 × 25 × 24.8 KiB ≈ 124 MB of float lists, of which the allocator keeps what it can reuse — the point is the order, not the exact figure), so the pinned ceiling must be picked from the ACTUAL measured pre-fix number: set it to roughly **one third** of what the pre-fix run measures, i.e. a ceiling the pre-fix run VIOLATES if it can be made to (if the pre-fix delta comes in under 40 MB on this box, LOWER the ceiling to a value the post-fix run clears with room and document both numbers — the assertion's job is to catch the hydrate-the-embeddings behaviour, never to fail on allocator noise). If the delta cannot be made discriminating on this hardware, keep the test but assert only `< 40 MB` and state in the docstring that the statement count is the load-bearing pin.
+2. **The module docstring (the measurement record):** the incident in one paragraph (the 2026-10-15 sync pruned 3,008 litellm docs; the app cgroup peaked at 3.3 GB twice that week; the VM has 3,904 MB, no swap, and shares RAM with a 3.1 GB Postgres); the mechanism (`app/models.py:227–229` has no `passive_deletes`, so `session.delete(doc)` must load the children; `app/db.py:30` sets `expire_on_commit=False`, so nothing is released before the single commit at `importer.py:3934`); the measured unit costs (**24.8 KiB** of Python per 768-dim embedding vs 3 KiB on disk; **~0.24 MB resident per document deleted** at 25 chunks/doc, measured 2026-10-15 against the dev DB with production-shaped rows); this test's own measured numbers (fill them in from the run); and the verdict line **CONFIRMED — one chunk SELECT per pruned document** (if the probe comes back with 0 selects, STOP and flag it in the docstring and the phase report instead of proceeding).
+3. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green; `uv run pytest` green (the event listener is scoped to the fixture, nothing else sees it).
+
+## Testing & Quality
+- Integration: the probe family above (statement count, orphan-row check, return value, `changed_paths`, the loose RSS ceiling).
+- Coverage: **>90%** on `app/` — test-only task, `app/` untouched, so the gate must simply stay green.
+
+## Completion Criteria
+- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 1` (the pre-fix assertion) and the 0-orphan-chunks assertion.
+- [ ] The module docstring records the incident, the mechanism, the 24.8 KiB / 0.24 MB-per-doc unit costs, and this run's measured statement count + RSS delta, plus the **CONFIRMED** verdict.
+- [ ] `uv run pytest` green; no `app/` change in this task.
@@ -0,0 +1,52 @@
+# Task 02 — The fix: `passive_deletes=True`, with the FK contract pinned
+
+**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
+**Story:** n/a
+
+## Objective
+Stop the ORM from hydrating chunk rows (and their 768-dim embeddings) when a `Document` is deleted: declare `passive_deletes=True` on `Document.chunks` so the database's existing `ON DELETE CASCADE` does the work, prove the schema really cascades with a drift guard, and flip task 01's probe to the post-fix assertion.
+
+## Work
+1. `app/models.py` — `Document.chunks` (**L227–229**):
+   ```python
+   chunks: Mapped[list[Chunk]] = relationship(
+       back_populates="document",
+       cascade="all, delete-orphan",
+       # Phase 183: deleting a Document removes its chunks IN THE DATABASE
+       # (chunks_document_id_fkey is ON DELETE CASCADE — alembic 0001 L54,
+       # pinned live by tests/integration/test_prune_cascade_statements.py).
+       # WITHOUT this keyword the ORM loads every child before deleting it,
+       # which hydrates every 768-dim embedding (~24.8 KiB of Python each):
+       # a 3,008-document prune cost ~720 MB and took the 2026-10-15 VM
+       # (3,904 MB, no swap) down. Do not remove this without re-checking
+       # the FK — the drift guard fails the suite if the cascade is gone.
+       passive_deletes=True,
+   )
+   ```
+   Nothing else in `models.py` changes (`Chunk.document` L247, `Chunk.embedding` L241 untouched).
+2. `app/rag/importer.py` — the `_prune` docstring (L3860): append the phase-183 paragraph — the children go through the FK cascade, so no chunk row is ever SELECTed here; the `changed_paths` / image-copy / page-copy cleanup is unchanged; and the **A2 note**: the re-embed path `_replace_content_chunks` (L992, `doc.chunks = [...]` at L1033) is collection REPLACEMENT, which `passive_deletes` does not cover and which legitimately needs the loaded collection — a known bounded non-fix.
+3. `app/api/git_sources.py` — `delete_git_source`'s docstring (**L863**, "documents (chunks + embeddings via the `all, delete-orphan` cascade)") → correct it to name the FK cascade (phase 183): the rows go in the database, the app never loads them. No logic change in the endpoint (verified: the loop at L919–927 only counts `docs_pruned`, it never reads `doc.chunks`).
+4. `tests/integration/test_prune_cascade_statements.py` — flip the probe: `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, comment updated (`0 post-phase-183 — the FK cascade; 1 pre-phase-183 was the lazy child load`). Keep every other assertion (0 orphan chunks, 0 documents, the return value, `changed_paths`, the RSS ceiling — now clearing it by a wide margin) and extend the docstring: `CONFIRMED → FIXED (phase 183, passive_deletes=True)` with the post-fix measured statement count and RSS delta.
+5. **The drift guard** — same file, `test_chunks_fk_cascades_at_database_level`: query the live schema and assert the cascade exists, so `passive_deletes=True` can never silently orphan rows.
+   ```python
+   row = session.execute(
+       text("select confdeltype from pg_constraint "
+            "where conrelid = 'chunks'::regclass and contype = 'f'")
+   ).all()
+   assert row and all(r[0] == "c" for r in row)   # 'c' = ON DELETE CASCADE
+   ```
+   Prove the guard is real (not a tautology) in the task's own verification: run it once against a scratch DB where the constraint is recreated `ON DELETE NO ACTION` (a throwaway database the task creates and drops — never the shared test DB) and record in the test's comment that the guard fails there, plus a unit pin `Document.__mapper__.relationships["chunks"].passive_deletes is True` in `tests/unit/test_importer.py`.
+6. **Audit note (no code change):** grep `session.delete` / `db.delete` for the `Document` deletes (`app/rag/importer.py:3931`, `app/api/git_sources.py:926`, `app/api/docs.py:700`) and confirm none reads `doc.chunks` after the delete; record the confirmation in the task's completion note. If any site DOES read the collection after deleting, stop and report it rather than papering over it.
+7. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green post-fix; then the families the keyword touches — `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/integration/test_sync_api.py tests/integration/test_git_sources_api.py -v`; then `uv run pytest`.
+
+## Testing & Quality
+- Integration: the flipped probe + the live-schema drift guard + the five families above, byte-identical counts and log lines.
+- Unit: the `passive_deletes` mapper pin in `tests/unit/test_importer.py`.
+- Coverage: **>90%** on `app/` (the `app/` change is one keyword + docstrings).
+
+## Completion Criteria
+- [ ] `app/models.py::Document.chunks` has `passive_deletes=True` with the explanatory comment; the unit mapper pin is green.
+- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, zero orphan chunks, and the docstring's CONFIRMED → FIXED record with both measured numbers.
+- [ ] The drift guard is green on the test DB and documented as failing on a scratch DB whose FK lacks `ON DELETE CASCADE`.
+- [ ] `_prune`'s and `delete_git_source`'s docstrings match the new behaviour (the A2 re-embed note included); the three-delete-site audit is recorded.
+- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,34 @@
+# Task 03 — E2E: a mass prune leaves nothing behind (the phase's dedicated Playwright suite)
+
+**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
+**Story:** n/a
+
+## Objective
+Pin the fix against the real pipeline the owner actually drives: upstream deletes files from a synced source, the admin re-syncs from the Sources page, and the prune removes exactly the vanished documents, leaves **zero orphan chunk rows**, keeps the surviving KB answerable, and reports the same counts line it always did. Then run the full gates, do the docs pass, and commit the phase.
+
+## Work
+1. `tests/e2e/test_prune_cascade_cleanup.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), following the `tests/e2e/test_force_push_resync.py` + `tests/e2e/test_source_removal_cleanup.py` house shapes point for point:
+   - **Isolation:** a per-module app on a dedicated port (the conftest session app is never started in the isolated run; `tests/e2e/conftest.py` — `APP_PORT` L32, `app_server` L106, `page` L219), the mock LLM on its conftest port, a dedicated `BOR_SOURCES_DIR` and `BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated **before AND after** every test, admin login via `tests/e2e/auth_helpers.py`.
+   - **Fixture:** a `prune_source` fixture dir with **9** importable `.md` files across two folders (`notes/keep-1..3.md`, `notes/gone-1..6.md`), one of the `keep-*` files carrying the sentinel phrase the chat test asks about.
+   - **Test 1 `test_deleted_files_prune_without_orphans`:** sync → assert the run reaches `success` and the Sources counts line reports `added=9` / `pruned=0` → delete the six `gone-*.md` files from disk → re-sync → assert the counts line reports `pruned=6` and `added=0` (the unchanged 3 stay unchanged) → the six documents are gone from the KB (the ls/tree row for `notes/gone-1.md` is absent, `GET /api/docs` for the path 404s via `page.request`, the Documents total is 3) → **zero orphan chunk rows** for the deleted document ids (the test's own `psycopg`/SQLAlchemy query against the E2E DB, the D4 assertion) and `select(count()).select_from(Chunk)` equals the sum of the 3 survivors' chunk counts → the run leaves no `documents` row whose `source` lost its file.
+   - **Test 2 `test_surviving_kb_still_answers_after_a_partial_prune`:** with the same pruned state, ask the chat the sentinel question (the `test_chat_rag.py` shape) and assert the answer cites the surviving sentinel document and the turn completes (no error state, no empty answer) — the prune did not damage retrieval.
+   - **Test 3 `test_source_removal_prunes_every_document_of_the_source`:** the admin path (`delete_git_source`, `app/api/git_sources.py:855`) through the Git sources UI or `page.request` `DELETE /api/git-sources/{id}` → every document AND every chunk row of that source is gone (the same orphan assertion), `docs_pruned` matches the seeded count via the log/response contract the endpoint already exposes, and the remaining source is intact.
+   - The module docstring: the incident reference, the fix (`passive_deletes=True` + the FK cascade), the test→observable mapping, the run-in-isolation command, the prerequisites (DB up: `podman compose up -d db`).
+2. **Docs pass (the phase's final documentation review):** `README.md` + `AGENTS.md` reviewed in full against the post-phase app and every inconsistency fixed (pre-existing included). Add to `README.md`'s operations/sync section: a prune deletes chunk rows through the database cascade, so the cost of a mass prune scales with documents, not embeddings — and the `chunks` FK's `ON DELETE CASCADE` is load-bearing (pinned by `tests/integration/test_prune_cascade_statements.py`), so it must never be dropped.
+3. **The full gates:**
+   - `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` — green **in isolation**.
+   - `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` and `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` — green in isolation (the two prune/removal E2E families this phase must not have broken; run them AFTER the new suite so the DB truncate ordering stays clean).
+   - `uv run pytest --cov=app --cov-report=term-missing` — the whole suite green, `app/` coverage **>90%**.
+   - `uv run ruff check . && uv run pyright` — clean.
+4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/models.py`, the two docstring updates, the new integration + E2E files, the unit pin and the `.agents/` tree — e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`, body naming the 2026-10-15 incident and the measured 24.8 KiB/embedding, ~0.24 MB/doc. Move the phase dir to `.agents/phases/complete/183_sync_passive_delete_cascade/`.
+
+## Testing & Quality
+- E2E: the dedicated suite above + the two regression E2E families in isolation.
+- Coverage: **>90%** on `app/`; ruff + pyright clean.
+
+## Completion Criteria
+- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB answers, the source-removal variant clean).
+- [ ] `tests/e2e/test_source_removal_cleanup.py` and `tests/e2e/test_force_push_resync.py` green in isolation.
+- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
+- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the cascade-is-load-bearing note included).
+- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.