brain-of-reese/.agents/phases/todo/184_sync_bounded_session/00_phase.md

Phase 184 — The sync's session stops growing with the knowledge base

Source: owner incident 2026-10-15 (as phase 183) — the import run held ONE session (app/db.py:30 sets expire_on_commit=False) for the whole walk + prune, so every row it touched stayed resident until the run ended. Story: n/a (incident-derived, owner directive 2026-10-15) Depends on: 183_sync_passive_delete_cascade (the batched prune deletes documents with the FK cascade doing the child work — it is only cheap because of passive_deletes=True).

Context: app/db.py — L30 SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False, future=True) (left EXACTLY as it is — every request path reads attributes after commit); app/rag/importer.py — import_sources (L708, owns_session = session is None at L878, session = SessionLocal() at L880, the total pre-walk at L884–899, the per-source loop at L901, the per-file loop at L916 with progress(source, rel, summary.files, total) at L943 and await _index_file(...) at L945, the EmbeddingError → session.rollback() handler at L951–958, the per-source post-passes _enrich_pending_images (L965) and _backfill_image_contexts (L972), _prune(...) (L979), session.close() in the finally at L984–985), _prune (L3860, docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all() at L3909, session.delete(doc) at L3931, the single commit at L3934–3935), _delete_image_copy (takes image_path: str | None), _delete_page_copies(doc, page_dir) (needs only doc.id + doc.page_count); app/api/sync.py — import_sources(...) called with no session= at L725 (the importer owns it — this phase never changes that contract); app/config.py (the BOR_-prefixed Settings pattern, chunk_target_chars at L228 as the shape), app/core/settings_catalog.py — GROUPS at L72 (the "Import & sources" group) and the CATALOG entry shape at L226, pinned complete by tests/unit/test_settings_catalog.py::test_catalog_covers_exactly_model_fields (set(CATALOG) == set(Settings.model_fields) — a new Settings field without a catalogue row FAILS the build); tests/unit/test_importer.py, tests/integration/test_importer_e2e.py, tests/integration/test_sync_api.py, tests/e2e/test_sync_changed_only.py, tests/e2e/test_sync_summary_visibility.py.

Measured baseline (2026-10-15, this repo's dev DB, production-shaped rows)

• A Document's content averages 15.2 KB (17,407 docs, 252.6 MB of markdown in the incident DB), and the identity map keeps every one the walk touched — see phase 185 for the column-level fix.
• Objects are released by NOTHING during the run: expire_on_commit=False means a commit does not drop attributes, and the importer never expunges. The identity map is therefore the run's memory profile, and it is unbounded: it scales with (files walked) + (chunks created) + (chunks hydrateable per prune — phase 183).

Objective

Bound the import run's resident set independently of the size of the knowledge base: session.expunge_all() on a fixed cadence at the one point where no previously-loaded object is reachable, and _prune released in committed batches instead of one transaction at the end of the whole import — with every count, log line, guard, cancel boundary and summary number byte-identical.

Design (shared by all tasks — the executor reads this, not the chat)

• D1 — the pin is the identity map, not RSS (task 01). len(session.identity_map) is exact, deterministic and it IS the retention surface. tests/integration/test_import_session_retention.py runs an import of N files through import_sources (the real entry point, the test_importer_e2e.py fixture shapes + the mock-embedder pattern from tests/unit/test_importer.py) and asserts the map size at the end of the run: PRE-fix it is ≈ docs + chunks + documents-touched (record the measured number), and the test pins >= N to prove accumulation. The post-fix assertion (task 02) is <= reset_files + slack. RSS is recorded in the docstring only, never asserted (phase 183 A4's rule, inherited).
• D2 — one new setting per new cadence, catalogue-compliant (tasks 02 + 03). sync_session_reset_files: int = 200 (BOR_SYNC_SESSION_RESET_FILES) and sync_prune_batch: int = 500 (BOR_SYNC_PRUNE_BATCH). 0 means "never reset / one transaction" = the exact pre-phase behaviour, the operator's rollback knob without a re-deploy (the BOR_LOG_LARGE_RESPONSE_BYTES precedent). Both get SettingMeta entries in the "Import & sources" group (the completeness pin makes this mandatory, not optional), both documented in .env.example and README.md, neither restart_required (the importer reads llm.settings per run) and neither secret.
• D3 — the reset point is the top of the per-file loop, and nowhere else (task 02). Inside import_sources, at the top of the L916 loop body — after the PREVIOUS file's _index_file has returned, before this file's progress(...) hook at L943 — call session.expunge_all() every sync_session_reset_files files (summary.files % reset == 0), plus once at each source boundary before the two post-passes (L965/L972) so a source's walk residue never crosses into the next source. Proof of safety that the task must state in the comment: no object from a finished file is reachable afterwards (_index_file returns None, the progress/chunk/image hooks only assign fields on _status, dates_map/ignore are plain data, and total was computed before the loop), and the EmbeddingError handler's session.rollback() (L951–958) is safe with detached objects because it touches none. The two post-passes select their own rows fresh (L3241, L3391), so they are unaffected.
• D4 — the prune is batched on light rows, not streamed cursors (task 03). _prune replaces select(Document).where(...).all() with select(Document).where(...).options(load_only(Document.id, Document.source, Document.path, Document.is_image, Document.image_path, Document.page_count)).order_by(Document.id) — the exact column set the guards and the copy cleanup need (is_image + path for the phase-122 toggle guard and the phase-126 _EMBEDDED_IMAGE_RE parent rule, image_path for _delete_image_copy, id + page_count for _delete_page_copies) — and commits every sync_prune_batch deletes instead of once at the end (sync_session_reset_files's sibling knob, 0 = the pre-phase single commit). Deletes stay session.delete(doc) (the phase-183 passive_deletes makes them child-load-free; a bulk delete() is NOT used, so changed_paths, the log line and the guards keep their exact per-document shape). yield_per/server-side cursors are explicitly rejected: a flush inside an open streaming result is a trap, and the light rows are ~1 KB each.
• D5 — batch commits change the prune's atomicity, deliberately (A2). Pre-phase a crash mid-prune rolled the whole prune back; post-phase a crash leaves a PREFIX pruned. That is safe here and must be stated in the docstring: seen is recomputed from the filesystem on every run, so the next sync converges; the import already commits per file throughout the walk (L1176–1190 and friends), so the run was never one transaction. The summary.pruned count is the count of documents actually deleted, unchanged in meaning.
• D6 — the E2E pins "nothing changed, at scale" (task 04). tests/e2e/test_sync_session_reset.py: a fixture source of more files than the reset cadence (the module pins BOR_SYNC_SESSION_RESET_FILES=3 with 10 files, so the cadence fires repeatedly mid-run), synced from the Sources page → every document present exactly once, the counts line identical to a BOR_SYNC_SESSION_RESET_FILES=0 run of the same fixture (the byte-identical assertion the phase lives on), a second sync reports all-unchanged with pruned=0, no 500s and no network-error state on the page, and the KB totals in the Sources stats match the DB. Plus the prune path with the batch pin at 2 (so a multi-batch prune is exercised through the UI).
• NOT touched: app/db.py:30 (the global expire_on_commit=False — changing it would re-fetch rows across every request path for no benefit here), the importer's owns_session contract, the per-file index/summary/vision logic, the walk's column set (phase 185), scripts/git_sync.py (phase 186), the retrieval path, the frontend.

Dependencies

• 183_sync_passive_delete_cascade — required: batched session.delete(doc) without child hydration is the whole point of the batch cadence.

Tasks

1. 01_identity_map_probe.md — the deterministic identity-map probe (an import_sources run pinned to accumulate; the measurement recorded) plus the sync_session_reset_files / sync_prune_batch settings + catalogue + .env.example groundwork with the completeness pin green.
2. 02_session_reset_cadence.md — the expunge_all cadence at the per-file loop top + source boundaries, the probe flipped to the bounded assertion, the safety-argument comment.
3. 03_batched_prune.md — _prune's light load_only select + per-batch commits + the atomicity note, the prune-side retention and statement-count pins.
4. 04_e2e_bounded_sync.md — tests/e2e/test_sync_session_reset.py (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.

Testing & Quality

• Unit: the settings defaults + 0-means-disabled semantics; tests/unit/test_settings_catalog.py completeness pin green with the two new fields in "Import & sources"; the existing prune guard pins (images-off skip, phase-126 parent rule, changed_paths) green as written.
• Integration: tests/integration/test_import_session_retention.py (the identity-map probe, both cadences), tests/integration/test_importer*.py, test_sync_api.py, test_source_removal_api.py — byte-identical counts and log lines.
• E2E (AGENTS rule 4, task 04): tests/e2e/test_sync_session_reset.py, run in isolation.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).
• Lint/types: uv run ruff check . && uv run pyright.

Documentation & Consistency

• Final pass reviews README.md and AGENTS.md in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
• README.md + .env.example: the two new settings (default, 0 = pre-phase behaviour, what each bounds); one operations paragraph stating that the sync's memory is now bounded per-run rather than per-KB, and where the operator sets the ceiling.

Completion Criteria

• sync_session_reset_files=200 and sync_prune_batch=500 exist on Settings, appear in CATALOG under "Import & sources", in .env.example, in README.md; uv run pytest tests/unit/test_settings_catalog.py -v green.
• uv run pytest tests/integration/test_import_session_retention.py -v green with the POST-fix bound (identity-map size ≤ reset_files + slack with the cadence on, unbounded with it 0) and both measured numbers in the module docstring.
• import_sources expunges only at the two proven-safe points (documented in the code comment); _prune deletes light rows in sync_prune_batch commits with the phase-122/126 guards and changed_paths intact.
• Every existing importer/sync/removal test is green with unchanged expectations (the byte-identical contract).
• uv run pytest green, app/ coverage >90%, ruff + pyright clean.
• uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov green in isolation.
• README.md and AGENTS.md accurate and consistent with the whole project (verified in the final pass).
• One atomic --no-gpg-sign Conventional Commit (e.g. perf(import): bound the sync session with a reset cadence and batched prune commits); the phase dir moves to .agents/phases/complete/.

Locked decisions (owner-confirmed 2026-10-15)

• A1 — mechanism: periodic expunge_all + batched prune commits. Chosen over "new session per source" (would re-derive the importer's owns_session/rollback semantics for a smaller win) and over "flip expire_on_commit=True for the sync session" (re-reads rows on every commit across a 27k-file walk and relies on implicit expiry instead of an explicit release).
• A2 — the prune's atomicity changes to per-batch, accepted. seen is recomputed from disk each run so a partial prune converges on the next sync; the import already commits per file, so no run-level transaction existed to lose.
• A3 — both knobs default ON, 0 restores pre-phase behaviour exactly. The operator keeps a no-redeploy rollback (the phase-167 BOR_LOG_LARGE_RESPONSE_BYTES precedent).
• A4 — no server-side cursors / yield_per. A flush inside an open streaming result is a correctness trap; the light prune rows cost ~1 KB each, so materialising them is not the problem this phase is fixing.