phase: 184_sync_bounded_session
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure. **Verified / implemented (phase 184 final pass)** - All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries. - `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched. - `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact. **Tests / lint / coverage (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%). - `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**. - `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**. - Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed. - `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings. **Completion criteria** 1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green 2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅ 3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅ 4. Every existing importer/sync/removal test green with unchanged expectations → ✅ 5. Full suite green, coverage >90%, ruff + pyright clean → ✅ 6. Phase E2E green in isolation → ✅ 7. README + AGENTS accurate/consistent → ✅ 8. Atomic commit — left to the harness (I ran no git write commands). **Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written). **Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched. **Next pending phase:** `185_sync_defer_content_fast_path`.
This commit is contained in:
@@ -457,7 +457,11 @@ reflect the new files.
|
||||
its `chunks` rows (the embeddings) go together, the latter through the
|
||||
database's own `ON DELETE CASCADE`, so the cost of a mass prune is
|
||||
bounded by documents, not chunks (see *A mass prune costs documents,
|
||||
not chunks* under [Sync from the UI](#sync-from-the-ui)).
|
||||
not chunks* under [Sync from the UI](#sync-from-the-ui)). The CLI run
|
||||
follows the same two phase-184 knobs as the UI sync
|
||||
(`BOR_SYNC_SESSION_RESET_FILES`, `BOR_SYNC_PRUNE_BATCH`): it loads only
|
||||
the prune's six columns and commits every Nth delete (`0` on either knob
|
||||
restores the pre-phase-184 run).
|
||||
- The import prints one line per file and ends with a greppable summary,
|
||||
so it is safe to run from cron or after every commit.
|
||||
- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman
|
||||
@@ -651,6 +655,34 @@ only) runs the whole source refresh in one click:
|
||||
that fails the suite if the cascade clause ever disappears; the owner
|
||||
flow itself (delete upstream → re-sync → exactly those documents and
|
||||
chunks gone) is pinned by `tests/e2e/test_prune_cascade_cleanup.py`.
|
||||
- **A sync's memory is bounded per RUN, not per knowledge base** (phase
|
||||
184). One import run holds one session, and `expire_on_commit=False`
|
||||
(`app/db.py`) means a commit releases nothing — so before this phase the
|
||||
run kept every row it touched (each unchanged document with its full
|
||||
~15 KB `content`) until it finished, and the prune read every document of
|
||||
every pruned source whole, deletes held in ONE transaction. Now the walk
|
||||
releases its session every `BOR_SYNC_SESSION_RESET_FILES` files (plus once
|
||||
per source), and the prune (`_prune`) reads only the SIX columns its
|
||||
guards and its copy cleanup need (`id`, `source`, `path`, `is_image`,
|
||||
`image_path`, `page_count` — never `content`) ordered by id, committing
|
||||
every `BOR_SYNC_PRUNE_BATCH` deletes. **Where the operator sets the
|
||||
ceiling:** those two `BOR_`-prefixed settings (defaults `200` /
|
||||
`500`, live on the next sync without a restart, `0` on either one restores
|
||||
the exact pre-phase-184 run as the no-redeploy rollback). Counts, log
|
||||
lines, `changed_paths`, the phase-122/126 prune guards and the database
|
||||
are byte-identical either way — pinned by
|
||||
`tests/integration/test_import_session_retention.py` (the identity-map
|
||||
bound, the no-`content` prune `SELECT` with no per-row lazy load, the
|
||||
3-commits-vs-1 cadence, both prune guards on light rows) and end to end
|
||||
through the Sources page by `tests/e2e/test_sync_session_reset.py` — two
|
||||
module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
|
||||
`E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
|
||||
every document exactly once with the totals the stat cards show, the
|
||||
re-sync is all-`unchanged` with an unchanged chunk total, a five-document
|
||||
prune at batch `2` clears both folders with zero orphan chunk rows, and a
|
||||
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
|
||||
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
|
||||
line).
|
||||
- **Cancel**: while a run is in flight, a **Cancel** button appears
|
||||
next to the sync button; clicking it posts `POST /api/sync/cancel`
|
||||
(admin only) and registers a cooperative cancel (`202`) — the run
|
||||
@@ -1548,6 +1580,8 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
|
||||
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
|
||||
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
||||
| `BOR_SYNC_SESSION_RESET_FILES` | `200` | Files between releases of the import run's SQLAlchemy session (phase 184): every Nth file the walk hands back everything the session still holds (`expunge_all`), so an unchanged-file sync stops keeping one full document row per file walked. `0` = never release — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. The release also runs once per source, before the two post-passes |
|
||||
| `BOR_SYNC_PRUNE_BATCH` | `500` | Pruned documents per COMMIT inside the prune (phase 184) instead of one commit at the end of the whole import, so a mass prune releases its deletes as it goes instead of holding one enormous transaction. `0` = one commit at the end — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. With a non-zero batch a crash mid-prune leaves a **prefix** pruned, and the next sync converges (`seen` is recomputed from disk every run) |
|
||||
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
||||
|
||||
### Image support (phases 122/123/124/181)
|
||||
|
||||
Reference in New Issue
Block a user