Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 184_sync_bounded_session
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.

**Verified / implemented (phase 184 final pass)**
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.

**Tests / lint / coverage (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria**
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
6. Phase E2E green in isolation → ✅
7. README + AGENTS accurate/consistent → ✅
8. Atomic commit — left to the harness (I ran no git write commands).

**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).

**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.

**Next pending phase:** `185_sync_defer_content_fast_path`.
This commit is contained in:
ducoterra
2026-10-15 23:19:27 -04:00
parent 7b5ea7c2ee
commit 065304b52f
40 changed files with 3512 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/184_sync_bounded_session/00_phase.md → .agents/phases/complete/184_sync_bounded_session/00_phase.md
View File
.agents/phases/todo/184_sync_bounded_session/01_identity_map_probe.md → .agents/phases/complete/184_sync_bounded_session/01_identity_map_probe.md
View File
.agents/phases/todo/184_sync_bounded_session/02_session_reset_cadence.md → .agents/phases/complete/184_sync_bounded_session/02_session_reset_cadence.md
View File
.agents/phases/todo/184_sync_bounded_session/03_batched_prune.md → .agents/phases/complete/184_sync_bounded_session/03_batched_prune.md
View File
.agents/phases/todo/184_sync_bounded_session/04_e2e_bounded_sync.md → .agents/phases/complete/184_sync_bounded_session/04_e2e_bounded_sync.md
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a1.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a1.md
+1
View File
@@ -0,0 +1 @@
(no final assistant message — see the .err log)
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
error: no final assistant text
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 862 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6870 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a2.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a2.md
+29
View File
@@ -0,0 +1,29 @@
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.
**Verified / implemented (phase 184 final pass)**
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.
**Tests / lint / coverage (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.
**Completion criteria**
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
6. Phase E2E green in isolation → ✅
7. README + AGENTS accurate/consistent → ✅
8. Atomic commit — left to the harness (I ran no git write commands).
**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).
**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.
**Next pending phase:** `185_sync_defer_content_fast_path`.
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a2.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__00_phase.a2.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 862 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6870 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__01_identity_map_probe.a1.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__01_identity_map_probe.a1.md
+19
View File
@@ -0,0 +1,19 @@
All gates green. Report:
**Implemented**
- `app/config.py`: `sync_session_reset_files: int = 200` + `sync_prune_batch: int = 500` (Import & sources section, `BOR_`-prefixed, each with a non-negative `field_validator` following the `agent_max_rounds`/`compress_min_bytes` precedent; `0` = exact pre-phase-184 behaviour, documented).
- `app/core/settings_catalog.py`: `CATALOG` rows `"Sync session reset files"` / `"Sync prune batch"` in `"Import & sources"` — plain, live, overridable.
- `.env.example`: new `--- Sync session bounds (phase 184) ---` block (defaults, `0` semantics, the batched-prune atomicity note, "inert until tasks 02/03").
- NEW `tests/integration/test_import_session_retention.py` — the retention probe with the `_map_size()` door: added-run release, the unchanged-run staircase pin (`0,1,…,11` → exactly 12 `Document`s resident, content included, **not** collectable, `expunge_all()` releases all), the `0`-means-pre-phase rollback pin, and the weak-clean/dirty-strong mechanism test.
- Unit pins: 4 in `tests/unit/test_config.py` (defaults, env override, `0` accepted, negative refuses startup), 2 in `tests/unit/test_settings_catalog.py` (group/label/flags + field types/defaults).
**Gates** — `uv run pytest --cov=app --cov-report=term-missing` → **4498 passed**, `TOTAL 99%` (`app/config.py` 100%, `settings_catalog.py` 100%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors; `tests/integration/test_import_session_retention.py tests/unit/test_settings_catalog.py -v` → 19 passed (run 4× for flake-stability); sanity E2E in isolation: `test_admin_settings_panel.py` 12 passed, `test_sync_changed_only.py` 3 passed.
**Docs** — `README.md`: two rows in the Configuration reference "Import & sources" table (flagged *inert until task 02/03 wires it*, since nothing reads them yet). No `AGENTS.md` change (it lists no env knobs).
**Notable findings / deviations (important for task 02)**
- D1's premise is **CONFIRMED in effect but wrong in mechanism**: the identity map is *weak*, so the ADD run does **not** accumulate (84/84 present only until one `gc.collect()`, which drops it to 0) — the task's `len(identity_map) >= 12*2` pin on a fresh add run is **not true**. The real, monotone, KB-proportional retention is the **unchanged** path: `_unchanged_doc`'s `doc.file_hash = file_digest` stamp marks every row `modified` while `session.is_modified()` is false (no UPDATE, commit skipped), so the row lands in `Session.dirty` — a *strong* session-side set — and survives every collection with its full `content` (extrapolated ≈265 MB for the 17,407-doc / 15.2 KB KB). `expire_on_commit=False` is not the retainer; the phase still must not change it.
- Hence `expunge_all()` **is** the right lever, but it discards unflushed changes: safe only at D3's point (per-file commit inside `_index_file`, `importer.py:1575`) — written into the module docstring as a hazard note for task 02. Task 02's post-fix bound against this fixture is "≤ reset + slack" starting from `MEASURED_UNCHANGED_MAP_SIZE = 12`.
- RSS is recorded, never asserted (inherited A4 rule): ≈35 KB/file, unreturned after `gc.collect()` while live-object count stays flat — allocator, not session; `expunge_all` cannot fix that part.
**Next pending task:** `.agents/phases/todo/184_sync_bounded_session/02_session_reset_cadence.md`.
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__01_identity_map_probe.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__01_identity_map_probe.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6863 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__02_session_reset_cadence.a1.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__02_session_reset_cadence.a1.md
+17
View File
@@ -0,0 +1,17 @@
**Implemented (phase 184 task 02)**
- `app/rag/importer.py` `import_sources`: `reset_every = llm.settings.sync_session_reset_files` read ONCE before the source loop; `session.expunge_all()` at exactly two guarded points — top of the per-file loop (`summary.files % reset_every == 0`) and once per source before the two post-passes — with the full safety argument (commits already done, `expunge_all` discards only the fast path's no-op stamp, hooks/`dates_map`/`seen`/`total` plain, rollback path touches nothing) plus a docstring paragraph. No other call sites.
- `tests/integration/test_import_session_retention.py`: added `test_reset_cadence_bounds_the_map` (cadence 3 vs the cadence-0 reference: identical `ImportSummary` via `asdict`, identical DB per path/content/summary/`file_hash`/chunk-count; end map ≤ `MAP_BOUND`=17, mid-run ≤ 2×7 objects), `test_reset_cadence_bounds_the_unchanged_walk` (sawtooth `[0,1,2]×4` vs the pre-fix ramp, end map `0` → pins the source boundary), `test_reset_cadence_survives_an_embedding_error` (stub refuses file #4 with cadence 2: `errors=1`, 11 survivors × 6 chunks, `changed_paths` correct); kept both cadence-`0` guarantee tests; docstring flipped to `CONFIRMED → FIXED` with both measured map sizes.
- Fixed one stale existing test: `tests/integration/test_importer.py::test_backfill_never_touches_standalone_and_copyless_rows` called `db.expire()` on objects a run-supplied session may now detach → re-reads by key. Counts/log expectations unchanged (only affected test in 4501).
**Gates (all green)**
- `uv run pytest tests/integration/test_import_session_retention.py --no-cov` → 7 passed
- importer/importer_e2e/importer_dates/sync_api/unit-importer + retention → 129 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **4501 passed**, `app/rag/importer.py` 99%, TOTAL 99% (>90%)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
- E2E spot-checks in isolation: `test_sync_changed_only.py` 3 passed; `test_prune_cascade_cleanup.py` + `test_sync_summary_visibility.py` 6 passed
**Docs**: `README.md` (reset knob no longer "inert", notes the per-source release), `.env.example` (status line: walk reads `BOR_SYNC_SESSION_RESET_FILES`; prune knob still inert until task 03), `AGENTS.md` (new rule 13: release only at the proven-safe points).
**Decisions/deviations**: added the unchanged-walk test beyond the task's list because task 01 proved the retention lives on the unchanged path (the ADD run's rows are GC-collectable, so its bound alone pins nothing); `MAP_BOUND = RESET_FILES + (RESET_FILES-1)×7 = 17` documented as a ceiling, measured end-of-run 0. No live-infra changes.
**Next pending task**: `.agents/phases/todo/184_sync_bounded_session/03_batched_prune.md`.
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__02_session_reset_cadence.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__02_session_reset_cadence.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
..................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 860 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6868 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__03_batched_prune.a1.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__03_batched_prune.a1.md
+20
View File
@@ -0,0 +1,20 @@
All green. Final report:
**Implemented (task 03 — `184_sync_bounded_session/03_batched_prune.md`)**
- `app/rag/importer.py` `_prune`: the candidate walk is now `select(Document).options(load_only(id, source, path, is_image, image_path, page_count)).where(source.in_(…)).order_by(Document.id))` — `content` (avg 15.2 KB) never loaded; `from sqlalchemy.orm import load_only` added.
- `_prune(..., prune_batch: int = 500)` keyword; commits every Nth delete, and the final commit is skipped only when the last batch boundary already committed (so `10` → exactly 3 commits for 30 docs; `0` → exactly the pre-phase 1 commit). Caller (`import_sources`, L1034) passes `prune_batch=llm.settings.sync_prune_batch` (read once per run).
- Docstring: the light-row contract (per-column rationale for the six, why `content` is out), the batch-commit contract, and the LOCKED A2 atomicity note (crash leaves a prefix pruned; `seen` recomputed from disk → next sync converges).
- Tests (`tests/integration/test_import_session_retention.py`, +5 prune tests + a `before_cursor_execute` capture): no `documents.content` in the prune SELECT and projection == the six columns; **exactly 1** `FROM documents` SELECT per prune (no per-row lazy load); page-copy cleanup (`id`+`page_count`) proven off light rows; 0 orphan chunks; commit counts 3 vs 1; the `Settings → _prune` wiring via a forwarding spy; both prune guards (phase-122 images-off, phase-126 parent-PDF) on light rows. Mutation-checked: dropping `page_count`/`image_path`, loading full rows, the unconditional trailing commit and dropping the caller kwarg each FAIL a test.
**Gates (all 100% green)**
- `uv run pytest tests/integration/test_import_session_retention.py -q --no-cov` → 12 passed
- `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/unit/test_importer.py tests/unit/test_importer_ignore.py` → 120 passed
- `uv run pytest` → **4506 passed**; `--cov=app` TOTAL **99%** (`app/rag/importer.py` 99%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- E2E spot-check (unchanged phases): `test_prune_cascade_cleanup.py` 3 passed, `test_sync_changed_only.py`+`test_sync_summary_visibility.py` 6 passed
**Docs**: `README.md` — new "A sync's memory is bounded per RUN, not per knowledge base" operations bullet + `BOR_SYNC_PRUNE_BATCH` row no longer "inert" + CLI `--prune` knob note; `AGENTS.md` — new rule 14 (six-column prune, batch commits, no bulk `delete()`/`raiseload`); `.env.example` — both knobs now live.
**Decisions**: default `prune_batch=500` (task-specified); `scripts/import_docs.py` has no direct `_prune` call (it goes through `import_sources`, so its counts/log lines are unchanged and it follows the setting). Deviation from the task's literal snippet: the end-of-loop commit is guarded so a batch that divides the count emits no redundant empty transaction — this is what makes the pinned count exactly 3.
**Next pending task**: `184_sync_bounded_session/04_e2e_bounded_sync.md`.
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__03_batched_prune.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__03_batched_prune.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 862 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6870 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__04_e2e_bounded_sync.a1.err
View File
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__04_e2e_bounded_sync.a1.md
+10
View File
@@ -0,0 +1,10 @@
All gates are green and the working tree is final. Report:
**Task 04 complete — phase 184's dedicated Playwright suite + docs pass.**
- **New** `tests/e2e/test_sync_session_reset.py` (919 lines, mock-only, `E2E_REAL_LLM=1` skipped): two module apps on their own ports (`E2E_APP_PORT_SYNCRESET` 8200 = reset 3 / batch 2, `E2E_APP_PORT_SYNCRESET_ZERO` 8201 = both `0`), one 10-file/two-folder fixture source with a sentinel, shared E2E Postgres truncated before **and** after every test, admin login via `auth_helpers`, `/api/` 4xx/5xx + request-failure watcher.
- Tests → observables: ① cadence-ON walk lands all 10 docs **exactly once** (`10 added`, `errors=0`, stat cards == `select(count())`, both folders' ls/tree rows, no wire errors); ② unchanged re-sync `0 added · 10 unchanged`, per-path chunk counts byte-still (no silent re-embed); ③ five-file prune at `BOR_SYNC_PRUNE_BATCH=2` (3 commits inside the UI run) → `0 added · 5 unchanged · 5 pruned`, rows gone from both folders, survivors' chunks intact, **0 orphan chunk rows**; ④ the byte-identical cross-check — cadence-`0` leg vs the recorded cadence-`3` leg: same paths/counts, `content`, `summary`, `content_hash`, `file_hash`, chunk contents in position order, same counts line (non-vacuous: it pins its own baseline shape first).
- Gates: `pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**; `test_sync_changed_only.py` → 3 passed; `test_sync_git_freeze.py` → 2 passed; `pytest --cov=app` → **4506 passed, app/ TOTAL 99%**; `ruff check .` → All checks passed; `pyright` → **0 errors, 0 warnings**.
- Docs: `README.md` (phase-184 operations paragraph, both knobs in the env table with defaults/`0` semantics, CLI-import note, E2E pointer), `AGENTS.md` rules 13/14 (now cite the E2E suite; corrected the prune pin to "exactly one **document-wide** SELECT"), `.env.example` (dropped the task-progress wording; added the two fields it omitted project-wide — `BOR_APP_VERSION`, `BOR_SESSION_COOKIE` — so every `Settings` field is exemplified). Audited programmatically: every `Settings` field documented in README and `.env.example`.
- No defects found in app code; no locked decision touched. RSS is asserted nowhere (phase 183 A4 inherited) — the memory teeth stay in `tests/integration/test_import_session_retention.py`.
- **Left to the harness:** the atomic `--no-gpg-sign` commit and the `todo/ → complete/` move (I ran no git write commands, per the pipeline rules).
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__04_e2e_bounded_sync.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/184_sync_bounded_session/184_sync_bounded_session__04_e2e_bounded_sync.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 862 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6870 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.env.example
+25
View File
@@ -21,6 +21,7 @@
# --- App ---
BOR_ENVIRONMENT=development
# BOR_APP_NAME=Brain of Reese # display name on all pages — titles, header brand, status labels, aria text (phase 39)
# BOR_APP_VERSION=0.1.0 # version the app reports (GET /api/health, the OpenAPI doc)
# BOR_INPUT_PLACEHOLDER=Ask me anything… # composer placeholder, chat page (phase 62)
# BOR_FOOTER_TEXT=Powered by self-hosted models # footer line on every page (phase 62)
# (Phase 91: the retired CSS-file theme env var is gone — the colors
@@ -157,6 +158,28 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
# BOR_UPLOAD_MAX_MB=512 # caps BOTH the compressed upload and the total
# extracted bytes (zip-bomb guard); must be > 0
# --- Sync session bounds (phase 184) ---
# The two knobs that bound ONE sync run, so its cost stops scaling with
# the size of the whole knowledge base. Both read per run (the importer
# reads its settings per sync), so a value saved in the admin Settings
# panel applies to the NEXT sync - no restart. 0 = the exact
# pre-phase-184 behaviour on both (the no-redeploy rollback); a negative
# value is a typo and refuses startup, naming the field.
# The walk reads BOR_SYNC_SESSION_RESET_FILES (released every Nth file plus
# once per source, before the two post-passes) and the prune reads
# BOR_SYNC_PRUNE_BATCH (one COMMIT every Nth delete, over a row that loads
# six columns and never the document's content).
# BOR_SYNC_SESSION_RESET_FILES=200 # files between session releases
# (expunge_all) inside the import walk;
# 0 = never release anything (pre-phase)
# BOR_SYNC_PRUNE_BATCH=500 # pruned documents per COMMIT inside the
# prune instead of one commit at the end
# of the whole import; 0 = one commit at
# the end (pre-phase). A crash mid-prune
# then leaves a PREFIX pruned - the next
# sync converges (seen is recomputed
# from disk every run)
# --- Image documents (phase 122) ---
# Master switch for indexing standalone image files (direct uploads,
# uploaded archives, git/local source walks) as first-class documents.
@@ -257,6 +280,8 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
BOR_ADMIN_PASSWORD=
BOR_SESSION_SECRET=
# BOR_SESSION_MAX_AGE=43200 # signed-cookie lifetime, seconds (default 12 h, sliding)
# BOR_SESSION_COOKIE=bor_session # cookie NAME; boot-wired, so a change here
# needs a restart (the panel marks it the same)
# --- Debugging (0/1 — 1 enables attach-on-demand debugpy on port 5678) ---
DEBUGPY=0
AGENTS.md
+39
View File
@@ -133,6 +133,45 @@
`doc.chunks = [...]` **re-embed** path is the deliberate, documented
exception (collection replacement legitimately needs the loaded
collection; changed files only).
13. **Release the sync session at the proven-safe points only:** an import
run holds one session and `expire_on_commit=False` (`app/db.py`) means a
commit releases nothing, so `_unchanged_doc`'s `file_hash` stamp keeps
every walked `Document` — content included — resident in the session's
own strong `dirty` set (not collectable): memory grew with the KB. The
cadence is therefore `session.expunge_all()` every
`BOR_SYNC_SESSION_RESET_FILES` files at the TOP of the per-file loop and
once per source before the two post-passes (phase 184; `0` = never, the
pre-phase rollback). `expunge_all` DISCARDS unflushed changes, so never
move it into `_index_file`, `_unchanged_doc`, `_store_summary` or a
post-pass — their locals are live across statements — and never add a
reset before a file's per-file commit. Pinned by
`tests/integration/test_import_session_retention.py` (bounded with the
cadence on, unbounded with it `0`, byte-identical summaries and DB) and
through the UI by `tests/e2e/test_sync_session_reset.py` (two module apps,
one per cadence: every document exactly once, the all-`unchanged` re-sync,
and the `0`-cadence run byte-identical to the `3`-cadence one).
14. **The prune reads six columns and commits in batches:** `_prune`
selects its candidates through
`load_only(id, source, path, is_image, image_path, page_count)`
`order_by(Document.id)` — the exact set the phase-122 images-off guard,
the phase-126 parent-PDF rule and the image/page copy cleanup read, and
NOT `content` (252 MB of markdown per full KB walk) — and commits every
`BOR_SYNC_PRUNE_BATCH` deletes (`0` = the pre-phase single commit;
phase 184). Never read another `Document` attribute inside that loop (a
deferred column costs one lazy SELECT per document and unbounds the
walk — no `raiseload` paper-over, add the column instead), and never
swap the per-document `session.delete(doc)` for a bulk `delete()` (the
guards, `changed_paths` and the log line depend on the per-document
shape). A non-zero batch deliberately trades the all-or-nothing prune
for a prefix (LOCKED A2): `seen` is recomputed from disk, so the next
sync converges. Pinned by
`tests/integration/test_import_session_retention.py` (the captured
`SELECT`'s column list, exactly one document-wide `SELECT` per prune
(a second one is a per-row lazy load), the commit counts at `10` and
`0`, both guards on light rows) and
`tests/e2e/test_sync_session_reset.py` (the five-document prune at batch
`2` through the Sources page: both folders cleared, the survivors' chunks
intact, zero orphan chunk rows).
## Quick reference
```bash
README.md
+35 -1
View File
@@ -457,7 +457,11 @@ reflect the new files.
its `chunks` rows (the embeddings) go together, the latter through the
database's own `ON DELETE CASCADE`, so the cost of a mass prune is
bounded by documents, not chunks (see *A mass prune costs documents,
not chunks* under [Sync from the UI](#sync-from-the-ui)).
not chunks* under [Sync from the UI](#sync-from-the-ui)). The CLI run
follows the same two phase-184 knobs as the UI sync
(`BOR_SYNC_SESSION_RESET_FILES`, `BOR_SYNC_PRUNE_BATCH`): it loads only
the prune's six columns and commits every Nth delete (`0` on either knob
restores the pre-phase-184 run).
- The import prints one line per file and ends with a greppable summary,
so it is safe to run from cron or after every commit.
- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman
@@ -651,6 +655,34 @@ only) runs the whole source refresh in one click:
that fails the suite if the cascade clause ever disappears; the owner
flow itself (delete upstream → re-sync → exactly those documents and
chunks gone) is pinned by `tests/e2e/test_prune_cascade_cleanup.py`.
- **A sync's memory is bounded per RUN, not per knowledge base** (phase
184). One import run holds one session, and `expire_on_commit=False`
(`app/db.py`) means a commit releases nothing — so before this phase the
run kept every row it touched (each unchanged document with its full
~15 KB `content`) until it finished, and the prune read every document of
every pruned source whole, deletes held in ONE transaction. Now the walk
releases its session every `BOR_SYNC_SESSION_RESET_FILES` files (plus once
per source), and the prune (`_prune`) reads only the SIX columns its
guards and its copy cleanup need (`id`, `source`, `path`, `is_image`,
`image_path`, `page_count` — never `content`) ordered by id, committing
every `BOR_SYNC_PRUNE_BATCH` deletes. **Where the operator sets the
ceiling:** those two `BOR_`-prefixed settings (defaults `200` /
`500`, live on the next sync without a restart, `0` on either one restores
the exact pre-phase-184 run as the no-redeploy rollback). Counts, log
lines, `changed_paths`, the phase-122/126 prune guards and the database
are byte-identical either way — pinned by
`tests/integration/test_import_session_retention.py` (the identity-map
bound, the no-`content` prune `SELECT` with no per-row lazy load, the
3-commits-vs-1 cadence, both prune guards on light rows) and end to end
through the Sources page by `tests/e2e/test_sync_session_reset.py` — two
module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
`E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
every document exactly once with the totals the stat cards show, the
re-sync is all-`unchanged` with an unchanged chunk total, a five-document
prune at batch `2` clears both folders with zero orphan chunk rows, and a
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
line).
- **Cancel**: while a run is in flight, a **Cancel** button appears
next to the sync button; clicking it posts `POST /api/sync/cancel`
(admin only) and registers a cooperative cancel (`202`) — the run
@@ -1548,6 +1580,8 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SYNC_SESSION_RESET_FILES` | `200` | Files between releases of the import run's SQLAlchemy session (phase 184): every Nth file the walk hands back everything the session still holds (`expunge_all`), so an unchanged-file sync stops keeping one full document row per file walked. `0` = never release — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. The release also runs once per source, before the two post-passes |
| `BOR_SYNC_PRUNE_BATCH` | `500` | Pruned documents per COMMIT inside the prune (phase 184) instead of one commit at the end of the whole import, so a mass prune releases its deletes as it goes instead of holding one enormous transaction. `0` = one commit at the end — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. With a non-zero batch a crash mid-prune leaves a **prefix** pruned, and the next sync converges (`seen` is recomputed from disk every run) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
### Image support (phases 122/123/124/181)
app/config.py
+51
View File
@@ -446,6 +446,29 @@ class Settings(BaseSettings):
#: the validator fails loudly at startup (the ``agent_max_rounds``
#: pattern).
upload_max_mb: int = 512
#: Files between identity-map releases inside
#: ``app.rag.importer.import_sources`` (phase 184,
#: ``BOR_SYNC_SESSION_RESET_FILES``): every Nth file of the walk — at
#: the one point where no object from a finished file is reachable —
#: the run's session calls ``expunge_all()``, so whatever the walk
#: still holds goes before the next file is read. ``0`` = NEVER reset
#: — the exact pre-phase-184 behaviour (the operator's no-redeploy
#: rollback, the ``BOR_LOG_LARGE_RESPONSE_BYTES`` precedent). A
#: negative value is a typo — the validator fails startup loudly (the
#: ``agent_max_rounds`` pattern). Read per run (the importer reads
#: ``llm.settings``, so a saved override applies to the NEXT sync).
sync_session_reset_files: int = 200
#: Pruned documents per commit inside ``app.rag.importer._prune``
#: (phase 184, ``BOR_SYNC_PRUNE_BATCH``): the delete loop commits
#: every Nth document instead of once at the end of the whole import,
#: so a mass prune releases its rows as it goes. ``0`` = ONE commit at
#: the end — the exact pre-phase-184 behaviour (the same rollback
#: knob). A negative value is a typo — the validator fails startup
#: loudly (the ``agent_max_rounds`` pattern). Note the deliberate
#: atomicity change of a non-zero batch (phase 184 A2): a crash
#: mid-prune leaves a PREFIX pruned, and the next sync converges
#: because ``seen`` is recomputed from the filesystem every run.
sync_prune_batch: int = 500
# --- Image documents (phase 122: standalone images as documents) ---
#: Master switch for image-document indexing (phase 122,
@@ -809,6 +832,34 @@ class Settings(BaseSettings):
raise ValueError("upload_max_mb must be > 0 (MiB)")
return v
@field_validator("sync_session_reset_files")
@classmethod
def _sync_session_reset_files_non_negative(cls, v: int) -> int:
"""``0`` is the pre-phase-184 rollback (never reset) — a negative
value is a typo that would make the cadence comparison constant,
so it fails startup naming the field (the
``agent_max_rounds`` pattern, phase 184)."""
if v < 0:
raise ValueError(
"sync_session_reset_files must be >= 0 (0 = never reset the"
" import session)"
)
return v
@field_validator("sync_prune_batch")
@classmethod
def _sync_prune_batch_non_negative(cls, v: int) -> int:
"""``0`` is the pre-phase-184 rollback (one commit at the end of
the prune) — a negative value is a typo that would make the batch
comparison constant, so it fails startup naming the field (the
``agent_max_rounds`` pattern, phase 184)."""
if v < 0:
raise ValueError(
"sync_prune_batch must be >= 0 (0 = one commit at the end of"
" the prune)"
)
return v
@field_validator("chat_image_max_mb")
@classmethod
def _chat_image_max_mb_positive(cls, v: int) -> int:
app/core/settings_catalog.py
+6
View File
@@ -288,6 +288,12 @@ CATALOG: dict[str, SettingMeta] = {
"upload_max_mb": SettingMeta(
"upload_max_mb", "Import & sources", "Upload max MB"
),
"sync_session_reset_files": SettingMeta(
"sync_session_reset_files", "Import & sources", "Sync session reset files"
),
"sync_prune_batch": SettingMeta(
"sync_prune_batch", "Import & sources", "Sync prune batch"
),
# --- Images ---
"images": SettingMeta("images", "Images", "Images"),
"image_extensions": SettingMeta(
app/rag/importer.py
+130 -6
View File
@@ -346,7 +346,7 @@ from typing import Any, Protocol
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
from sqlalchemy import select
from sqlalchemy.orm import Session
from sqlalchemy.orm import Session, load_only
from app.config import Settings
from app.db import SessionLocal
@@ -867,6 +867,14 @@ async def import_sources(
failure leaves the row untouched — the next sync retries it).
Driven by committed doc rows, so it is safe under ``limit`` too
(it deletes nothing).
Bounded session (phase 184, task 02): the walk releases its SQLAlchemy
identity map every ``llm.settings.sync_session_reset_files`` files and
once per source before the two post-passes, so the run's resident set
scales with the CADENCE rather than with the size of the knowledge base
(``0`` = never — the exact pre-phase behaviour, the operator's
no-redeploy rollback). The cadence changes no count, no log line and no
committed byte: see the safety argument at the reset site.
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -875,6 +883,12 @@ async def import_sources(
# toggle is on — the empty set admits nothing, so the toggle-off run
# (walk, counts, prune) stays byte-identical to pre-phase.
image_exts = llm.settings.image_extension_set if llm.settings.images else frozenset()
# Phase 184 (task 02): the identity-map release cadence, read ONCE here —
# before the source loop — so a settings overlay saved mid-run cannot
# change the cadence between two files of the same walk (the importer
# reads ``llm.settings`` per RUN, never per file). ``0`` disables the
# release entirely.
reset_every = llm.settings.sync_session_reset_files
owns_session = session is None
if session is None:
session = SessionLocal()
@@ -922,6 +936,40 @@ async def import_sources(
):
if limit is not None and summary.files >= limit:
break
# Phase 184 (task 02): the identity-map release. There is NO
# automatic one — ``expire_on_commit=False`` (``app/db.py:30``)
# means a commit drops nothing, and ``_unchanged_doc``'s
# ``doc.file_hash = file_digest`` stamp marks every
# already-current row DIRTY, which puts it in the session's
# own strong ``dirty`` set (cyclic GC cannot reach it, see
# ``tests/integration/test_import_session_retention.py``).
# Without this the run holds every Document — its full
# ``content`` included — it walked until the import ENDED:
# the 2026-10-15 incident's second driver, and the reason
# memory grew with the KB rather than with the work.
#
# This is the ONLY safe point, and WHY it is safe:
# ``expunge_all()`` discards UNFLUSHED changes, so it may run
# only where every real write of the finished file is already
# committed. Here the previous file's ``_index_file`` has
# RETURNED — each path commits per file (the changed path
# commits its chunks, ``_store_summary`` commits, and
# ``_unchanged_doc`` commits its date refresh, its stamp and
# whatever the phase-163 strip re-gate wrote) — and what
# ``expunge_all`` then drops is at most the fast path's stamp
# of a value the database ALREADY holds (no UPDATE was
# pending: ``session.is_modified`` is false for it). Nothing
# downstream reaches back into a finished file: ``_index_file``
# returns ``None``, the progress/chunk/image hooks only assign
# fields on the sync's own status object, ``dates_map``/
# ``ignore``/``seen`` are plain data, and ``total`` was
# computed before the loop. The ``EmbeddingError`` handler
# below calls ``session.rollback()`` and touches no object of
# its own, so detached state is irrelevant to it.
# ``reset_every == 0`` → never (the exact pre-phase
# behaviour, the operator's rollback knob).
if reset_every and summary.files and summary.files % reset_every == 0:
session.expunge_all()
rel = path.relative_to(root).as_posix()
seen.add((source, rel))
summary.files += 1
@@ -955,6 +1003,13 @@ async def import_sources(
session.rollback()
summary.errors += 1
logger.error("import: error source=%s path=%s — %s", source, rel, e)
# Phase 184 (task 02): the source-boundary release — one source's
# walk residue never crosses into the next source. Safe for the
# same reason as the per-file reset and additionally because BOTH
# post-passes below select the rows they need fresh (their own
# ``select(Document)``), so no walk object has to survive it.
if reset_every:
session.expunge_all()
# Phase 125 (task 04): the post-index enrichment pass — the
# markdown-FIRST case (a markdown indexed while its
# referenced image had no document yet recorded the
@@ -980,6 +1035,10 @@ async def import_sources(
session, source_names, seen, images=llm.settings.images,
summary=summary,
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
# Phase 184 (task 03): the commit cadence, read ONCE
# here (the same per-RUN rule as the identity-map
# reset) — ``0`` = the pre-phase single commit.
prune_batch=llm.settings.sync_prune_batch,
)
summary.embed_batches = llm.embed_batches
summary.log()
@@ -3864,6 +3923,7 @@ def _prune(
images: bool = False,
summary: ImportSummary | None = None,
page_dir: Path | None = None,
prune_batch: int = 500,
) -> int:
"""Delete documents of *source_names* whose file is no longer in *seen*.
@@ -3910,10 +3970,51 @@ def _prune(
chunk row, never hydrates a 768-dim embedding, and a mass prune costs
documents instead of chunks × 24.8 KiB (the 2026-10-15 incident: a
3,008-doc prune materialised ~720 MB to delete 72k rows and locked up
a 3,904 MB VM with no swap). The ``changed_paths`` feed and the
image/page copy cleanup above are unchanged, and they only ever read
a 3,904 MB VM with no swap). That fix removed the CHILD rows; a SECOND
driver of the same incident stood on this very walk, and phase 184
(task 03) removed it with the two contracts below.
Phase 184 (task 03) — the light-row contract. The walk below selects
through ``load_only`` and is ordered by ``Document.id``, so its resident
set is independent of the knowledge base: ``content`` averages 15.2 KB
per row (252 MB of markdown across the incident KB), and loading it for
every document of every pruned source — to decide a deletion and then
throw the string away — was that driver. The SIX columns loaded are
exactly what the guards and the cleanup read: ``is_image`` (the
phase-122 images-off toggle guard), ``path`` (that same guard's
``seen`` tuple and the phase-126 ``_EMBEDDED_IMAGE_RE`` parent-PDF
rule), ``source`` (the ``seen`` tuple, ``changed_paths`` and the log
line), ``image_path`` (:func:`_delete_image_copy`), ``id`` +
``page_count`` (:func:`_delete_page_copies`) — and ``content`` is NOT
among them (nor ``summary``, ``full_path``, ``title``,
``image_context``, either hash, or any date). Reading anything else
inside the loop would emit one lazy SELECT per document and undo the
bound; ``tests/integration/test_import_session_retention.py`` pins both
halves (no ``documents.content`` in the SELECT, exactly ONE
``FROM documents`` SELECT for the whole walk). Phase 185 does the same
job for the walk's fast path.
Phase 184 (task 03) — the batch-commit contract. *prune_batch* is
``Settings.sync_prune_batch`` (``BOR_SYNC_PRUNE_BATCH``, read once by
the caller from ``llm.settings``): the loop commits every Nth delete,
so a long prune no longer holds one enormous transaction and its rows
are released as it goes. ``prune_batch=0`` is the exact pre-phase
behaviour — ONE commit after the whole loop — the operator's
no-redeploy rollback (LOCKED A3).
**Atomicity (LOCKED A2, deliberate).** With a non-zero batch a crash
mid-prune leaves a PREFIX pruned instead of rolling the whole prune
back. That is safe HERE, and only here: ``seen`` is recomputed from the
filesystem on every run, so the next sync converges on whatever this
one left, and the import already committed per file throughout the walk
— a run-level transaction never existed to lose. ``summary.pruned``
stays the count of documents actually deleted, unchanged in meaning.
What phase 183 left in place is unchanged: the ``changed_paths`` feed
and the image/page copy cleanup above, which only ever read
``doc.image_path`` / ``doc.page_count`` / ``doc.source`` / ``doc.path``
— never ``doc.chunks``. The **re-embed** path is the deliberate
— never ``doc.chunks`` — and they are exactly the reads the six light
columns cover. The **re-embed** path is the deliberate
exception (LOCKED A2): :func:`_replace_content_chunks` reassigns
``doc.chunks = [...]``, which is collection REPLACEMENT (the ORM must
know which old rows to orphan) and legitimately needs the loaded
@@ -3923,7 +4024,25 @@ def _prune(
if not source_names:
return 0
pruned = 0
docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()
# Phase 184 (task 03): the light-row walk — see the docstring's
# light-row contract. SIX columns, ``content`` deliberately absent
# (252 MB of markdown per full KB walk), deterministic order so the
# batch prefix is reproducible.
docs = session.scalars(
select(Document)
.options(
load_only(
Document.id,
Document.source,
Document.path,
Document.is_image,
Document.image_path,
Document.page_count,
)
)
.where(Document.source.in_(source_names))
.order_by(Document.id)
).all()
for doc in docs:
if (doc.source, doc.path) not in seen:
if doc.is_image and not images:
@@ -3948,6 +4067,11 @@ def _prune(
session.delete(doc)
pruned += 1
logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
if pruned:
# Phase 184 (task 03): a long prune no longer holds a huge
# transaction — commit the batch (``prune_batch=0`` → never
# here, the single commit below, i.e. the pre-phase run).
if prune_batch and pruned and pruned % prune_batch == 0:
session.commit()
if pruned and not (prune_batch and pruned % prune_batch == 0):
session.commit()
return pruned
tests/e2e/test_sync_session_reset.py
+919
View File
@@ -0,0 +1,919 @@
"""Phase 184 E2E (Playwright, mock-only): the BOUNDED sync still syncs all.
Source: owner incident 2026-10-15 — the same run whose prune phase 183 fixed.
An import run holds ONE session and ``app/db.py`` builds it with
``expire_on_commit=False``, so a commit released NOTHING: the walk kept every
``Document`` it touched resident (each unchanged row holds its full ~15 KB
``content``, and ``_unchanged_doc``'s ``file_hash`` stamp puts the row in the
session's own strong ``dirty`` set, out of reach of the cycle collector), and
``_prune`` read every document of every pruned source WHOLE — ``content``
included — holding all of its deletes in ONE transaction until the end. The
run's memory therefore grew with the SIZE OF THE KNOWLEDGE BASE instead of
with the work the run did.
The fix is two operator knobs, both ``BOR_``-prefixed settings the importer
reads once per RUN:
* ``BOR_SYNC_SESSION_RESET_FILES`` (default ``200``) — every Nth file of the
walk, at the one point where no object from a finished file is reachable,
the session calls ``expunge_all()``; it also runs once per source, before
the two post-passes. ``0`` = NEVER release = the exact pre-phase-184 run.
* ``BOR_SYNC_PRUNE_BATCH`` (default ``500``) — the prune commits every Nth
delete instead of once at the end of the whole import. ``0`` = the exact
pre-phase-184 single commit. A non-zero batch deliberately trades the
all-or-nothing prune for a PREFIX (LOCKED A2: ``seen`` is recomputed from
disk, so the next sync converges).
Both knobs change the run's MEMORY PROFILE and nothing else — and that
"nothing else" is precisely what a browser can see. This module pins it: the
same counts line, every document exactly once, an all-``unchanged`` re-sync,
a multi-batch prune that behaves, and a Sources page that never stalls, never
errors and reports the same totals as the database. The retention itself (the
identity-map bound, the prune's six-column ``SELECT``, the commit counts) is
deterministic and pinned in
``tests/integration/test_import_session_retention.py``; RSS is asserted
nowhere in this phase (phase 183 LOCKED A4, inherited).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is skipped — every number here (the
per-path chunk counts, the byte-identical cross-check) is a property of the
deterministic mock embedder/summarizer; against the real endpoint the
comparison would be a measurement, not a pin.
TWO module apps on dedicated ports (the ``test_sync_summary_visibility.py``
pattern — the conftest session app owns 8123 in a combined run, and a
same-port second uvicorn would drive the wrong server):
* the CADENCE app — ``BOR_SYNC_SESSION_RESET_FILES=3`` and
``BOR_SYNC_PRUNE_BATCH=2``, so a 10-file walk releases the session THREE
times mid-run (before files 4, 7 and 10) plus once at the source boundary,
and a 5-document prune commits THREE times (at 2, at 4, and the final one).
Tests 1–3 drive it;
* the ZERO app — both knobs ``0``, i.e. the exact pre-phase-184 run. Test 4
drives it over the SAME fixture directory, which is what makes the two runs
comparable at all.
Both boot against the mock LLM with ``BOR_GIT_SOURCES`` forced empty (the
owner's local gitignored ``.env`` must never leak a real source into a run),
their own ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` under
``tmp_path_factory`` (nothing here clones or renders a PDF — the dirs exist so
no run writes into the operator's), and the mock-calibrated gate (the conftest
convention). The fixture KB is ONE plain (non-git) directory named
:data:`SOURCE_NAME` (``kind='local'`` rows index under ``Path(path).name``, so
the source label is stable whatever temp root pytest picked) holding TEN
``.md`` files across TWO folders — ``a/1..5.md`` and ``b/1..5.md`` — with
``a/1.md`` carrying the sentinel phrase.
Isolation: the shared E2E Postgres is truncated BEFORE **and** AFTER every
test (``chunks, documents, query_log, kb_overview, git_sources, sources_meta,
folder_summaries``), after waiting for the app to hold no running sync (the
run state lives in the app's memory, and a leaked run would keep importing
into the next test's truncated KB). Each test registers its own source row.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_reset_cadence_produces_the_identical_run`` — the cadence ON is a
complete, healthy sync: the counts line reads ``10 added``, the terminal
reports ``added=10`` with ``errors=0``, every one of the ten paths exists
EXACTLY ONCE in the KB (the test's own SQL), the ls/tree rows of both
folders render all ten files, the stat cards equal ``select(count())`` over
``documents`` (10) and ``chunks`` (> 10), the sentinel's phrase is in the
stored content, and THROUGH THE RUN no ``/api/`` request answered 4xx/5xx
and none failed — no error banner, no modal.
2. ``test_resync_is_unchanged_with_the_cadence_on`` — the second sync of the
same bytes settles ``0 added · 10 unchanged`` with ``pruned=0`` and
``success``, and the chunk total and per-path chunk counts are byte-still:
the cadence loses no rows (a lost row would come back as ``added``) and
re-embeds nothing.
3. ``test_multi_batch_prune_from_the_ui`` — five of the ten files go and the
re-sync settles ``0 added · 5 unchanged · 5 pruned`` (5 deletes at batch
``2`` = THREE commits inside the UI's own run); the five documents are gone
from the KB AND from the ls/tree rows of BOTH folders, the survivors' chunk
total is unchanged, and ZERO orphan chunk rows exist (the test's own SQL).
4. ``test_cadence_zero_matches_cadence_three`` — the byte-identical
cross-check the phase lives on: the SAME fixture walked with
``BOR_SYNC_SESSION_RESET_FILES=0`` lands the same paths (each exactly
once), the same per-path chunk counts, the same ``content`` / ``summary`` /
``content_hash`` / ``file_hash`` on every document, the same chunk contents
in position order and the same counts line as the cadence-``3`` run. The
reference is test 1's recorded run when the module runs in order; alone,
the test walks the cadence-``3`` app itself first — and either way it
asserts its own 10-documents / >10-chunks shape BEFORE comparing, so an
empty baseline can never make it pass vacuously.
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import func, select, text
from app.db import SessionLocal
from app.models import Chunk, Document
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
#: The CADENCE app (tests 1–3). Env-overridable, the house port pattern: the
#: conftest session app owns 8123 in a combined run.
CADENCE_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET", "8200"))
#: The ZERO app (test 4's reference leg): both knobs ``0`` — the exact
#: pre-phase-184 run (LOCKED A3's no-redeploy rollback).
ZERO_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET_ZERO", "8201"))
SOURCES_URL = "/sources.html"
#: The fixture directory's basename — the ``source`` the importer records for
#: a ``kind='local'`` row (``Path(path).name``), so it is the same string in
#: the DB whatever temp root this run got.
SOURCE_NAME = "sync-session-reset"
#: The fixture layout: two folders × five importable files. After the prune,
#: ``a`` keeps three survivors and ``b`` two — so BOTH folders still render
#: rows, and a prune that stopped after its first batch would show up.
ALL_PATHS = [*[f"a/{n}.md" for n in range(1, 6)], *[f"b/{n}.md" for n in range(1, 6)]]
KEEP_PATHS = ["a/1.md", "a/2.md", "a/3.md", "b/1.md", "b/2.md"]
GONE_PATHS = ["a/4.md", "a/5.md", "b/3.md", "b/4.md", "b/5.md"]
N_FILES = len(ALL_PATHS)
#: The sentinel phrase, carried by :data:`SENTINEL_PATH` ONLY. Finding it in
#: the stored ``content`` proves the walk really indexed the fixture rather
#: than leaving a KB of merely the right SIZE.
SENTINEL = "RESE-SYNC-RESET-SENTINEL-4f27"
SENTINEL_PATH = "a/1.md"
#: The cadences the two apps boot with — module constants so the narrative and
#: the app env cannot drift apart. 10 files / every 3rd = releases before
#: files 4, 7 and 10 (THREE mid-run resets, then the source boundary); a
#: 5-document prune at batch 2 = commits at 2, at 4 and the final one.
RESET_FILES = 3
PRUNE_BATCH = 2
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: Settle budgets: a ten-file mock sync (walk + embeds + overview + three
#: folder summaries) lands in seconds and the UI settles a terminal within one
#: 2 s poll tick, so this is an order of magnitude of head-room with no
#: client-side hard timeout (the test_prune_cascade_cleanup budget).
SETTLE_TIMEOUT_MS = 90_000
#: Test 1 records its numbers here for test 4's cross-check (the
#: ``test_sync_changed_only.py`` "state from the earlier test" idiom). Test 4
#: runs its own cadence leg when this is empty, so it stays honest alone.
_CADENCE_RUN: dict[str, Any] = {}
# ---------------------------------------------------------------------------
# The fixture source (ONE directory BOTH apps walk)
# ---------------------------------------------------------------------------
def _doc_body(path: str) -> str:
"""One small markdown file per fixture path, byte-stable across runs.
The bodies differ from each other (so a per-path chunk count means
something) but never carry a timestamp or the absolute path, and only
:data:`SENTINEL_PATH` carries the sentinel — that is what lets test 4
compare two runs of two different processes line for line.
"""
if path == SENTINEL_PATH:
return (
"# Session reset sentinel\n"
"\n"
"The bounded sync keeps the sentinel phrase in the knowledge base.\n"
f"Marker: {SENTINEL}\n"
)
folder, name = path.rsplit("/", 1)
stem = name.removesuffix(".md")
return (
f"# Session reset note {folder} {stem}\n"
"\n"
f"One unrelated note from the {folder} folder of the {SOURCE_NAME} "
"fixture source, indexed so the cadence has something to walk.\n"
)
def _seed_source(root: Path) -> Path:
"""(Re-)write all ten fixture files under ``root / SOURCE_NAME``.
Idempotent on purpose: test 3 deletes five files from the tree and test 4
re-seeds the SAME directory, so the reference leg walks byte-identical
content through the other app.
"""
base = root / SOURCE_NAME
for rel in ALL_PATHS:
target = base / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(_doc_body(rel), encoding="utf-8")
assert not (base / ".git").exists() # a plain directory — no git anywhere
return base
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one fixture tree BOTH module apps walk (the app servers run on the
same host, so the path is visible to them)."""
return _seed_source(tmp_path_factory.mktemp("bor_sync_session_reset"))
# ---------------------------------------------------------------------------
# The two module apps
# ---------------------------------------------------------------------------
def _app_env(scratch: Path, reset_files: int, prune_batch: int) -> dict[str, str]:
"""The per-module app env (the conftest pattern): the mock LLM, the
mock-calibrated gate, no git fallback, dedicated scratch dirs — and THIS
app's two phase-184 cadences, pinned explicitly so an operator's local
(gitignored) ``.env`` cannot retune them."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{MOCK_PORT}/v1"
)
# Mock-calibrated gate (the conftest pattern): the token-overlap mock
# embeddings score far below the production 0.62, and BOTH floors must
# stay <= the threshold or the app refuses to boot (app/config.py).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# No env-fallback rows: the registry each test writes through the
# authenticated API is the only source list, so every sync's counts are
# exactly this test's doing.
env["BOR_GIT_SOURCES"] = ""
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
# THE SUBJECT OF THE PHASE.
env["BOR_SYNC_SESSION_RESET_FILES"] = str(reset_files)
env["BOR_SYNC_PRUNE_BATCH"] = str(prune_batch)
return env
def _serve(env: dict[str, str], port: int) -> Iterator[str]:
"""Boot one uvicorn app under test on *port* and yield its URL."""
proc = subprocess.Popen(
[
sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning",
],
cwd=REPO,
env=env,
)
url = f"http://127.0.0.1:{port}"
try:
_wait_http(f"{url}/api/health")
yield url
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The CADENCE app (tests 1–3): ``reset=3``, ``prune_batch=2``.
``mock_llm`` and ``source_dir`` are dependencies for the ORDERING only —
the mock answers before the app boots, and the fixture tree exists before
the first test registers it.
"""
if USE_REAL_LLM:
pytest.skip(
"mock-only suite — the byte-identical cross-check is a property "
"of the deterministic mock (E2E_REAL_LLM unsupported)"
)
assert mock_llm and source_dir
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_cadence")
yield from _serve(_app_env(scratch, RESET_FILES, PRUNE_BATCH), CADENCE_PORT)
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
@pytest.fixture(scope="module")
def zero_app_server(
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The ZERO app (test 4's reference leg): both knobs ``0``, the exact
pre-phase-184 run the bounded one must be byte-identical to."""
if USE_REAL_LLM:
pytest.skip(
"mock-only suite — the byte-identical cross-check is a property "
"of the deterministic mock (E2E_REAL_LLM unsupported)"
)
assert mock_llm and source_dir
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_zero")
yield from _serve(_app_env(scratch, 0, 0), ZERO_PORT)
@pytest.fixture(scope="module")
def zero_app_url(zero_app_server: str) -> str:
return zero_app_server
# ---------------------------------------------------------------------------
# Isolation
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern). ``sources_meta`` (the
KB generation counter) and ``folder_summaries`` reset with the KB: the
sync's overview/bump is an upsert, and a leftover folder summary would
ride the very ``changed_paths`` feed the prune writes."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No run may leak across a truncate: wait for a non-running status with
an own admin session (the endpoint is admin-only). Usually a no-op — the
tests settle before they end."""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
with httpx.Client(base_url=base_url, timeout=5.0) as client:
client.post("/api/login", json={"password": ADMIN_PASSWORD})
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation: the shared E2E Postgres is truncated before AND
after every test, the app holding no running sync at both boundaries.
The ZERO app is deliberately NOT waited on: it is booting lazily for test 4
only, and that test's own run is settled inside the test body, so the
post-truncate can never race it."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_wait_sync_settled(app_url)
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers: the UI side
# ---------------------------------------------------------------------------
def _watch_network(page: Page) -> list[str]:
"""Collect every ``/api/`` 4xx/5xx and every failed ``/api/`` request the
page sees — the owner's "the Sources page errored / stalled".
Returns the (growing) list the caller asserts empty at the end of the
run: the page's own 2 s status poll lands in this list, so a run that made
the API answer 500 is caught here rather than at some later assertion.
"""
problems: list[str] = []
def _on_response(response: Any) -> None:
if "/api/" in response.url and response.status >= 400:
problems.append(f"HTTP {response.status} {response.url}")
def _on_failed(request: Any) -> None:
if "/api/" in request.url:
problems.append(f"FAILED {request.url} ({request.failure})")
page.on("response", _on_response)
page.on("requestfailed", _on_failed)
return problems
def _add_local_source(page: Page, base_url: str, path: Path) -> None:
"""Register the fixture directory through the authenticated API (the
phase-49 shape — the page's own form for this is gone; the admin cookie
rides ``page.request``)."""
r = page.request.post(
f"{base_url}/api/git-sources",
data={"kind": "local", "path": str(path)},
)
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
def _started_at(body: dict[str, Any]) -> float:
"""The status body's ``started_at`` as a wall-clock epoch (the server
stamps it in UTC and this process runs on the same host)."""
started = body.get("started_at")
if not started:
return 0.0
try:
return datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return 0.0
def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]:
"""Click the Sources page **Sync sources** button and wait for THIS run's
terminal state (the module app is shared, so a terminal counts only after
a ``running`` tick or when its ``started_at`` post-dates the click).
Settles the UI on success and proves the owner saw no error surface: the
button re-enabled, ``Synced HH:MM``, no error banner, no modal."""
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(btn).to_be_enabled()
clicked_at = time.time()
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
deadline = time.monotonic() + SETTLE_TIMEOUT_MS / 1000
saw_running = False
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = page.request.get(f"{base_url}/api/sync/status")
assert r.status == 200, r.text
body = r.json()
state = body["state"]
if state == "running":
saw_running = True
elif state in ("success", "failed", "cancelled") and (
saw_running or _started_at(body) >= clicked_at - 1.0
):
break
time.sleep(0.2)
else:
raise AssertionError(f"sync never reached a terminal state: {body}")
assert body["state"] == "success", f"sync failed: {body}"
assert body["error"] is None, body
expect(btn).to_be_enabled(timeout=15_000)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
expect(page.locator("#sync-error-banner")).to_be_hidden()
assert page.locator(".sync-modal-backdrop").count() == 0
return body
def _expect_counts_line(page: Page, expected: str) -> None:
"""The counts line — the owner-visible sync report on the single sync text
surface (phase 150; sources.js fmtSyncResult)."""
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
def _counts_line(page: Page) -> str:
"""The counts line as text — the string test 4 compares across the apps."""
return str(page.locator("#sync-progress-text").inner_text())
def _open_level(page: Page, *names: str) -> None:
"""Open ONE level at a time from the catalogue top — each name the EXACT
link text of the source or folder row at the current level (the phase-97
drill idiom, the ``test_kb_tree_lazy_load.py`` settle).
The top is reached through the breadcrumb's first segment when the page is
drilled (the phase-99 drill state rides ``history.state``, so a nav
re-click would KEEP the level — the crumb is the control that actually
leaves it), and is a no-op when the page already sits there (the crumb
hidden). Every click after that awaits the level it produced: the phase-167
node fetch is over once the breadcrumb names the level and no in-flight
``.kb-level-loading`` row remains."""
if not page.locator("#kb-crumb").is_hidden():
page.click('#kb-crumb a.kb-crumb-link:text-is("Knowledge base")')
expect(page.locator("#kb-crumb")).to_be_hidden(timeout=30_000)
expect(page.locator("#folders-tbody tr")).to_have_count(1, timeout=30_000)
for name in names:
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(
name, timeout=30_000
)
expect(page.locator(".kb-level-loading")).to_have_count(0)
# ---------------------------------------------------------------------------
# Helpers: the DB side
# ---------------------------------------------------------------------------
def _paths_with_counts() -> dict[str, int]:
"""``path`` → number of ``documents`` rows for that path — THE exactly-once
probe: a reset that dropped work and re-added it shows up here."""
with SessionLocal() as db:
rows = db.execute(
select(Document.path, func.count(Document.id))
.where(Document.source == SOURCE_NAME)
.group_by(Document.path)
).all()
return {str(path): int(count) for path, count in rows}
def _chunk_count_by_path(paths: list[str] | None = None) -> dict[str, int]:
"""``path`` → chunk rows per document (an absent path is not listed)."""
query = (
select(Document.path, func.count(Chunk.id))
.outerjoin(Chunk, Chunk.document_id == Document.id)
.where(Document.source == SOURCE_NAME)
.group_by(Document.path)
)
if paths is not None:
query = query.where(Document.path.in_(paths))
with SessionLocal() as db:
rows = db.execute(query).all()
return {str(path): int(count) for path, count in rows}
def _total_chunks() -> int:
with SessionLocal() as db:
return int(db.scalar(select(func.count()).select_from(Chunk)) or 0)
def _doc_ids(paths: list[str]) -> list[str]:
"""The indexed document ids of ``(SOURCE_NAME, path)``, sorted. Test 3
needs them BEFORE the prune — the orphan query runs on ids that no longer
exist afterwards."""
with SessionLocal() as db:
return sorted(
str(doc_id)
for (doc_id,) in db.execute(
select(Document.id).where(
Document.source == SOURCE_NAME, Document.path.in_(paths)
)
).all()
)
def _orphan_chunks() -> int:
"""Chunk rows whose document is gone — phase 183's contract, asserted here
too so a BATCHED prune cannot leave any behind."""
with SessionLocal() as db:
return int(
db.scalar(
select(func.count())
.select_from(Chunk)
.where(Chunk.document_id.not_in(select(Document.id)))
)
or 0
)
def _kb_snapshot() -> dict[str, tuple[str, str | None, str, str, list[str]]]:
"""The byte-level shape of the fixture source:
``path`` → ``(content, summary, content_hash, file_hash, chunk contents in
position order)`` — every field the two runs must agree on. Timestamps and
``full_path`` are deliberately absent: they belong to the run and to the
temp root, not to the imported bytes.
"""
with SessionLocal() as db:
docs = db.scalars(
select(Document)
.where(Document.source == SOURCE_NAME)
.order_by(Document.path)
).all()
snapshot: dict[str, tuple[str, str | None, str, str, list[str]]] = {}
for doc in docs:
assert doc.content_hash and doc.file_hash, doc.path
contents = db.scalars(
select(Chunk.content)
.where(Chunk.document_id == doc.id)
.order_by(Chunk.position)
).all()
snapshot[str(doc.path)] = (
doc.content,
doc.summary,
str(doc.content_hash),
str(doc.file_hash),
[str(c) for c in contents],
)
return snapshot
def _sentinel_in_kb() -> bool:
"""Does the stored content of the sentinel document carry the sentinel?
(The walk really indexed the fixture; the KB is not merely the right size.)"""
with SessionLocal() as db:
content = db.scalar(
select(Document.content).where(
Document.source == SOURCE_NAME, Document.path == SENTINEL_PATH
)
)
return bool(content and SENTINEL in content)
def _file_rows(page: Page) -> list[str]:
"""The current level's file rows, by path — the ls/tree rows the owner
reads (a pruned document must leave this list)."""
links = page.locator("#docs-tbody tr a.doc-link")
return [str(links.nth(i).inner_text()) for i in range(links.count())]
def _run_fresh_sync(page: Page, base_url: str, source: Path) -> dict[str, Any]:
"""Register the fixture and run ONE fresh-KB sync through the Sources
page, returning the shape the cross-check compares."""
_add_local_source(page, base_url, source)
body = _sync_from_sources_page(page, base_url)
return {
"detail": dict(body["detail"]),
"line": _counts_line(page),
"paths": _paths_with_counts(),
"chunks_by_path": _chunk_count_by_path(),
"total_chunks": _total_chunks(),
"snapshot": _kb_snapshot(),
}
# ---------------------------------------------------------------------------
# 1. The cadence ON syncs everything, cleanly, and the page agrees
# ---------------------------------------------------------------------------
def test_reset_cadence_produces_the_identical_run(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""``BOR_SYNC_SESSION_RESET_FILES=3`` over ten files: three mid-run
releases — and a sync whose every owner-visible number is the number the
unbounded run produces.
The counts line reads ``10 added``, the terminal reports ``added=10`` with
``errors=0``, each of the ten paths exists EXACTLY ONCE (a release that
discarded unflushed work would show up as a missing or a duplicated path),
the ls/tree rows of both folders render all ten files, the stat cards
equal the database's own ``select(count())`` over ``documents`` and
``chunks``, the sentinel's phrase is in the stored content — and not one
``/api/`` request answered 4xx/5xx or failed through the whole run."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent) # idempotent — the tree is byte-stable
login(page, app_url, next=SOURCES_URL)
problems = _watch_network(page)
leg = _run_fresh_sync(page, app_url, source_dir)
# --- the owner's settle line + the terminal detail -------------------
assert leg["line"] == f"{N_FILES} added", leg["line"]
detail = leg["detail"]
assert detail["added"] == N_FILES, detail
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
assert detail["files"] == N_FILES, detail
# --- every document landed EXACTLY once, fully embedded --------------
assert leg["paths"] == dict.fromkeys(ALL_PATHS, 1), leg["paths"]
assert all(count > 0 for count in leg["chunks_by_path"].values()), (
f"a document has no chunks — the release lost its embed work: "
f"{leg['chunks_by_path']}"
)
assert leg["total_chunks"] > N_FILES, leg["total_chunks"]
assert _sentinel_in_kb(), "the sentinel document never reached the KB"
# --- the page agrees with the database (the phase-167 totals) --------
expect(page.locator("#stat-docs")).to_have_text(str(N_FILES), timeout=15_000)
expect(page.locator("#stat-chunks")).to_have_text(
str(leg["total_chunks"]), timeout=15_000
)
# --- the ls/tree rows render the whole fixture -----------------------
_open_level(page, SOURCE_NAME, "a")
assert _file_rows(page) == [f"a/{n}.md" for n in range(1, 6)]
_open_level(page, SOURCE_NAME, "b")
assert _file_rows(page) == [f"b/{n}.md" for n in range(1, 6)]
# --- the run never errored on the wire or in the UI ------------------
assert problems == [], f"the run put an error on the wire: {problems}"
# Record for test 4's cross-check (the phase's byte-identical contract).
_CADENCE_RUN.clear()
_CADENCE_RUN.update(leg)
# ---------------------------------------------------------------------------
# 2. The unchanged re-sync with the cadence ON: nothing moves
# ---------------------------------------------------------------------------
def test_resync_is_unchanged_with_the_cadence_on(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""The run a production sync actually takes — every file unchanged — with
the release firing three times through it: the phase-147 fast path stays
the fast path.
The second sync settles ``0 added · 10 unchanged`` with ``pruned=0``, the
terminal is ``success``, and the database's chunk total and per-path chunk
counts are byte-still: the cadence neither loses rows (which the next sync
would report as ``added``) nor silently re-embeds an unchanged document."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent)
login(page, app_url, next=SOURCES_URL)
problems = _watch_network(page)
first = _run_fresh_sync(page, app_url, source_dir)
assert first["detail"]["added"] == N_FILES, first["detail"]
_expect_counts_line(page, f"{N_FILES} added")
chunks_after_first = _chunk_count_by_path()
paths_after_first = _paths_with_counts()
total_after_first = _total_chunks()
assert total_after_first == sum(chunks_after_first.values())
# --- the unchanged re-sync -------------------------------------------
second = _sync_from_sources_page(page, app_url)
detail = second["detail"]
assert detail["unchanged"] == N_FILES, detail
assert detail["added"] == 0 and detail["updated"] == 0, detail
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
# No silent re-embed: the same chunk total, per path, after the run.
assert _chunk_count_by_path() == chunks_after_first, (
"an unchanged file changed chunk count — the cadence re-embedded a "
"document the fast path had left alone"
)
assert _total_chunks() == total_after_first
assert _paths_with_counts() == paths_after_first
assert _sentinel_in_kb()
assert problems == [], f"the re-sync put an error on the wire: {problems}"
# ---------------------------------------------------------------------------
# 3. The prune through the UI at BOR_SYNC_PRUNE_BATCH=2
# ---------------------------------------------------------------------------
def test_multi_batch_prune_from_the_ui(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""Five documents go, and the prune that takes them is a MULTI-BATCH run:
at ``BOR_SYNC_PRUNE_BATCH=2`` the five deletes commit at 2, at 4 and once
more at the end — THREE commits inside the UI's own run — while the owner
still sees exactly one prune report.
The counts line settles ``0 added · 5 unchanged · 5 pruned``; the five
documents are gone from the KB AND from the ls/tree rows of BOTH folders
(three survivors in ``a``, two in ``b``); the survivors' chunk total is
unchanged; and ZERO orphan chunk rows exist. A batched commit changes WHEN
a delete becomes durable, never what the run removes."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent)
login(page, app_url, next=SOURCES_URL)
problems = _watch_network(page)
first = _run_fresh_sync(page, app_url, source_dir)
assert first["detail"]["added"] == N_FILES, first["detail"]
# The subject of the prune, captured BEFORE it happens: the doomed rows'
# ids (the orphan query needs ids that no longer exist by the time it
# runs) and every survivor's chunk count.
doomed_ids = _doc_ids(GONE_PATHS)
keep_counts = _chunk_count_by_path(KEEP_PATHS)
assert len(doomed_ids) == len(GONE_PATHS), doomed_ids
assert set(keep_counts) == set(KEEP_PATHS), keep_counts
assert all(count > 0 for count in keep_counts.values()), keep_counts
# --- the upstream deletion (this test owns the fixture tree) ---------
for rel in GONE_PATHS:
(source_dir / rel).unlink()
assert not any((source_dir / rel).exists() for rel in GONE_PATHS)
# --- the prune --------------------------------------------------------
body = _sync_from_sources_page(page, app_url)
detail = body["detail"]
assert detail["pruned"] == len(GONE_PATHS), detail
assert detail["added"] == 0 and detail["updated"] == 0, detail
assert detail["unchanged"] == len(KEEP_PATHS), detail
assert detail["errors"] == 0, detail
_expect_counts_line(
page,
f"0 added · {len(KEEP_PATHS)} unchanged · {len(GONE_PATHS)} pruned",
)
# The catalogue rows went WITH the documents — in both folders, which is
# where a prune that stopped after one batch would leave a stale row.
_open_level(page, SOURCE_NAME, "a")
assert _file_rows(page) == ["a/1.md", "a/2.md", "a/3.md"]
_open_level(page, SOURCE_NAME, "b")
assert _file_rows(page) == ["b/1.md", "b/2.md"]
# The survivors are untouched and nothing was orphaned.
assert _chunk_count_by_path() == keep_counts, (
"the prune took a survivor's chunks with it, or left a doomed row"
)
assert _total_chunks() == sum(keep_counts.values())
assert _orphan_chunks() == 0, (
"a batched prune left orphan chunk rows — the deletes committed "
"without the FK cascade doing the child work"
)
assert _sentinel_in_kb()
assert problems == [], f"the prune put an error on the wire: {problems}"
# ---------------------------------------------------------------------------
# 4. cadence 0 == cadence 3 (the byte-identical cross-check)
# ---------------------------------------------------------------------------
def test_cadence_zero_matches_cadence_three(
page: Page,
app_url: str,
zero_app_url: str,
source_dir: Path,
db_ready: None,
) -> None:
"""THE PIN the phase lives on: the cadences change the run's memory
profile and NOTHING the owner or the database can see.
The same fixture tree, walked fresh by the ZERO app
(``BOR_SYNC_SESSION_RESET_FILES=0``, ``BOR_SYNC_PRUNE_BATCH=0`` — the
pre-phase-184 run) against the CADENCE app's numbers (``3`` / ``2``): the
same paths (each exactly once), the same chunk count per path, the same
``content`` / ``summary`` / ``content_hash`` / ``file_hash`` on every
document, the same chunk contents in position order, and the same counts
line.
The reference is test 1's recorded run when the module runs in order; run
ALONE, the test walks the cadence-``3`` app itself first. Either way it
asserts the reference's own 10-documents / >10-chunks shape BEFORE
comparing, so an empty baseline can never make it pass vacuously."""
page.set_default_timeout(30_000)
# Re-seed: test 3 deleted five files from the fixture tree.
_seed_source(source_dir.parent)
on_disk = sorted(p.relative_to(source_dir).as_posix() for p in source_dir.rglob("*.md"))
assert on_disk == sorted(ALL_PATHS), on_disk
# --- the reference leg: cadence 3 (test 1's numbers, or a fresh run) --
if _CADENCE_RUN:
reference = dict(_CADENCE_RUN)
else:
login(page, app_url, next=SOURCES_URL)
reference = _run_fresh_sync(page, app_url, source_dir)
_truncate_all()
assert reference["paths"] == dict.fromkeys(ALL_PATHS, 1), reference["paths"]
assert reference["total_chunks"] > N_FILES, reference["total_chunks"]
# --- the subject leg: cadence 0, the exact pre-phase run --------------
login(page, zero_app_url, next=SOURCES_URL)
problems = _watch_network(page)
zero = _run_fresh_sync(page, zero_app_url, source_dir)
assert zero["detail"]["added"] == N_FILES, zero["detail"]
assert zero["detail"]["errors"] == 0, zero["detail"]
assert zero["detail"]["pruned"] == 0, zero["detail"]
# --- every number agrees ---------------------------------------------
assert zero["line"] == reference["line"], (
f"the owner-visible counts line moved with the cadence: "
f"{zero['line']!r} != {reference['line']!r}"
)
assert zero["paths"] == reference["paths"], (
"the set of paths (or a path's row count) moved with the cadence"
)
assert zero["chunks_by_path"] == reference["chunks_by_path"], (
"the per-path chunk counts moved with the cadence: "
f"{zero['chunks_by_path']} != {reference['chunks_by_path']}"
)
assert zero["total_chunks"] == reference["total_chunks"]
assert zero["snapshot"].keys() == reference["snapshot"].keys()
for path, row in zero["snapshot"].items():
assert row == reference["snapshot"][path], (
f"{path} is not byte-identical across the cadences"
)
assert _orphan_chunks() == 0
assert problems == [], f"the reference run put an error on the wire: {problems}"
tests/integration/test_import_session_retention.py
+1247
View File
File diff suppressed because it is too large Load Diff
tests/integration/test_importer.py
+10 -6
View File
@@ -1287,12 +1287,16 @@ def test_backfill_never_touches_standalone_and_copyless_rows(db, tmp_path) -> No
assert s.image_context_backfilled == 0
assert s.images_failed == 0 and s.errors == 0
assert llm.text_parts == [] # zero describe calls, both runs
db.expire(standalone)
db.expire(missing_copy)
assert standalone.image_context is None
assert standalone.content == DESCRIPTION
assert missing_copy.image_context is None
assert missing_copy.content == DESCRIPTION
# Phase 184: an import run is now ALLOWED to release the session it
# walks (the identity-map cadence expunges every object at the source
# boundary), so the two seeded rows are re-read by key instead of
# expiring the objects the run detached. Same assertion, one level
# closer to the database — nothing about the run's output changed.
for row in (standalone, missing_copy):
reloaded = db.get(Document, row.id)
assert reloaded is not None
assert reloaded.image_context is None
assert reloaded.content == DESCRIPTION
finally:
_cleanup_source(db, source_root.name)
tests/unit/test_config.py
+55
View File
@@ -906,3 +906,58 @@ def test_ui_customization_env_overrides(monkeypatch: pytest.MonkeyPatch) -> None
assert s.footer_text == "Powered by my own models"
monkeypatch.setenv("BOR_INPUT_PLACEHOLDER", "")
assert _settings().input_placeholder == "" # empty stands
# --- Sync session bounds (phase 184) ---
def test_sync_session_reset_files_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 184 (LOCKED A3): the import walk releases its session every
``BOR_SYNC_SESSION_RESET_FILES`` files (default 200). ``0`` is accepted
and means NEVER release — the exact pre-phase-184 behaviour, the
operator's no-redeploy rollback."""
monkeypatch.delenv("BOR_SYNC_SESSION_RESET_FILES", raising=False)
assert _settings().sync_session_reset_files == 200
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "50")
assert _settings().sync_session_reset_files == 50
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "0")
assert _settings().sync_session_reset_files == 0
def test_sync_session_reset_files_rejects_negative(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A negative cadence is a typo — a ``%`` against it never fires, so
startup refuses naming the field (the ``agent_max_rounds`` pattern,
phase 184)."""
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "-1")
with pytest.raises(ValidationError, match="sync_session_reset_files"):
_settings()
def test_sync_prune_batch_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 184 (LOCKED A3): the prune commits every
``BOR_SYNC_PRUNE_BATCH`` deleted documents (default 500). ``0`` is
accepted and means ONE commit at the end of the whole prune — the
exact pre-phase-184 behaviour."""
monkeypatch.delenv("BOR_SYNC_PRUNE_BATCH", raising=False)
assert _settings().sync_prune_batch == 500
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "25")
assert _settings().sync_prune_batch == 25
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "0")
assert _settings().sync_prune_batch == 0
def test_sync_prune_batch_rejects_negative(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A negative batch size is a typo — the batch comparison would never
fire, so startup refuses naming the field (the ``agent_max_rounds``
pattern, phase 184)."""
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
with pytest.raises(ValidationError, match="sync_prune_batch"):
_settings()
tests/unit/test_settings_catalog.py
+35
View File
@@ -188,6 +188,41 @@ def test_verbatim_display_field_types() -> None:
assert Settings.model_fields["show_text_max_blocks"].default == 5
def test_sync_session_bound_rows_are_plain_live_overridable() -> None:
"""Phase 184 (task 01, LOCKED A3/D2): the two sync-run knobs sit in
``Import & sources`` (beside ``upload_max_mb`` — they tune the sync, not
the walk's content) with the ``Sync session reset files`` / ``Sync prune
batch`` labels and are plain — not ``secret`` (two ints), not
``restart_required`` (the importer reads ``llm.settings`` per run, so a
saved override applies to the NEXT sync), overridable (the overlay is
generic over ``int`` fields). Both accept ``0`` = the exact pre-phase
behaviour (the no-redeploy rollback); a negative value fails startup,
not the panel."""
for key, label in (
("sync_session_reset_files", "Sync session reset files"),
("sync_prune_batch", "Sync prune batch"),
):
meta = CATALOG[key]
assert meta.group == "Import & sources"
assert meta.label == label
assert meta.secret is False
assert meta.restart_required is False
assert meta.overridable is True
assert key not in SECRETS
assert key not in RESTART_REQUIRED
assert key not in NON_OVERRIDABLE
def test_sync_session_bound_field_types() -> None:
"""Phase 184 (task 01, D2): both knobs are ``int``s with the LOCKED A3
defaults (``200`` files / ``500`` documents) — the type the panel renders
its input for and the overlay coerces from the stored raw string."""
assert Settings.model_fields["sync_session_reset_files"].annotation is int
assert Settings.model_fields["sync_prune_batch"].annotation is int
assert Settings.model_fields["sync_session_reset_files"].default == 200
assert Settings.model_fields["sync_prune_batch"].default == 500
def test_labels_nonempty_and_unique() -> None:
labels = [meta.label for meta in CATALOG.values()]
assert all(label.strip() for label in labels), "empty label in the catalogue"