phase: 184_sync_bounded_session
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure. **Verified / implemented (phase 184 final pass)** - All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries. - `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched. - `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact. **Tests / lint / coverage (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%). - `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**. - `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**. - Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed. - `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings. **Completion criteria** 1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green 2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅ 3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅ 4. Every existing importer/sync/removal test green with unchanged expectations → ✅ 5. Full suite green, coverage >90%, ruff + pyright clean → ✅ 6. Phase E2E green in isolation → ✅ 7. README + AGENTS accurate/consistent → ✅ 8. Atomic commit — left to the harness (I ran no git write commands). **Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written). **Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched. **Next pending phase:** `185_sync_defer_content_fast_path`.
This commit is contained in:
@@ -0,0 +1 @@
|
||||
(no final assistant message — see the .err log)
|
||||
+1
@@ -0,0 +1 @@
|
||||
error: no final assistant text
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.......................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 862 3 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6870 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -0,0 +1,29 @@
|
||||
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.
|
||||
|
||||
**Verified / implemented (phase 184 final pass)**
|
||||
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
|
||||
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
|
||||
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.
|
||||
|
||||
**Tests / lint / coverage (all 100% green)**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
|
||||
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
|
||||
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
|
||||
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
|
||||
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.
|
||||
|
||||
**Completion criteria**
|
||||
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
|
||||
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
|
||||
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
|
||||
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
|
||||
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
|
||||
6. Phase E2E green in isolation → ✅
|
||||
7. README + AGENTS accurate/consistent → ✅
|
||||
8. Atomic commit — left to the harness (I ran no git write commands).
|
||||
|
||||
**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).
|
||||
|
||||
**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.
|
||||
|
||||
**Next pending phase:** `185_sync_defer_content_fast_path`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.......................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 862 3 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6870 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
All gates green. Report:
|
||||
|
||||
**Implemented**
|
||||
- `app/config.py`: `sync_session_reset_files: int = 200` + `sync_prune_batch: int = 500` (Import & sources section, `BOR_`-prefixed, each with a non-negative `field_validator` following the `agent_max_rounds`/`compress_min_bytes` precedent; `0` = exact pre-phase-184 behaviour, documented).
|
||||
- `app/core/settings_catalog.py`: `CATALOG` rows `"Sync session reset files"` / `"Sync prune batch"` in `"Import & sources"` — plain, live, overridable.
|
||||
- `.env.example`: new `--- Sync session bounds (phase 184) ---` block (defaults, `0` semantics, the batched-prune atomicity note, "inert until tasks 02/03").
|
||||
- NEW `tests/integration/test_import_session_retention.py` — the retention probe with the `_map_size()` door: added-run release, the unchanged-run staircase pin (`0,1,…,11` → exactly 12 `Document`s resident, content included, **not** collectable, `expunge_all()` releases all), the `0`-means-pre-phase rollback pin, and the weak-clean/dirty-strong mechanism test.
|
||||
- Unit pins: 4 in `tests/unit/test_config.py` (defaults, env override, `0` accepted, negative refuses startup), 2 in `tests/unit/test_settings_catalog.py` (group/label/flags + field types/defaults).
|
||||
|
||||
**Gates** — `uv run pytest --cov=app --cov-report=term-missing` → **4498 passed**, `TOTAL 99%` (`app/config.py` 100%, `settings_catalog.py` 100%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors; `tests/integration/test_import_session_retention.py tests/unit/test_settings_catalog.py -v` → 19 passed (run 4× for flake-stability); sanity E2E in isolation: `test_admin_settings_panel.py` 12 passed, `test_sync_changed_only.py` 3 passed.
|
||||
|
||||
**Docs** — `README.md`: two rows in the Configuration reference "Import & sources" table (flagged *inert until task 02/03 wires it*, since nothing reads them yet). No `AGENTS.md` change (it lists no env knobs).
|
||||
|
||||
**Notable findings / deviations (important for task 02)**
|
||||
- D1's premise is **CONFIRMED in effect but wrong in mechanism**: the identity map is *weak*, so the ADD run does **not** accumulate (84/84 present only until one `gc.collect()`, which drops it to 0) — the task's `len(identity_map) >= 12*2` pin on a fresh add run is **not true**. The real, monotone, KB-proportional retention is the **unchanged** path: `_unchanged_doc`'s `doc.file_hash = file_digest` stamp marks every row `modified` while `session.is_modified()` is false (no UPDATE, commit skipped), so the row lands in `Session.dirty` — a *strong* session-side set — and survives every collection with its full `content` (extrapolated ≈265 MB for the 17,407-doc / 15.2 KB KB). `expire_on_commit=False` is not the retainer; the phase still must not change it.
|
||||
- Hence `expunge_all()` **is** the right lever, but it discards unflushed changes: safe only at D3's point (per-file commit inside `_index_file`, `importer.py:1575`) — written into the module docstring as a hazard note for task 02. Task 02's post-fix bound against this fixture is "≤ reset + slack" starting from `MEASURED_UNCHANGED_MAP_SIZE = 12`.
|
||||
- RSS is recorded, never asserted (inherited A4 rule): ≈35 KB/file, unreturned after `gc.collect()` while live-object count stays flat — allocator, not session; `expunge_all` cannot fix that part.
|
||||
|
||||
**Next pending task:** `.agents/phases/todo/184_sync_bounded_session/02_session_reset_cadence.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 855 4 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6863 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
**Implemented (phase 184 task 02)**
|
||||
- `app/rag/importer.py` `import_sources`: `reset_every = llm.settings.sync_session_reset_files` read ONCE before the source loop; `session.expunge_all()` at exactly two guarded points — top of the per-file loop (`summary.files % reset_every == 0`) and once per source before the two post-passes — with the full safety argument (commits already done, `expunge_all` discards only the fast path's no-op stamp, hooks/`dates_map`/`seen`/`total` plain, rollback path touches nothing) plus a docstring paragraph. No other call sites.
|
||||
- `tests/integration/test_import_session_retention.py`: added `test_reset_cadence_bounds_the_map` (cadence 3 vs the cadence-0 reference: identical `ImportSummary` via `asdict`, identical DB per path/content/summary/`file_hash`/chunk-count; end map ≤ `MAP_BOUND`=17, mid-run ≤ 2×7 objects), `test_reset_cadence_bounds_the_unchanged_walk` (sawtooth `[0,1,2]×4` vs the pre-fix ramp, end map `0` → pins the source boundary), `test_reset_cadence_survives_an_embedding_error` (stub refuses file #4 with cadence 2: `errors=1`, 11 survivors × 6 chunks, `changed_paths` correct); kept both cadence-`0` guarantee tests; docstring flipped to `CONFIRMED → FIXED` with both measured map sizes.
|
||||
- Fixed one stale existing test: `tests/integration/test_importer.py::test_backfill_never_touches_standalone_and_copyless_rows` called `db.expire()` on objects a run-supplied session may now detach → re-reads by key. Counts/log expectations unchanged (only affected test in 4501).
|
||||
|
||||
**Gates (all green)**
|
||||
- `uv run pytest tests/integration/test_import_session_retention.py --no-cov` → 7 passed
|
||||
- importer/importer_e2e/importer_dates/sync_api/unit-importer + retention → 129 passed
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **4501 passed**, `app/rag/importer.py` 99%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
|
||||
- E2E spot-checks in isolation: `test_sync_changed_only.py` 3 passed; `test_prune_cascade_cleanup.py` + `test_sync_summary_visibility.py` 6 passed
|
||||
|
||||
**Docs**: `README.md` (reset knob no longer "inert", notes the per-source release), `.env.example` (status line: walk reads `BOR_SYNC_SESSION_RESET_FILES`; prune knob still inert until task 03), `AGENTS.md` (new rule 13: release only at the proven-safe points).
|
||||
|
||||
**Decisions/deviations**: added the unchanged-walk test beyond the task's list because task 01 proved the retention lives on the unchanged path (the ADD run's rows are GC-collectable, so its bound alone pins nothing); `MAP_BOUND = RESET_FILES + (RESET_FILES-1)×7 = 17` documented as a ceiling, measured end-of-run 0. No live-infra changes.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/184_sync_bounded_session/03_batched_prune.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
..................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 860 4 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6868 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
All green. Final report:
|
||||
|
||||
**Implemented (task 03 — `184_sync_bounded_session/03_batched_prune.md`)**
|
||||
- `app/rag/importer.py` `_prune`: the candidate walk is now `select(Document).options(load_only(id, source, path, is_image, image_path, page_count)).where(source.in_(…)).order_by(Document.id))` — `content` (avg 15.2 KB) never loaded; `from sqlalchemy.orm import load_only` added.
|
||||
- `_prune(..., prune_batch: int = 500)` keyword; commits every Nth delete, and the final commit is skipped only when the last batch boundary already committed (so `10` → exactly 3 commits for 30 docs; `0` → exactly the pre-phase 1 commit). Caller (`import_sources`, L1034) passes `prune_batch=llm.settings.sync_prune_batch` (read once per run).
|
||||
- Docstring: the light-row contract (per-column rationale for the six, why `content` is out), the batch-commit contract, and the LOCKED A2 atomicity note (crash leaves a prefix pruned; `seen` recomputed from disk → next sync converges).
|
||||
- Tests (`tests/integration/test_import_session_retention.py`, +5 prune tests + a `before_cursor_execute` capture): no `documents.content` in the prune SELECT and projection == the six columns; **exactly 1** `FROM documents` SELECT per prune (no per-row lazy load); page-copy cleanup (`id`+`page_count`) proven off light rows; 0 orphan chunks; commit counts 3 vs 1; the `Settings → _prune` wiring via a forwarding spy; both prune guards (phase-122 images-off, phase-126 parent-PDF) on light rows. Mutation-checked: dropping `page_count`/`image_path`, loading full rows, the unconditional trailing commit and dropping the caller kwarg each FAIL a test.
|
||||
|
||||
**Gates (all 100% green)**
|
||||
- `uv run pytest tests/integration/test_import_session_retention.py -q --no-cov` → 12 passed
|
||||
- `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/unit/test_importer.py tests/unit/test_importer_ignore.py` → 120 passed
|
||||
- `uv run pytest` → **4506 passed**; `--cov=app` TOTAL **99%** (`app/rag/importer.py` 99%)
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
- E2E spot-check (unchanged phases): `test_prune_cascade_cleanup.py` 3 passed, `test_sync_changed_only.py`+`test_sync_summary_visibility.py` 6 passed
|
||||
|
||||
**Docs**: `README.md` — new "A sync's memory is bounded per RUN, not per knowledge base" operations bullet + `BOR_SYNC_PRUNE_BATCH` row no longer "inert" + CLI `--prune` knob note; `AGENTS.md` — new rule 14 (six-column prune, batch commits, no bulk `delete()`/`raiseload`); `.env.example` — both knobs now live.
|
||||
|
||||
**Decisions**: default `prune_batch=500` (task-specified); `scripts/import_docs.py` has no direct `_prune` call (it goes through `import_sources`, so its counts/log lines are unchanged and it follows the setting). Deviation from the task's literal snippet: the end-of-loop commit is guarded so a batch that divides the count emits no redundant empty transaction — this is what makes the pinned count exactly 3.
|
||||
|
||||
**Next pending task**: `184_sync_bounded_session/04_e2e_bounded_sync.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.......................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 862 3 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6870 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
All gates are green and the working tree is final. Report:
|
||||
|
||||
**Task 04 complete — phase 184's dedicated Playwright suite + docs pass.**
|
||||
|
||||
- **New** `tests/e2e/test_sync_session_reset.py` (919 lines, mock-only, `E2E_REAL_LLM=1` skipped): two module apps on their own ports (`E2E_APP_PORT_SYNCRESET` 8200 = reset 3 / batch 2, `E2E_APP_PORT_SYNCRESET_ZERO` 8201 = both `0`), one 10-file/two-folder fixture source with a sentinel, shared E2E Postgres truncated before **and** after every test, admin login via `auth_helpers`, `/api/` 4xx/5xx + request-failure watcher.
|
||||
- Tests → observables: ① cadence-ON walk lands all 10 docs **exactly once** (`10 added`, `errors=0`, stat cards == `select(count())`, both folders' ls/tree rows, no wire errors); ② unchanged re-sync `0 added · 10 unchanged`, per-path chunk counts byte-still (no silent re-embed); ③ five-file prune at `BOR_SYNC_PRUNE_BATCH=2` (3 commits inside the UI run) → `0 added · 5 unchanged · 5 pruned`, rows gone from both folders, survivors' chunks intact, **0 orphan chunk rows**; ④ the byte-identical cross-check — cadence-`0` leg vs the recorded cadence-`3` leg: same paths/counts, `content`, `summary`, `content_hash`, `file_hash`, chunk contents in position order, same counts line (non-vacuous: it pins its own baseline shape first).
|
||||
- Gates: `pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**; `test_sync_changed_only.py` → 3 passed; `test_sync_git_freeze.py` → 2 passed; `pytest --cov=app` → **4506 passed, app/ TOTAL 99%**; `ruff check .` → All checks passed; `pyright` → **0 errors, 0 warnings**.
|
||||
- Docs: `README.md` (phase-184 operations paragraph, both knobs in the env table with defaults/`0` semantics, CLI-import note, E2E pointer), `AGENTS.md` rules 13/14 (now cite the E2E suite; corrected the prune pin to "exactly one **document-wide** SELECT"), `.env.example` (dropped the task-progress wording; added the two fields it omitted project-wide — `BOR_APP_VERSION`, `BOR_SESSION_COOKIE` — so every `Settings` field is exemplified). Audited programmatically: every `Settings` field documented in README and `.env.example`.
|
||||
- No defects found in app code; no locked decision touched. RSS is asserted nowhere (phase 183 A4 inherited) — the memory teeth stay in `tests/integration/test_import_session_retention.py`.
|
||||
- **Left to the harness:** the atomic `--no-gpg-sign` commit and the `todo/ → complete/` move (I ran no git write commands, per the pipeline rules).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
........................................................................ [ 1%]
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.......................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/admin_settings.py 114 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 292 1 99%
|
||||
app/api/chat_images.py 52 0 100%
|
||||
app/api/chats.py 111 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 239 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 204 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 318 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/compression.py 59 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 43 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/http_cache.py 22 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/response_budget.py 63 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/settings_catalog.py 18 0 100%
|
||||
app/core/settings_migrate.py 32 0 100%
|
||||
app/core/settings_overlay.py 111 0 100%
|
||||
app/core/theming.py 40 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 105 0 100%
|
||||
app/models.py 141 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 494 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 862 3 99%
|
||||
app/rag/llm.py 253 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 78 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 162 0 100%
|
||||
app/rag/retriever.py 252 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/rag/svg_text.py 111 0 100%
|
||||
app/rag/verbatim.py 160 0 100%
|
||||
app/schemas.py 549 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 6870 16 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
@@ -21,6 +21,7 @@
|
||||
# --- App ---
|
||||
BOR_ENVIRONMENT=development
|
||||
# BOR_APP_NAME=Brain of Reese # display name on all pages — titles, header brand, status labels, aria text (phase 39)
|
||||
# BOR_APP_VERSION=0.1.0 # version the app reports (GET /api/health, the OpenAPI doc)
|
||||
# BOR_INPUT_PLACEHOLDER=Ask me anything… # composer placeholder, chat page (phase 62)
|
||||
# BOR_FOOTER_TEXT=Powered by self-hosted models # footer line on every page (phase 62)
|
||||
# (Phase 91: the retired CSS-file theme env var is gone — the colors
|
||||
@@ -157,6 +158,28 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
|
||||
# BOR_UPLOAD_MAX_MB=512 # caps BOTH the compressed upload and the total
|
||||
# extracted bytes (zip-bomb guard); must be > 0
|
||||
|
||||
# --- Sync session bounds (phase 184) ---
|
||||
# The two knobs that bound ONE sync run, so its cost stops scaling with
|
||||
# the size of the whole knowledge base. Both read per run (the importer
|
||||
# reads its settings per sync), so a value saved in the admin Settings
|
||||
# panel applies to the NEXT sync - no restart. 0 = the exact
|
||||
# pre-phase-184 behaviour on both (the no-redeploy rollback); a negative
|
||||
# value is a typo and refuses startup, naming the field.
|
||||
# The walk reads BOR_SYNC_SESSION_RESET_FILES (released every Nth file plus
|
||||
# once per source, before the two post-passes) and the prune reads
|
||||
# BOR_SYNC_PRUNE_BATCH (one COMMIT every Nth delete, over a row that loads
|
||||
# six columns and never the document's content).
|
||||
# BOR_SYNC_SESSION_RESET_FILES=200 # files between session releases
|
||||
# (expunge_all) inside the import walk;
|
||||
# 0 = never release anything (pre-phase)
|
||||
# BOR_SYNC_PRUNE_BATCH=500 # pruned documents per COMMIT inside the
|
||||
# prune instead of one commit at the end
|
||||
# of the whole import; 0 = one commit at
|
||||
# the end (pre-phase). A crash mid-prune
|
||||
# then leaves a PREFIX pruned - the next
|
||||
# sync converges (seen is recomputed
|
||||
# from disk every run)
|
||||
|
||||
# --- Image documents (phase 122) ---
|
||||
# Master switch for indexing standalone image files (direct uploads,
|
||||
# uploaded archives, git/local source walks) as first-class documents.
|
||||
@@ -257,6 +280,8 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
|
||||
BOR_ADMIN_PASSWORD=
|
||||
BOR_SESSION_SECRET=
|
||||
# BOR_SESSION_MAX_AGE=43200 # signed-cookie lifetime, seconds (default 12 h, sliding)
|
||||
# BOR_SESSION_COOKIE=bor_session # cookie NAME; boot-wired, so a change here
|
||||
# needs a restart (the panel marks it the same)
|
||||
|
||||
# --- Debugging (0/1 — 1 enables attach-on-demand debugpy on port 5678) ---
|
||||
DEBUGPY=0
|
||||
|
||||
@@ -133,6 +133,45 @@
|
||||
`doc.chunks = [...]` **re-embed** path is the deliberate, documented
|
||||
exception (collection replacement legitimately needs the loaded
|
||||
collection; changed files only).
|
||||
13. **Release the sync session at the proven-safe points only:** an import
|
||||
run holds one session and `expire_on_commit=False` (`app/db.py`) means a
|
||||
commit releases nothing, so `_unchanged_doc`'s `file_hash` stamp keeps
|
||||
every walked `Document` — content included — resident in the session's
|
||||
own strong `dirty` set (not collectable): memory grew with the KB. The
|
||||
cadence is therefore `session.expunge_all()` every
|
||||
`BOR_SYNC_SESSION_RESET_FILES` files at the TOP of the per-file loop and
|
||||
once per source before the two post-passes (phase 184; `0` = never, the
|
||||
pre-phase rollback). `expunge_all` DISCARDS unflushed changes, so never
|
||||
move it into `_index_file`, `_unchanged_doc`, `_store_summary` or a
|
||||
post-pass — their locals are live across statements — and never add a
|
||||
reset before a file's per-file commit. Pinned by
|
||||
`tests/integration/test_import_session_retention.py` (bounded with the
|
||||
cadence on, unbounded with it `0`, byte-identical summaries and DB) and
|
||||
through the UI by `tests/e2e/test_sync_session_reset.py` (two module apps,
|
||||
one per cadence: every document exactly once, the all-`unchanged` re-sync,
|
||||
and the `0`-cadence run byte-identical to the `3`-cadence one).
|
||||
14. **The prune reads six columns and commits in batches:** `_prune`
|
||||
selects its candidates through
|
||||
`load_only(id, source, path, is_image, image_path, page_count)`
|
||||
`order_by(Document.id)` — the exact set the phase-122 images-off guard,
|
||||
the phase-126 parent-PDF rule and the image/page copy cleanup read, and
|
||||
NOT `content` (252 MB of markdown per full KB walk) — and commits every
|
||||
`BOR_SYNC_PRUNE_BATCH` deletes (`0` = the pre-phase single commit;
|
||||
phase 184). Never read another `Document` attribute inside that loop (a
|
||||
deferred column costs one lazy SELECT per document and unbounds the
|
||||
walk — no `raiseload` paper-over, add the column instead), and never
|
||||
swap the per-document `session.delete(doc)` for a bulk `delete()` (the
|
||||
guards, `changed_paths` and the log line depend on the per-document
|
||||
shape). A non-zero batch deliberately trades the all-or-nothing prune
|
||||
for a prefix (LOCKED A2): `seen` is recomputed from disk, so the next
|
||||
sync converges. Pinned by
|
||||
`tests/integration/test_import_session_retention.py` (the captured
|
||||
`SELECT`'s column list, exactly one document-wide `SELECT` per prune
|
||||
(a second one is a per-row lazy load), the commit counts at `10` and
|
||||
`0`, both guards on light rows) and
|
||||
`tests/e2e/test_sync_session_reset.py` (the five-document prune at batch
|
||||
`2` through the Sources page: both folders cleared, the survivors' chunks
|
||||
intact, zero orphan chunk rows).
|
||||
|
||||
## Quick reference
|
||||
```bash
|
||||
|
||||
@@ -457,7 +457,11 @@ reflect the new files.
|
||||
its `chunks` rows (the embeddings) go together, the latter through the
|
||||
database's own `ON DELETE CASCADE`, so the cost of a mass prune is
|
||||
bounded by documents, not chunks (see *A mass prune costs documents,
|
||||
not chunks* under [Sync from the UI](#sync-from-the-ui)).
|
||||
not chunks* under [Sync from the UI](#sync-from-the-ui)). The CLI run
|
||||
follows the same two phase-184 knobs as the UI sync
|
||||
(`BOR_SYNC_SESSION_RESET_FILES`, `BOR_SYNC_PRUNE_BATCH`): it loads only
|
||||
the prune's six columns and commits every Nth delete (`0` on either knob
|
||||
restores the pre-phase-184 run).
|
||||
- The import prints one line per file and ends with a greppable summary,
|
||||
so it is safe to run from cron or after every commit.
|
||||
- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman
|
||||
@@ -651,6 +655,34 @@ only) runs the whole source refresh in one click:
|
||||
that fails the suite if the cascade clause ever disappears; the owner
|
||||
flow itself (delete upstream → re-sync → exactly those documents and
|
||||
chunks gone) is pinned by `tests/e2e/test_prune_cascade_cleanup.py`.
|
||||
- **A sync's memory is bounded per RUN, not per knowledge base** (phase
|
||||
184). One import run holds one session, and `expire_on_commit=False`
|
||||
(`app/db.py`) means a commit releases nothing — so before this phase the
|
||||
run kept every row it touched (each unchanged document with its full
|
||||
~15 KB `content`) until it finished, and the prune read every document of
|
||||
every pruned source whole, deletes held in ONE transaction. Now the walk
|
||||
releases its session every `BOR_SYNC_SESSION_RESET_FILES` files (plus once
|
||||
per source), and the prune (`_prune`) reads only the SIX columns its
|
||||
guards and its copy cleanup need (`id`, `source`, `path`, `is_image`,
|
||||
`image_path`, `page_count` — never `content`) ordered by id, committing
|
||||
every `BOR_SYNC_PRUNE_BATCH` deletes. **Where the operator sets the
|
||||
ceiling:** those two `BOR_`-prefixed settings (defaults `200` /
|
||||
`500`, live on the next sync without a restart, `0` on either one restores
|
||||
the exact pre-phase-184 run as the no-redeploy rollback). Counts, log
|
||||
lines, `changed_paths`, the phase-122/126 prune guards and the database
|
||||
are byte-identical either way — pinned by
|
||||
`tests/integration/test_import_session_retention.py` (the identity-map
|
||||
bound, the no-`content` prune `SELECT` with no per-row lazy load, the
|
||||
3-commits-vs-1 cadence, both prune guards on light rows) and end to end
|
||||
through the Sources page by `tests/e2e/test_sync_session_reset.py` — two
|
||||
module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
|
||||
`E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
|
||||
every document exactly once with the totals the stat cards show, the
|
||||
re-sync is all-`unchanged` with an unchanged chunk total, a five-document
|
||||
prune at batch `2` clears both folders with zero orphan chunk rows, and a
|
||||
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
|
||||
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
|
||||
line).
|
||||
- **Cancel**: while a run is in flight, a **Cancel** button appears
|
||||
next to the sync button; clicking it posts `POST /api/sync/cancel`
|
||||
(admin only) and registers a cooperative cancel (`202`) — the run
|
||||
@@ -1548,6 +1580,8 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
|
||||
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
|
||||
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
||||
| `BOR_SYNC_SESSION_RESET_FILES` | `200` | Files between releases of the import run's SQLAlchemy session (phase 184): every Nth file the walk hands back everything the session still holds (`expunge_all`), so an unchanged-file sync stops keeping one full document row per file walked. `0` = never release — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. The release also runs once per source, before the two post-passes |
|
||||
| `BOR_SYNC_PRUNE_BATCH` | `500` | Pruned documents per COMMIT inside the prune (phase 184) instead of one commit at the end of the whole import, so a mass prune releases its deletes as it goes instead of holding one enormous transaction. `0` = one commit at the end — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. With a non-zero batch a crash mid-prune leaves a **prefix** pruned, and the next sync converges (`seen` is recomputed from disk every run) |
|
||||
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
||||
|
||||
### Image support (phases 122/123/124/181)
|
||||
|
||||
@@ -446,6 +446,29 @@ class Settings(BaseSettings):
|
||||
#: the validator fails loudly at startup (the ``agent_max_rounds``
|
||||
#: pattern).
|
||||
upload_max_mb: int = 512
|
||||
#: Files between identity-map releases inside
|
||||
#: ``app.rag.importer.import_sources`` (phase 184,
|
||||
#: ``BOR_SYNC_SESSION_RESET_FILES``): every Nth file of the walk — at
|
||||
#: the one point where no object from a finished file is reachable —
|
||||
#: the run's session calls ``expunge_all()``, so whatever the walk
|
||||
#: still holds goes before the next file is read. ``0`` = NEVER reset
|
||||
#: — the exact pre-phase-184 behaviour (the operator's no-redeploy
|
||||
#: rollback, the ``BOR_LOG_LARGE_RESPONSE_BYTES`` precedent). A
|
||||
#: negative value is a typo — the validator fails startup loudly (the
|
||||
#: ``agent_max_rounds`` pattern). Read per run (the importer reads
|
||||
#: ``llm.settings``, so a saved override applies to the NEXT sync).
|
||||
sync_session_reset_files: int = 200
|
||||
#: Pruned documents per commit inside ``app.rag.importer._prune``
|
||||
#: (phase 184, ``BOR_SYNC_PRUNE_BATCH``): the delete loop commits
|
||||
#: every Nth document instead of once at the end of the whole import,
|
||||
#: so a mass prune releases its rows as it goes. ``0`` = ONE commit at
|
||||
#: the end — the exact pre-phase-184 behaviour (the same rollback
|
||||
#: knob). A negative value is a typo — the validator fails startup
|
||||
#: loudly (the ``agent_max_rounds`` pattern). Note the deliberate
|
||||
#: atomicity change of a non-zero batch (phase 184 A2): a crash
|
||||
#: mid-prune leaves a PREFIX pruned, and the next sync converges
|
||||
#: because ``seen`` is recomputed from the filesystem every run.
|
||||
sync_prune_batch: int = 500
|
||||
|
||||
# --- Image documents (phase 122: standalone images as documents) ---
|
||||
#: Master switch for image-document indexing (phase 122,
|
||||
@@ -809,6 +832,34 @@ class Settings(BaseSettings):
|
||||
raise ValueError("upload_max_mb must be > 0 (MiB)")
|
||||
return v
|
||||
|
||||
@field_validator("sync_session_reset_files")
|
||||
@classmethod
|
||||
def _sync_session_reset_files_non_negative(cls, v: int) -> int:
|
||||
"""``0`` is the pre-phase-184 rollback (never reset) — a negative
|
||||
value is a typo that would make the cadence comparison constant,
|
||||
so it fails startup naming the field (the
|
||||
``agent_max_rounds`` pattern, phase 184)."""
|
||||
if v < 0:
|
||||
raise ValueError(
|
||||
"sync_session_reset_files must be >= 0 (0 = never reset the"
|
||||
" import session)"
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("sync_prune_batch")
|
||||
@classmethod
|
||||
def _sync_prune_batch_non_negative(cls, v: int) -> int:
|
||||
"""``0`` is the pre-phase-184 rollback (one commit at the end of
|
||||
the prune) — a negative value is a typo that would make the batch
|
||||
comparison constant, so it fails startup naming the field (the
|
||||
``agent_max_rounds`` pattern, phase 184)."""
|
||||
if v < 0:
|
||||
raise ValueError(
|
||||
"sync_prune_batch must be >= 0 (0 = one commit at the end of"
|
||||
" the prune)"
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("chat_image_max_mb")
|
||||
@classmethod
|
||||
def _chat_image_max_mb_positive(cls, v: int) -> int:
|
||||
|
||||
@@ -288,6 +288,12 @@ CATALOG: dict[str, SettingMeta] = {
|
||||
"upload_max_mb": SettingMeta(
|
||||
"upload_max_mb", "Import & sources", "Upload max MB"
|
||||
),
|
||||
"sync_session_reset_files": SettingMeta(
|
||||
"sync_session_reset_files", "Import & sources", "Sync session reset files"
|
||||
),
|
||||
"sync_prune_batch": SettingMeta(
|
||||
"sync_prune_batch", "Import & sources", "Sync prune batch"
|
||||
),
|
||||
# --- Images ---
|
||||
"images": SettingMeta("images", "Images", "Images"),
|
||||
"image_extensions": SettingMeta(
|
||||
|
||||
+130
-6
@@ -346,7 +346,7 @@ from typing import Any, Protocol
|
||||
|
||||
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy.orm import Session, load_only
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
@@ -867,6 +867,14 @@ async def import_sources(
|
||||
failure leaves the row untouched — the next sync retries it).
|
||||
Driven by committed doc rows, so it is safe under ``limit`` too
|
||||
(it deletes nothing).
|
||||
|
||||
Bounded session (phase 184, task 02): the walk releases its SQLAlchemy
|
||||
identity map every ``llm.settings.sync_session_reset_files`` files and
|
||||
once per source before the two post-passes, so the run's resident set
|
||||
scales with the CADENCE rather than with the size of the knowledge base
|
||||
(``0`` = never — the exact pre-phase behaviour, the operator's
|
||||
no-redeploy rollback). The cadence changes no count, no log line and no
|
||||
committed byte: see the safety argument at the reset site.
|
||||
"""
|
||||
if limit is not None and limit <= 0:
|
||||
raise ValueError("limit must be >= 1")
|
||||
@@ -875,6 +883,12 @@ async def import_sources(
|
||||
# toggle is on — the empty set admits nothing, so the toggle-off run
|
||||
# (walk, counts, prune) stays byte-identical to pre-phase.
|
||||
image_exts = llm.settings.image_extension_set if llm.settings.images else frozenset()
|
||||
# Phase 184 (task 02): the identity-map release cadence, read ONCE here —
|
||||
# before the source loop — so a settings overlay saved mid-run cannot
|
||||
# change the cadence between two files of the same walk (the importer
|
||||
# reads ``llm.settings`` per RUN, never per file). ``0`` disables the
|
||||
# release entirely.
|
||||
reset_every = llm.settings.sync_session_reset_files
|
||||
owns_session = session is None
|
||||
if session is None:
|
||||
session = SessionLocal()
|
||||
@@ -922,6 +936,40 @@ async def import_sources(
|
||||
):
|
||||
if limit is not None and summary.files >= limit:
|
||||
break
|
||||
# Phase 184 (task 02): the identity-map release. There is NO
|
||||
# automatic one — ``expire_on_commit=False`` (``app/db.py:30``)
|
||||
# means a commit drops nothing, and ``_unchanged_doc``'s
|
||||
# ``doc.file_hash = file_digest`` stamp marks every
|
||||
# already-current row DIRTY, which puts it in the session's
|
||||
# own strong ``dirty`` set (cyclic GC cannot reach it, see
|
||||
# ``tests/integration/test_import_session_retention.py``).
|
||||
# Without this the run holds every Document — its full
|
||||
# ``content`` included — it walked until the import ENDED:
|
||||
# the 2026-10-15 incident's second driver, and the reason
|
||||
# memory grew with the KB rather than with the work.
|
||||
#
|
||||
# This is the ONLY safe point, and WHY it is safe:
|
||||
# ``expunge_all()`` discards UNFLUSHED changes, so it may run
|
||||
# only where every real write of the finished file is already
|
||||
# committed. Here the previous file's ``_index_file`` has
|
||||
# RETURNED — each path commits per file (the changed path
|
||||
# commits its chunks, ``_store_summary`` commits, and
|
||||
# ``_unchanged_doc`` commits its date refresh, its stamp and
|
||||
# whatever the phase-163 strip re-gate wrote) — and what
|
||||
# ``expunge_all`` then drops is at most the fast path's stamp
|
||||
# of a value the database ALREADY holds (no UPDATE was
|
||||
# pending: ``session.is_modified`` is false for it). Nothing
|
||||
# downstream reaches back into a finished file: ``_index_file``
|
||||
# returns ``None``, the progress/chunk/image hooks only assign
|
||||
# fields on the sync's own status object, ``dates_map``/
|
||||
# ``ignore``/``seen`` are plain data, and ``total`` was
|
||||
# computed before the loop. The ``EmbeddingError`` handler
|
||||
# below calls ``session.rollback()`` and touches no object of
|
||||
# its own, so detached state is irrelevant to it.
|
||||
# ``reset_every == 0`` → never (the exact pre-phase
|
||||
# behaviour, the operator's rollback knob).
|
||||
if reset_every and summary.files and summary.files % reset_every == 0:
|
||||
session.expunge_all()
|
||||
rel = path.relative_to(root).as_posix()
|
||||
seen.add((source, rel))
|
||||
summary.files += 1
|
||||
@@ -955,6 +1003,13 @@ async def import_sources(
|
||||
session.rollback()
|
||||
summary.errors += 1
|
||||
logger.error("import: error source=%s path=%s — %s", source, rel, e)
|
||||
# Phase 184 (task 02): the source-boundary release — one source's
|
||||
# walk residue never crosses into the next source. Safe for the
|
||||
# same reason as the per-file reset and additionally because BOTH
|
||||
# post-passes below select the rows they need fresh (their own
|
||||
# ``select(Document)``), so no walk object has to survive it.
|
||||
if reset_every:
|
||||
session.expunge_all()
|
||||
# Phase 125 (task 04): the post-index enrichment pass — the
|
||||
# markdown-FIRST case (a markdown indexed while its
|
||||
# referenced image had no document yet recorded the
|
||||
@@ -980,6 +1035,10 @@ async def import_sources(
|
||||
session, source_names, seen, images=llm.settings.images,
|
||||
summary=summary,
|
||||
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
|
||||
# Phase 184 (task 03): the commit cadence, read ONCE
|
||||
# here (the same per-RUN rule as the identity-map
|
||||
# reset) — ``0`` = the pre-phase single commit.
|
||||
prune_batch=llm.settings.sync_prune_batch,
|
||||
)
|
||||
summary.embed_batches = llm.embed_batches
|
||||
summary.log()
|
||||
@@ -3864,6 +3923,7 @@ def _prune(
|
||||
images: bool = False,
|
||||
summary: ImportSummary | None = None,
|
||||
page_dir: Path | None = None,
|
||||
prune_batch: int = 500,
|
||||
) -> int:
|
||||
"""Delete documents of *source_names* whose file is no longer in *seen*.
|
||||
|
||||
@@ -3910,10 +3970,51 @@ def _prune(
|
||||
chunk row, never hydrates a 768-dim embedding, and a mass prune costs
|
||||
documents instead of chunks × 24.8 KiB (the 2026-10-15 incident: a
|
||||
3,008-doc prune materialised ~720 MB to delete 72k rows and locked up
|
||||
a 3,904 MB VM with no swap). The ``changed_paths`` feed and the
|
||||
image/page copy cleanup above are unchanged, and they only ever read
|
||||
a 3,904 MB VM with no swap). That fix removed the CHILD rows; a SECOND
|
||||
driver of the same incident stood on this very walk, and phase 184
|
||||
(task 03) removed it with the two contracts below.
|
||||
|
||||
Phase 184 (task 03) — the light-row contract. The walk below selects
|
||||
through ``load_only`` and is ordered by ``Document.id``, so its resident
|
||||
set is independent of the knowledge base: ``content`` averages 15.2 KB
|
||||
per row (252 MB of markdown across the incident KB), and loading it for
|
||||
every document of every pruned source — to decide a deletion and then
|
||||
throw the string away — was that driver. The SIX columns loaded are
|
||||
exactly what the guards and the cleanup read: ``is_image`` (the
|
||||
phase-122 images-off toggle guard), ``path`` (that same guard's
|
||||
``seen`` tuple and the phase-126 ``_EMBEDDED_IMAGE_RE`` parent-PDF
|
||||
rule), ``source`` (the ``seen`` tuple, ``changed_paths`` and the log
|
||||
line), ``image_path`` (:func:`_delete_image_copy`), ``id`` +
|
||||
``page_count`` (:func:`_delete_page_copies`) — and ``content`` is NOT
|
||||
among them (nor ``summary``, ``full_path``, ``title``,
|
||||
``image_context``, either hash, or any date). Reading anything else
|
||||
inside the loop would emit one lazy SELECT per document and undo the
|
||||
bound; ``tests/integration/test_import_session_retention.py`` pins both
|
||||
halves (no ``documents.content`` in the SELECT, exactly ONE
|
||||
``FROM documents`` SELECT for the whole walk). Phase 185 does the same
|
||||
job for the walk's fast path.
|
||||
|
||||
Phase 184 (task 03) — the batch-commit contract. *prune_batch* is
|
||||
``Settings.sync_prune_batch`` (``BOR_SYNC_PRUNE_BATCH``, read once by
|
||||
the caller from ``llm.settings``): the loop commits every Nth delete,
|
||||
so a long prune no longer holds one enormous transaction and its rows
|
||||
are released as it goes. ``prune_batch=0`` is the exact pre-phase
|
||||
behaviour — ONE commit after the whole loop — the operator's
|
||||
no-redeploy rollback (LOCKED A3).
|
||||
|
||||
**Atomicity (LOCKED A2, deliberate).** With a non-zero batch a crash
|
||||
mid-prune leaves a PREFIX pruned instead of rolling the whole prune
|
||||
back. That is safe HERE, and only here: ``seen`` is recomputed from the
|
||||
filesystem on every run, so the next sync converges on whatever this
|
||||
one left, and the import already committed per file throughout the walk
|
||||
— a run-level transaction never existed to lose. ``summary.pruned``
|
||||
stays the count of documents actually deleted, unchanged in meaning.
|
||||
|
||||
What phase 183 left in place is unchanged: the ``changed_paths`` feed
|
||||
and the image/page copy cleanup above, which only ever read
|
||||
``doc.image_path`` / ``doc.page_count`` / ``doc.source`` / ``doc.path``
|
||||
— never ``doc.chunks``. The **re-embed** path is the deliberate
|
||||
— never ``doc.chunks`` — and they are exactly the reads the six light
|
||||
columns cover. The **re-embed** path is the deliberate
|
||||
exception (LOCKED A2): :func:`_replace_content_chunks` reassigns
|
||||
``doc.chunks = [...]``, which is collection REPLACEMENT (the ORM must
|
||||
know which old rows to orphan) and legitimately needs the loaded
|
||||
@@ -3923,7 +4024,25 @@ def _prune(
|
||||
if not source_names:
|
||||
return 0
|
||||
pruned = 0
|
||||
docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()
|
||||
# Phase 184 (task 03): the light-row walk — see the docstring's
|
||||
# light-row contract. SIX columns, ``content`` deliberately absent
|
||||
# (252 MB of markdown per full KB walk), deterministic order so the
|
||||
# batch prefix is reproducible.
|
||||
docs = session.scalars(
|
||||
select(Document)
|
||||
.options(
|
||||
load_only(
|
||||
Document.id,
|
||||
Document.source,
|
||||
Document.path,
|
||||
Document.is_image,
|
||||
Document.image_path,
|
||||
Document.page_count,
|
||||
)
|
||||
)
|
||||
.where(Document.source.in_(source_names))
|
||||
.order_by(Document.id)
|
||||
).all()
|
||||
for doc in docs:
|
||||
if (doc.source, doc.path) not in seen:
|
||||
if doc.is_image and not images:
|
||||
@@ -3948,6 +4067,11 @@ def _prune(
|
||||
session.delete(doc)
|
||||
pruned += 1
|
||||
logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
|
||||
if pruned:
|
||||
# Phase 184 (task 03): a long prune no longer holds a huge
|
||||
# transaction — commit the batch (``prune_batch=0`` → never
|
||||
# here, the single commit below, i.e. the pre-phase run).
|
||||
if prune_batch and pruned and pruned % prune_batch == 0:
|
||||
session.commit()
|
||||
if pruned and not (prune_batch and pruned % prune_batch == 0):
|
||||
session.commit()
|
||||
return pruned
|
||||
|
||||
@@ -0,0 +1,919 @@
|
||||
"""Phase 184 E2E (Playwright, mock-only): the BOUNDED sync still syncs all.
|
||||
|
||||
Source: owner incident 2026-10-15 — the same run whose prune phase 183 fixed.
|
||||
An import run holds ONE session and ``app/db.py`` builds it with
|
||||
``expire_on_commit=False``, so a commit released NOTHING: the walk kept every
|
||||
``Document`` it touched resident (each unchanged row holds its full ~15 KB
|
||||
``content``, and ``_unchanged_doc``'s ``file_hash`` stamp puts the row in the
|
||||
session's own strong ``dirty`` set, out of reach of the cycle collector), and
|
||||
``_prune`` read every document of every pruned source WHOLE — ``content``
|
||||
included — holding all of its deletes in ONE transaction until the end. The
|
||||
run's memory therefore grew with the SIZE OF THE KNOWLEDGE BASE instead of
|
||||
with the work the run did.
|
||||
|
||||
The fix is two operator knobs, both ``BOR_``-prefixed settings the importer
|
||||
reads once per RUN:
|
||||
|
||||
* ``BOR_SYNC_SESSION_RESET_FILES`` (default ``200``) — every Nth file of the
|
||||
walk, at the one point where no object from a finished file is reachable,
|
||||
the session calls ``expunge_all()``; it also runs once per source, before
|
||||
the two post-passes. ``0`` = NEVER release = the exact pre-phase-184 run.
|
||||
* ``BOR_SYNC_PRUNE_BATCH`` (default ``500``) — the prune commits every Nth
|
||||
delete instead of once at the end of the whole import. ``0`` = the exact
|
||||
pre-phase-184 single commit. A non-zero batch deliberately trades the
|
||||
all-or-nothing prune for a PREFIX (LOCKED A2: ``seen`` is recomputed from
|
||||
disk, so the next sync converges).
|
||||
|
||||
Both knobs change the run's MEMORY PROFILE and nothing else — and that
|
||||
"nothing else" is precisely what a browser can see. This module pins it: the
|
||||
same counts line, every document exactly once, an all-``unchanged`` re-sync,
|
||||
a multi-batch prune that behaves, and a Sources page that never stalls, never
|
||||
errors and reports the same totals as the database. The retention itself (the
|
||||
identity-map bound, the prune's six-column ``SELECT``, the commit counts) is
|
||||
deterministic and pinned in
|
||||
``tests/integration/test_import_session_retention.py``; RSS is asserted
|
||||
nowhere in this phase (phase 183 LOCKED A4, inherited).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is skipped — every number here (the
|
||||
per-path chunk counts, the byte-identical cross-check) is a property of the
|
||||
deterministic mock embedder/summarizer; against the real endpoint the
|
||||
comparison would be a measurement, not a pin.
|
||||
|
||||
TWO module apps on dedicated ports (the ``test_sync_summary_visibility.py``
|
||||
pattern — the conftest session app owns 8123 in a combined run, and a
|
||||
same-port second uvicorn would drive the wrong server):
|
||||
|
||||
* the CADENCE app — ``BOR_SYNC_SESSION_RESET_FILES=3`` and
|
||||
``BOR_SYNC_PRUNE_BATCH=2``, so a 10-file walk releases the session THREE
|
||||
times mid-run (before files 4, 7 and 10) plus once at the source boundary,
|
||||
and a 5-document prune commits THREE times (at 2, at 4, and the final one).
|
||||
Tests 1–3 drive it;
|
||||
* the ZERO app — both knobs ``0``, i.e. the exact pre-phase-184 run. Test 4
|
||||
drives it over the SAME fixture directory, which is what makes the two runs
|
||||
comparable at all.
|
||||
|
||||
Both boot against the mock LLM with ``BOR_GIT_SOURCES`` forced empty (the
|
||||
owner's local gitignored ``.env`` must never leak a real source into a run),
|
||||
their own ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` under
|
||||
``tmp_path_factory`` (nothing here clones or renders a PDF — the dirs exist so
|
||||
no run writes into the operator's), and the mock-calibrated gate (the conftest
|
||||
convention). The fixture KB is ONE plain (non-git) directory named
|
||||
:data:`SOURCE_NAME` (``kind='local'`` rows index under ``Path(path).name``, so
|
||||
the source label is stable whatever temp root pytest picked) holding TEN
|
||||
``.md`` files across TWO folders — ``a/1..5.md`` and ``b/1..5.md`` — with
|
||||
``a/1.md`` carrying the sentinel phrase.
|
||||
|
||||
Isolation: the shared E2E Postgres is truncated BEFORE **and** AFTER every
|
||||
test (``chunks, documents, query_log, kb_overview, git_sources, sources_meta,
|
||||
folder_summaries``), after waiting for the app to hold no running sync (the
|
||||
run state lives in the app's memory, and a leaked run would keep importing
|
||||
into the next test's truncated KB). Each test registers its own source row.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_reset_cadence_produces_the_identical_run`` — the cadence ON is a
|
||||
complete, healthy sync: the counts line reads ``10 added``, the terminal
|
||||
reports ``added=10`` with ``errors=0``, every one of the ten paths exists
|
||||
EXACTLY ONCE in the KB (the test's own SQL), the ls/tree rows of both
|
||||
folders render all ten files, the stat cards equal ``select(count())`` over
|
||||
``documents`` (10) and ``chunks`` (> 10), the sentinel's phrase is in the
|
||||
stored content, and THROUGH THE RUN no ``/api/`` request answered 4xx/5xx
|
||||
and none failed — no error banner, no modal.
|
||||
2. ``test_resync_is_unchanged_with_the_cadence_on`` — the second sync of the
|
||||
same bytes settles ``0 added · 10 unchanged`` with ``pruned=0`` and
|
||||
``success``, and the chunk total and per-path chunk counts are byte-still:
|
||||
the cadence loses no rows (a lost row would come back as ``added``) and
|
||||
re-embeds nothing.
|
||||
3. ``test_multi_batch_prune_from_the_ui`` — five of the ten files go and the
|
||||
re-sync settles ``0 added · 5 unchanged · 5 pruned`` (5 deletes at batch
|
||||
``2`` = THREE commits inside the UI's own run); the five documents are gone
|
||||
from the KB AND from the ls/tree rows of BOTH folders, the survivors' chunk
|
||||
total is unchanged, and ZERO orphan chunk rows exist (the test's own SQL).
|
||||
4. ``test_cadence_zero_matches_cadence_three`` — the byte-identical
|
||||
cross-check the phase lives on: the SAME fixture walked with
|
||||
``BOR_SYNC_SESSION_RESET_FILES=0`` lands the same paths (each exactly
|
||||
once), the same per-path chunk counts, the same ``content`` / ``summary`` /
|
||||
``content_hash`` / ``file_hash`` on every document, the same chunk contents
|
||||
in position order and the same counts line as the cadence-``3`` run. The
|
||||
reference is test 1's recorded run when the module runs in order; alone,
|
||||
the test walks the cadence-``3`` app itself first — and either way it
|
||||
asserts its own 10-documents / >10-chunks shape BEFORE comparing, so an
|
||||
empty baseline can never make it pass vacuously.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import func, select, text
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
MOCK_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The CADENCE app (tests 1–3). Env-overridable, the house port pattern: the
|
||||
#: conftest session app owns 8123 in a combined run.
|
||||
CADENCE_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET", "8200"))
|
||||
|
||||
#: The ZERO app (test 4's reference leg): both knobs ``0`` — the exact
|
||||
#: pre-phase-184 run (LOCKED A3's no-redeploy rollback).
|
||||
ZERO_PORT = int(os.environ.get("E2E_APP_PORT_SYNCRESET_ZERO", "8201"))
|
||||
|
||||
SOURCES_URL = "/sources.html"
|
||||
|
||||
#: The fixture directory's basename — the ``source`` the importer records for
|
||||
#: a ``kind='local'`` row (``Path(path).name``), so it is the same string in
|
||||
#: the DB whatever temp root this run got.
|
||||
SOURCE_NAME = "sync-session-reset"
|
||||
|
||||
#: The fixture layout: two folders × five importable files. After the prune,
|
||||
#: ``a`` keeps three survivors and ``b`` two — so BOTH folders still render
|
||||
#: rows, and a prune that stopped after its first batch would show up.
|
||||
ALL_PATHS = [*[f"a/{n}.md" for n in range(1, 6)], *[f"b/{n}.md" for n in range(1, 6)]]
|
||||
KEEP_PATHS = ["a/1.md", "a/2.md", "a/3.md", "b/1.md", "b/2.md"]
|
||||
GONE_PATHS = ["a/4.md", "a/5.md", "b/3.md", "b/4.md", "b/5.md"]
|
||||
N_FILES = len(ALL_PATHS)
|
||||
|
||||
#: The sentinel phrase, carried by :data:`SENTINEL_PATH` ONLY. Finding it in
|
||||
#: the stored ``content`` proves the walk really indexed the fixture rather
|
||||
#: than leaving a KB of merely the right SIZE.
|
||||
SENTINEL = "RESE-SYNC-RESET-SENTINEL-4f27"
|
||||
SENTINEL_PATH = "a/1.md"
|
||||
|
||||
#: The cadences the two apps boot with — module constants so the narrative and
|
||||
#: the app env cannot drift apart. 10 files / every 3rd = releases before
|
||||
#: files 4, 7 and 10 (THREE mid-run resets, then the source boundary); a
|
||||
#: 5-document prune at batch 2 = commits at 2, at 4 and the final one.
|
||||
RESET_FILES = 3
|
||||
PRUNE_BATCH = 2
|
||||
|
||||
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
||||
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
|
||||
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
|
||||
|
||||
#: Settle budgets: a ten-file mock sync (walk + embeds + overview + three
|
||||
#: folder summaries) lands in seconds and the UI settles a terminal within one
|
||||
#: 2 s poll tick, so this is an order of magnitude of head-room with no
|
||||
#: client-side hard timeout (the test_prune_cascade_cleanup budget).
|
||||
SETTLE_TIMEOUT_MS = 90_000
|
||||
|
||||
#: Test 1 records its numbers here for test 4's cross-check (the
|
||||
#: ``test_sync_changed_only.py`` "state from the earlier test" idiom). Test 4
|
||||
#: runs its own cadence leg when this is empty, so it stays honest alone.
|
||||
_CADENCE_RUN: dict[str, Any] = {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The fixture source (ONE directory BOTH apps walk)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _doc_body(path: str) -> str:
|
||||
"""One small markdown file per fixture path, byte-stable across runs.
|
||||
|
||||
The bodies differ from each other (so a per-path chunk count means
|
||||
something) but never carry a timestamp or the absolute path, and only
|
||||
:data:`SENTINEL_PATH` carries the sentinel — that is what lets test 4
|
||||
compare two runs of two different processes line for line.
|
||||
"""
|
||||
if path == SENTINEL_PATH:
|
||||
return (
|
||||
"# Session reset sentinel\n"
|
||||
"\n"
|
||||
"The bounded sync keeps the sentinel phrase in the knowledge base.\n"
|
||||
f"Marker: {SENTINEL}\n"
|
||||
)
|
||||
folder, name = path.rsplit("/", 1)
|
||||
stem = name.removesuffix(".md")
|
||||
return (
|
||||
f"# Session reset note {folder} {stem}\n"
|
||||
"\n"
|
||||
f"One unrelated note from the {folder} folder of the {SOURCE_NAME} "
|
||||
"fixture source, indexed so the cadence has something to walk.\n"
|
||||
)
|
||||
|
||||
|
||||
def _seed_source(root: Path) -> Path:
|
||||
"""(Re-)write all ten fixture files under ``root / SOURCE_NAME``.
|
||||
|
||||
Idempotent on purpose: test 3 deletes five files from the tree and test 4
|
||||
re-seeds the SAME directory, so the reference leg walks byte-identical
|
||||
content through the other app.
|
||||
"""
|
||||
base = root / SOURCE_NAME
|
||||
for rel in ALL_PATHS:
|
||||
target = base / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(_doc_body(rel), encoding="utf-8")
|
||||
assert not (base / ".git").exists() # a plain directory — no git anywhere
|
||||
return base
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one fixture tree BOTH module apps walk (the app servers run on the
|
||||
same host, so the path is visible to them)."""
|
||||
return _seed_source(tmp_path_factory.mktemp("bor_sync_session_reset"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The two module apps
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(scratch: Path, reset_files: int, prune_batch: int) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern): the mock LLM, the
|
||||
mock-calibrated gate, no git fallback, dedicated scratch dirs — and THIS
|
||||
app's two phase-184 cadences, pinned explicitly so an operator's local
|
||||
(gitignored) ``.env`` cannot retune them."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{MOCK_PORT}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (the conftest pattern): the token-overlap mock
|
||||
# embeddings score far below the production 0.62, and BOTH floors must
|
||||
# stay <= the threshold or the app refuses to boot (app/config.py).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# No env-fallback rows: the registry each test writes through the
|
||||
# authenticated API is the only source list, so every sync's counts are
|
||||
# exactly this test's doing.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
|
||||
# THE SUBJECT OF THE PHASE.
|
||||
env["BOR_SYNC_SESSION_RESET_FILES"] = str(reset_files)
|
||||
env["BOR_SYNC_PRUNE_BATCH"] = str(prune_batch)
|
||||
return env
|
||||
|
||||
|
||||
def _serve(env: dict[str, str], port: int) -> Iterator[str]:
|
||||
"""Boot one uvicorn app under test on *port* and yield its URL."""
|
||||
proc = subprocess.Popen(
|
||||
[
|
||||
sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning",
|
||||
],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
try:
|
||||
_wait_http(f"{url}/api/health")
|
||||
yield url
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The CADENCE app (tests 1–3): ``reset=3``, ``prune_batch=2``.
|
||||
|
||||
``mock_llm`` and ``source_dir`` are dependencies for the ORDERING only —
|
||||
the mock answers before the app boots, and the fixture tree exists before
|
||||
the first test registers it.
|
||||
"""
|
||||
if USE_REAL_LLM:
|
||||
pytest.skip(
|
||||
"mock-only suite — the byte-identical cross-check is a property "
|
||||
"of the deterministic mock (E2E_REAL_LLM unsupported)"
|
||||
)
|
||||
assert mock_llm and source_dir
|
||||
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_cadence")
|
||||
yield from _serve(_app_env(scratch, RESET_FILES, PRUNE_BATCH), CADENCE_PORT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zero_app_server(
|
||||
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The ZERO app (test 4's reference leg): both knobs ``0``, the exact
|
||||
pre-phase-184 run the bounded one must be byte-identical to."""
|
||||
if USE_REAL_LLM:
|
||||
pytest.skip(
|
||||
"mock-only suite — the byte-identical cross-check is a property "
|
||||
"of the deterministic mock (E2E_REAL_LLM unsupported)"
|
||||
)
|
||||
assert mock_llm and source_dir
|
||||
scratch = tmp_path_factory.mktemp("bor_sync_session_reset_zero")
|
||||
yield from _serve(_app_env(scratch, 0, 0), ZERO_PORT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zero_app_url(zero_app_server: str) -> str:
|
||||
return zero_app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Isolation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern). ``sources_meta`` (the
|
||||
KB generation counter) and ``folder_summaries`` reset with the KB: the
|
||||
sync's overview/bump is an upsert, and a leftover folder summary would
|
||||
ride the very ``changed_paths`` feed the prune writes."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No run may leak across a truncate: wait for a non-running status with
|
||||
an own admin session (the endpoint is admin-only). Usually a no-op — the
|
||||
tests settle before they end."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation: the shared E2E Postgres is truncated before AND
|
||||
after every test, the app holding no running sync at both boundaries.
|
||||
|
||||
The ZERO app is deliberately NOT waited on: it is booting lazily for test 4
|
||||
only, and that test's own run is settled inside the test body, so the
|
||||
post-truncate can never race it."""
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
yield
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers: the UI side
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _watch_network(page: Page) -> list[str]:
|
||||
"""Collect every ``/api/`` 4xx/5xx and every failed ``/api/`` request the
|
||||
page sees — the owner's "the Sources page errored / stalled".
|
||||
|
||||
Returns the (growing) list the caller asserts empty at the end of the
|
||||
run: the page's own 2 s status poll lands in this list, so a run that made
|
||||
the API answer 500 is caught here rather than at some later assertion.
|
||||
"""
|
||||
problems: list[str] = []
|
||||
|
||||
def _on_response(response: Any) -> None:
|
||||
if "/api/" in response.url and response.status >= 400:
|
||||
problems.append(f"HTTP {response.status} {response.url}")
|
||||
|
||||
def _on_failed(request: Any) -> None:
|
||||
if "/api/" in request.url:
|
||||
problems.append(f"FAILED {request.url} ({request.failure})")
|
||||
|
||||
page.on("response", _on_response)
|
||||
page.on("requestfailed", _on_failed)
|
||||
return problems
|
||||
|
||||
|
||||
def _add_local_source(page: Page, base_url: str, path: Path) -> None:
|
||||
"""Register the fixture directory through the authenticated API (the
|
||||
phase-49 shape — the page's own form for this is gone; the admin cookie
|
||||
rides ``page.request``)."""
|
||||
r = page.request.post(
|
||||
f"{base_url}/api/git-sources",
|
||||
data={"kind": "local", "path": str(path)},
|
||||
)
|
||||
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the server
|
||||
stamps it in UTC and this process runs on the same host)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]:
|
||||
"""Click the Sources page **Sync sources** button and wait for THIS run's
|
||||
terminal state (the module app is shared, so a terminal counts only after
|
||||
a ``running`` tick or when its ``started_at`` post-dates the click).
|
||||
|
||||
Settles the UI on success and proves the owner saw no error surface: the
|
||||
button re-enabled, ``Synced HH:MM``, no error banner, no modal."""
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(btn).to_be_enabled()
|
||||
clicked_at = time.time()
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
|
||||
deadline = time.monotonic() + SETTLE_TIMEOUT_MS / 1000
|
||||
saw_running = False
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = page.request.get(f"{base_url}/api/sync/status")
|
||||
assert r.status == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed", "cancelled") and (
|
||||
saw_running or _started_at(body) >= clicked_at - 1.0
|
||||
):
|
||||
break
|
||||
time.sleep(0.2)
|
||||
else:
|
||||
raise AssertionError(f"sync never reached a terminal state: {body}")
|
||||
|
||||
assert body["state"] == "success", f"sync failed: {body}"
|
||||
assert body["error"] is None, body
|
||||
expect(btn).to_be_enabled(timeout=15_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
|
||||
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
||||
assert page.locator(".sync-modal-backdrop").count() == 0
|
||||
return body
|
||||
|
||||
|
||||
def _expect_counts_line(page: Page, expected: str) -> None:
|
||||
"""The counts line — the owner-visible sync report on the single sync text
|
||||
surface (phase 150; sources.js fmtSyncResult)."""
|
||||
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
|
||||
|
||||
|
||||
def _counts_line(page: Page) -> str:
|
||||
"""The counts line as text — the string test 4 compares across the apps."""
|
||||
return str(page.locator("#sync-progress-text").inner_text())
|
||||
|
||||
|
||||
def _open_level(page: Page, *names: str) -> None:
|
||||
"""Open ONE level at a time from the catalogue top — each name the EXACT
|
||||
link text of the source or folder row at the current level (the phase-97
|
||||
drill idiom, the ``test_kb_tree_lazy_load.py`` settle).
|
||||
|
||||
The top is reached through the breadcrumb's first segment when the page is
|
||||
drilled (the phase-99 drill state rides ``history.state``, so a nav
|
||||
re-click would KEEP the level — the crumb is the control that actually
|
||||
leaves it), and is a no-op when the page already sits there (the crumb
|
||||
hidden). Every click after that awaits the level it produced: the phase-167
|
||||
node fetch is over once the breadcrumb names the level and no in-flight
|
||||
``.kb-level-loading`` row remains."""
|
||||
if not page.locator("#kb-crumb").is_hidden():
|
||||
page.click('#kb-crumb a.kb-crumb-link:text-is("Knowledge base")')
|
||||
expect(page.locator("#kb-crumb")).to_be_hidden(timeout=30_000)
|
||||
expect(page.locator("#folders-tbody tr")).to_have_count(1, timeout=30_000)
|
||||
for name in names:
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(
|
||||
name, timeout=30_000
|
||||
)
|
||||
expect(page.locator(".kb-level-loading")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers: the DB side
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _paths_with_counts() -> dict[str, int]:
|
||||
"""``path`` → number of ``documents`` rows for that path — THE exactly-once
|
||||
probe: a reset that dropped work and re-added it shows up here."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(Document.path, func.count(Document.id))
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.group_by(Document.path)
|
||||
).all()
|
||||
return {str(path): int(count) for path, count in rows}
|
||||
|
||||
|
||||
def _chunk_count_by_path(paths: list[str] | None = None) -> dict[str, int]:
|
||||
"""``path`` → chunk rows per document (an absent path is not listed)."""
|
||||
query = (
|
||||
select(Document.path, func.count(Chunk.id))
|
||||
.outerjoin(Chunk, Chunk.document_id == Document.id)
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.group_by(Document.path)
|
||||
)
|
||||
if paths is not None:
|
||||
query = query.where(Document.path.in_(paths))
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(query).all()
|
||||
return {str(path): int(count) for path, count in rows}
|
||||
|
||||
|
||||
def _total_chunks() -> int:
|
||||
with SessionLocal() as db:
|
||||
return int(db.scalar(select(func.count()).select_from(Chunk)) or 0)
|
||||
|
||||
|
||||
def _doc_ids(paths: list[str]) -> list[str]:
|
||||
"""The indexed document ids of ``(SOURCE_NAME, path)``, sorted. Test 3
|
||||
needs them BEFORE the prune — the orphan query runs on ids that no longer
|
||||
exist afterwards."""
|
||||
with SessionLocal() as db:
|
||||
return sorted(
|
||||
str(doc_id)
|
||||
for (doc_id,) in db.execute(
|
||||
select(Document.id).where(
|
||||
Document.source == SOURCE_NAME, Document.path.in_(paths)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
|
||||
|
||||
def _orphan_chunks() -> int:
|
||||
"""Chunk rows whose document is gone — phase 183's contract, asserted here
|
||||
too so a BATCHED prune cannot leave any behind."""
|
||||
with SessionLocal() as db:
|
||||
return int(
|
||||
db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.where(Chunk.document_id.not_in(select(Document.id)))
|
||||
)
|
||||
or 0
|
||||
)
|
||||
|
||||
|
||||
def _kb_snapshot() -> dict[str, tuple[str, str | None, str, str, list[str]]]:
|
||||
"""The byte-level shape of the fixture source:
|
||||
|
||||
``path`` → ``(content, summary, content_hash, file_hash, chunk contents in
|
||||
position order)`` — every field the two runs must agree on. Timestamps and
|
||||
``full_path`` are deliberately absent: they belong to the run and to the
|
||||
temp root, not to the imported bytes.
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
docs = db.scalars(
|
||||
select(Document)
|
||||
.where(Document.source == SOURCE_NAME)
|
||||
.order_by(Document.path)
|
||||
).all()
|
||||
snapshot: dict[str, tuple[str, str | None, str, str, list[str]]] = {}
|
||||
for doc in docs:
|
||||
assert doc.content_hash and doc.file_hash, doc.path
|
||||
contents = db.scalars(
|
||||
select(Chunk.content)
|
||||
.where(Chunk.document_id == doc.id)
|
||||
.order_by(Chunk.position)
|
||||
).all()
|
||||
snapshot[str(doc.path)] = (
|
||||
doc.content,
|
||||
doc.summary,
|
||||
str(doc.content_hash),
|
||||
str(doc.file_hash),
|
||||
[str(c) for c in contents],
|
||||
)
|
||||
return snapshot
|
||||
|
||||
|
||||
def _sentinel_in_kb() -> bool:
|
||||
"""Does the stored content of the sentinel document carry the sentinel?
|
||||
(The walk really indexed the fixture; the KB is not merely the right size.)"""
|
||||
with SessionLocal() as db:
|
||||
content = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE_NAME, Document.path == SENTINEL_PATH
|
||||
)
|
||||
)
|
||||
return bool(content and SENTINEL in content)
|
||||
|
||||
|
||||
def _file_rows(page: Page) -> list[str]:
|
||||
"""The current level's file rows, by path — the ls/tree rows the owner
|
||||
reads (a pruned document must leave this list)."""
|
||||
links = page.locator("#docs-tbody tr a.doc-link")
|
||||
return [str(links.nth(i).inner_text()) for i in range(links.count())]
|
||||
|
||||
|
||||
def _run_fresh_sync(page: Page, base_url: str, source: Path) -> dict[str, Any]:
|
||||
"""Register the fixture and run ONE fresh-KB sync through the Sources
|
||||
page, returning the shape the cross-check compares."""
|
||||
_add_local_source(page, base_url, source)
|
||||
body = _sync_from_sources_page(page, base_url)
|
||||
return {
|
||||
"detail": dict(body["detail"]),
|
||||
"line": _counts_line(page),
|
||||
"paths": _paths_with_counts(),
|
||||
"chunks_by_path": _chunk_count_by_path(),
|
||||
"total_chunks": _total_chunks(),
|
||||
"snapshot": _kb_snapshot(),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The cadence ON syncs everything, cleanly, and the page agrees
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_reset_cadence_produces_the_identical_run(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""``BOR_SYNC_SESSION_RESET_FILES=3`` over ten files: three mid-run
|
||||
releases — and a sync whose every owner-visible number is the number the
|
||||
unbounded run produces.
|
||||
|
||||
The counts line reads ``10 added``, the terminal reports ``added=10`` with
|
||||
``errors=0``, each of the ten paths exists EXACTLY ONCE (a release that
|
||||
discarded unflushed work would show up as a missing or a duplicated path),
|
||||
the ls/tree rows of both folders render all ten files, the stat cards
|
||||
equal the database's own ``select(count())`` over ``documents`` and
|
||||
``chunks``, the sentinel's phrase is in the stored content — and not one
|
||||
``/api/`` request answered 4xx/5xx or failed through the whole run."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent) # idempotent — the tree is byte-stable
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
leg = _run_fresh_sync(page, app_url, source_dir)
|
||||
|
||||
# --- the owner's settle line + the terminal detail -------------------
|
||||
assert leg["line"] == f"{N_FILES} added", leg["line"]
|
||||
detail = leg["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
|
||||
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
|
||||
assert detail["files"] == N_FILES, detail
|
||||
|
||||
# --- every document landed EXACTLY once, fully embedded --------------
|
||||
assert leg["paths"] == dict.fromkeys(ALL_PATHS, 1), leg["paths"]
|
||||
assert all(count > 0 for count in leg["chunks_by_path"].values()), (
|
||||
f"a document has no chunks — the release lost its embed work: "
|
||||
f"{leg['chunks_by_path']}"
|
||||
)
|
||||
assert leg["total_chunks"] > N_FILES, leg["total_chunks"]
|
||||
assert _sentinel_in_kb(), "the sentinel document never reached the KB"
|
||||
|
||||
# --- the page agrees with the database (the phase-167 totals) --------
|
||||
expect(page.locator("#stat-docs")).to_have_text(str(N_FILES), timeout=15_000)
|
||||
expect(page.locator("#stat-chunks")).to_have_text(
|
||||
str(leg["total_chunks"]), timeout=15_000
|
||||
)
|
||||
|
||||
# --- the ls/tree rows render the whole fixture -----------------------
|
||||
_open_level(page, SOURCE_NAME, "a")
|
||||
assert _file_rows(page) == [f"a/{n}.md" for n in range(1, 6)]
|
||||
_open_level(page, SOURCE_NAME, "b")
|
||||
assert _file_rows(page) == [f"b/{n}.md" for n in range(1, 6)]
|
||||
|
||||
# --- the run never errored on the wire or in the UI ------------------
|
||||
assert problems == [], f"the run put an error on the wire: {problems}"
|
||||
|
||||
# Record for test 4's cross-check (the phase's byte-identical contract).
|
||||
_CADENCE_RUN.clear()
|
||||
_CADENCE_RUN.update(leg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The unchanged re-sync with the cadence ON: nothing moves
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_resync_is_unchanged_with_the_cadence_on(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""The run a production sync actually takes — every file unchanged — with
|
||||
the release firing three times through it: the phase-147 fast path stays
|
||||
the fast path.
|
||||
|
||||
The second sync settles ``0 added · 10 unchanged`` with ``pruned=0``, the
|
||||
terminal is ``success``, and the database's chunk total and per-path chunk
|
||||
counts are byte-still: the cadence neither loses rows (which the next sync
|
||||
would report as ``added``) nor silently re-embeds an unchanged document."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent)
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
first = _run_fresh_sync(page, app_url, source_dir)
|
||||
assert first["detail"]["added"] == N_FILES, first["detail"]
|
||||
_expect_counts_line(page, f"{N_FILES} added")
|
||||
chunks_after_first = _chunk_count_by_path()
|
||||
paths_after_first = _paths_with_counts()
|
||||
total_after_first = _total_chunks()
|
||||
assert total_after_first == sum(chunks_after_first.values())
|
||||
|
||||
# --- the unchanged re-sync -------------------------------------------
|
||||
second = _sync_from_sources_page(page, app_url)
|
||||
detail = second["detail"]
|
||||
assert detail["unchanged"] == N_FILES, detail
|
||||
assert detail["added"] == 0 and detail["updated"] == 0, detail
|
||||
assert detail["pruned"] == 0 and detail["errors"] == 0, detail
|
||||
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
|
||||
|
||||
# No silent re-embed: the same chunk total, per path, after the run.
|
||||
assert _chunk_count_by_path() == chunks_after_first, (
|
||||
"an unchanged file changed chunk count — the cadence re-embedded a "
|
||||
"document the fast path had left alone"
|
||||
)
|
||||
assert _total_chunks() == total_after_first
|
||||
assert _paths_with_counts() == paths_after_first
|
||||
assert _sentinel_in_kb()
|
||||
assert problems == [], f"the re-sync put an error on the wire: {problems}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The prune through the UI at BOR_SYNC_PRUNE_BATCH=2
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_multi_batch_prune_from_the_ui(
|
||||
page: Page, app_url: str, source_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Five documents go, and the prune that takes them is a MULTI-BATCH run:
|
||||
at ``BOR_SYNC_PRUNE_BATCH=2`` the five deletes commit at 2, at 4 and once
|
||||
more at the end — THREE commits inside the UI's own run — while the owner
|
||||
still sees exactly one prune report.
|
||||
|
||||
The counts line settles ``0 added · 5 unchanged · 5 pruned``; the five
|
||||
documents are gone from the KB AND from the ls/tree rows of BOTH folders
|
||||
(three survivors in ``a``, two in ``b``); the survivors' chunk total is
|
||||
unchanged; and ZERO orphan chunk rows exist. A batched commit changes WHEN
|
||||
a delete becomes durable, never what the run removes."""
|
||||
page.set_default_timeout(30_000)
|
||||
_seed_source(source_dir.parent)
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
|
||||
first = _run_fresh_sync(page, app_url, source_dir)
|
||||
assert first["detail"]["added"] == N_FILES, first["detail"]
|
||||
|
||||
# The subject of the prune, captured BEFORE it happens: the doomed rows'
|
||||
# ids (the orphan query needs ids that no longer exist by the time it
|
||||
# runs) and every survivor's chunk count.
|
||||
doomed_ids = _doc_ids(GONE_PATHS)
|
||||
keep_counts = _chunk_count_by_path(KEEP_PATHS)
|
||||
assert len(doomed_ids) == len(GONE_PATHS), doomed_ids
|
||||
assert set(keep_counts) == set(KEEP_PATHS), keep_counts
|
||||
assert all(count > 0 for count in keep_counts.values()), keep_counts
|
||||
|
||||
# --- the upstream deletion (this test owns the fixture tree) ---------
|
||||
for rel in GONE_PATHS:
|
||||
(source_dir / rel).unlink()
|
||||
assert not any((source_dir / rel).exists() for rel in GONE_PATHS)
|
||||
|
||||
# --- the prune --------------------------------------------------------
|
||||
body = _sync_from_sources_page(page, app_url)
|
||||
detail = body["detail"]
|
||||
assert detail["pruned"] == len(GONE_PATHS), detail
|
||||
assert detail["added"] == 0 and detail["updated"] == 0, detail
|
||||
assert detail["unchanged"] == len(KEEP_PATHS), detail
|
||||
assert detail["errors"] == 0, detail
|
||||
_expect_counts_line(
|
||||
page,
|
||||
f"0 added · {len(KEEP_PATHS)} unchanged · {len(GONE_PATHS)} pruned",
|
||||
)
|
||||
|
||||
# The catalogue rows went WITH the documents — in both folders, which is
|
||||
# where a prune that stopped after one batch would leave a stale row.
|
||||
_open_level(page, SOURCE_NAME, "a")
|
||||
assert _file_rows(page) == ["a/1.md", "a/2.md", "a/3.md"]
|
||||
_open_level(page, SOURCE_NAME, "b")
|
||||
assert _file_rows(page) == ["b/1.md", "b/2.md"]
|
||||
|
||||
# The survivors are untouched and nothing was orphaned.
|
||||
assert _chunk_count_by_path() == keep_counts, (
|
||||
"the prune took a survivor's chunks with it, or left a doomed row"
|
||||
)
|
||||
assert _total_chunks() == sum(keep_counts.values())
|
||||
assert _orphan_chunks() == 0, (
|
||||
"a batched prune left orphan chunk rows — the deletes committed "
|
||||
"without the FK cascade doing the child work"
|
||||
)
|
||||
assert _sentinel_in_kb()
|
||||
assert problems == [], f"the prune put an error on the wire: {problems}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. cadence 0 == cadence 3 (the byte-identical cross-check)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cadence_zero_matches_cadence_three(
|
||||
page: Page,
|
||||
app_url: str,
|
||||
zero_app_url: str,
|
||||
source_dir: Path,
|
||||
db_ready: None,
|
||||
) -> None:
|
||||
"""THE PIN the phase lives on: the cadences change the run's memory
|
||||
profile and NOTHING the owner or the database can see.
|
||||
|
||||
The same fixture tree, walked fresh by the ZERO app
|
||||
(``BOR_SYNC_SESSION_RESET_FILES=0``, ``BOR_SYNC_PRUNE_BATCH=0`` — the
|
||||
pre-phase-184 run) against the CADENCE app's numbers (``3`` / ``2``): the
|
||||
same paths (each exactly once), the same chunk count per path, the same
|
||||
``content`` / ``summary`` / ``content_hash`` / ``file_hash`` on every
|
||||
document, the same chunk contents in position order, and the same counts
|
||||
line.
|
||||
|
||||
The reference is test 1's recorded run when the module runs in order; run
|
||||
ALONE, the test walks the cadence-``3`` app itself first. Either way it
|
||||
asserts the reference's own 10-documents / >10-chunks shape BEFORE
|
||||
comparing, so an empty baseline can never make it pass vacuously."""
|
||||
page.set_default_timeout(30_000)
|
||||
|
||||
# Re-seed: test 3 deleted five files from the fixture tree.
|
||||
_seed_source(source_dir.parent)
|
||||
on_disk = sorted(p.relative_to(source_dir).as_posix() for p in source_dir.rglob("*.md"))
|
||||
assert on_disk == sorted(ALL_PATHS), on_disk
|
||||
|
||||
# --- the reference leg: cadence 3 (test 1's numbers, or a fresh run) --
|
||||
if _CADENCE_RUN:
|
||||
reference = dict(_CADENCE_RUN)
|
||||
else:
|
||||
login(page, app_url, next=SOURCES_URL)
|
||||
reference = _run_fresh_sync(page, app_url, source_dir)
|
||||
_truncate_all()
|
||||
assert reference["paths"] == dict.fromkeys(ALL_PATHS, 1), reference["paths"]
|
||||
assert reference["total_chunks"] > N_FILES, reference["total_chunks"]
|
||||
|
||||
# --- the subject leg: cadence 0, the exact pre-phase run --------------
|
||||
login(page, zero_app_url, next=SOURCES_URL)
|
||||
problems = _watch_network(page)
|
||||
zero = _run_fresh_sync(page, zero_app_url, source_dir)
|
||||
|
||||
assert zero["detail"]["added"] == N_FILES, zero["detail"]
|
||||
assert zero["detail"]["errors"] == 0, zero["detail"]
|
||||
assert zero["detail"]["pruned"] == 0, zero["detail"]
|
||||
|
||||
# --- every number agrees ---------------------------------------------
|
||||
assert zero["line"] == reference["line"], (
|
||||
f"the owner-visible counts line moved with the cadence: "
|
||||
f"{zero['line']!r} != {reference['line']!r}"
|
||||
)
|
||||
assert zero["paths"] == reference["paths"], (
|
||||
"the set of paths (or a path's row count) moved with the cadence"
|
||||
)
|
||||
assert zero["chunks_by_path"] == reference["chunks_by_path"], (
|
||||
"the per-path chunk counts moved with the cadence: "
|
||||
f"{zero['chunks_by_path']} != {reference['chunks_by_path']}"
|
||||
)
|
||||
assert zero["total_chunks"] == reference["total_chunks"]
|
||||
assert zero["snapshot"].keys() == reference["snapshot"].keys()
|
||||
for path, row in zero["snapshot"].items():
|
||||
assert row == reference["snapshot"][path], (
|
||||
f"{path} is not byte-identical across the cadences"
|
||||
)
|
||||
assert _orphan_chunks() == 0
|
||||
assert problems == [], f"the reference run put an error on the wire: {problems}"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1287,12 +1287,16 @@ def test_backfill_never_touches_standalone_and_copyless_rows(db, tmp_path) -> No
|
||||
assert s.image_context_backfilled == 0
|
||||
assert s.images_failed == 0 and s.errors == 0
|
||||
assert llm.text_parts == [] # zero describe calls, both runs
|
||||
db.expire(standalone)
|
||||
db.expire(missing_copy)
|
||||
assert standalone.image_context is None
|
||||
assert standalone.content == DESCRIPTION
|
||||
assert missing_copy.image_context is None
|
||||
assert missing_copy.content == DESCRIPTION
|
||||
# Phase 184: an import run is now ALLOWED to release the session it
|
||||
# walks (the identity-map cadence expunges every object at the source
|
||||
# boundary), so the two seeded rows are re-read by key instead of
|
||||
# expiring the objects the run detached. Same assertion, one level
|
||||
# closer to the database — nothing about the run's output changed.
|
||||
for row in (standalone, missing_copy):
|
||||
reloaded = db.get(Document, row.id)
|
||||
assert reloaded is not None
|
||||
assert reloaded.image_context is None
|
||||
assert reloaded.content == DESCRIPTION
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
@@ -906,3 +906,58 @@ def test_ui_customization_env_overrides(monkeypatch: pytest.MonkeyPatch) -> None
|
||||
assert s.footer_text == "Powered by my own models"
|
||||
monkeypatch.setenv("BOR_INPUT_PLACEHOLDER", "")
|
||||
assert _settings().input_placeholder == "" # empty stands
|
||||
|
||||
|
||||
# --- Sync session bounds (phase 184) ---
|
||||
|
||||
|
||||
def test_sync_session_reset_files_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 184 (LOCKED A3): the import walk releases its session every
|
||||
``BOR_SYNC_SESSION_RESET_FILES`` files (default 200). ``0`` is accepted
|
||||
and means NEVER release — the exact pre-phase-184 behaviour, the
|
||||
operator's no-redeploy rollback."""
|
||||
monkeypatch.delenv("BOR_SYNC_SESSION_RESET_FILES", raising=False)
|
||||
assert _settings().sync_session_reset_files == 200
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "50")
|
||||
assert _settings().sync_session_reset_files == 50
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "0")
|
||||
assert _settings().sync_session_reset_files == 0
|
||||
|
||||
|
||||
def test_sync_session_reset_files_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative cadence is a typo — a ``%`` against it never fires, so
|
||||
startup refuses naming the field (the ``agent_max_rounds`` pattern,
|
||||
phase 184)."""
|
||||
monkeypatch.setenv("BOR_SYNC_SESSION_RESET_FILES", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_session_reset_files"):
|
||||
_settings()
|
||||
|
||||
|
||||
def test_sync_prune_batch_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 184 (LOCKED A3): the prune commits every
|
||||
``BOR_SYNC_PRUNE_BATCH`` deleted documents (default 500). ``0`` is
|
||||
accepted and means ONE commit at the end of the whole prune — the
|
||||
exact pre-phase-184 behaviour."""
|
||||
monkeypatch.delenv("BOR_SYNC_PRUNE_BATCH", raising=False)
|
||||
assert _settings().sync_prune_batch == 500
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "25")
|
||||
assert _settings().sync_prune_batch == 25
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "0")
|
||||
assert _settings().sync_prune_batch == 0
|
||||
|
||||
|
||||
def test_sync_prune_batch_rejects_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A negative batch size is a typo — the batch comparison would never
|
||||
fire, so startup refuses naming the field (the ``agent_max_rounds``
|
||||
pattern, phase 184)."""
|
||||
monkeypatch.setenv("BOR_SYNC_PRUNE_BATCH", "-1")
|
||||
with pytest.raises(ValidationError, match="sync_prune_batch"):
|
||||
_settings()
|
||||
|
||||
@@ -188,6 +188,41 @@ def test_verbatim_display_field_types() -> None:
|
||||
assert Settings.model_fields["show_text_max_blocks"].default == 5
|
||||
|
||||
|
||||
def test_sync_session_bound_rows_are_plain_live_overridable() -> None:
|
||||
"""Phase 184 (task 01, LOCKED A3/D2): the two sync-run knobs sit in
|
||||
``Import & sources`` (beside ``upload_max_mb`` — they tune the sync, not
|
||||
the walk's content) with the ``Sync session reset files`` / ``Sync prune
|
||||
batch`` labels and are plain — not ``secret`` (two ints), not
|
||||
``restart_required`` (the importer reads ``llm.settings`` per run, so a
|
||||
saved override applies to the NEXT sync), overridable (the overlay is
|
||||
generic over ``int`` fields). Both accept ``0`` = the exact pre-phase
|
||||
behaviour (the no-redeploy rollback); a negative value fails startup,
|
||||
not the panel."""
|
||||
for key, label in (
|
||||
("sync_session_reset_files", "Sync session reset files"),
|
||||
("sync_prune_batch", "Sync prune batch"),
|
||||
):
|
||||
meta = CATALOG[key]
|
||||
assert meta.group == "Import & sources"
|
||||
assert meta.label == label
|
||||
assert meta.secret is False
|
||||
assert meta.restart_required is False
|
||||
assert meta.overridable is True
|
||||
assert key not in SECRETS
|
||||
assert key not in RESTART_REQUIRED
|
||||
assert key not in NON_OVERRIDABLE
|
||||
|
||||
|
||||
def test_sync_session_bound_field_types() -> None:
|
||||
"""Phase 184 (task 01, D2): both knobs are ``int``s with the LOCKED A3
|
||||
defaults (``200`` files / ``500`` documents) — the type the panel renders
|
||||
its input for and the overlay coerces from the stored raw string."""
|
||||
assert Settings.model_fields["sync_session_reset_files"].annotation is int
|
||||
assert Settings.model_fields["sync_prune_batch"].annotation is int
|
||||
assert Settings.model_fields["sync_session_reset_files"].default == 200
|
||||
assert Settings.model_fields["sync_prune_batch"].default == 500
|
||||
|
||||
|
||||
def test_labels_nonempty_and_unique() -> None:
|
||||
labels = [meta.label for meta in CATALOG.values()]
|
||||
assert all(label.strip() for label in labels), "empty label in the catalogue"
|
||||
|
||||
Reference in New Issue
Block a user