Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Details
Build and Push Containers / build-and-push-db (push) Successful in 11s
Details

Browse Source 
All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
ducoterra
2026-10-16 04:54:01 -04:00
parent 065304b52f
commit 632fb88e43
35 changed files with 2862 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/185_sync_defer_content_fast_path/00_phase.md → .agents/phases/complete/185_sync_defer_content_fast_path/00_phase.md
View File
.agents/phases/todo/185_sync_defer_content_fast_path/01_fast_path_statement_probe.md → .agents/phases/complete/185_sync_defer_content_fast_path/01_fast_path_statement_probe.md
View File
.agents/phases/todo/185_sync_defer_content_fast_path/02_defer_content.md → .agents/phases/complete/185_sync_defer_content_fast_path/02_defer_content.md
View File
.agents/phases/todo/185_sync_defer_content_fast_path/03_e2e_deferred_content.md → .agents/phases/complete/185_sync_defer_content_fast_path/03_e2e_deferred_content.md
View File
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__00_phase.a1.err
View File
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__00_phase.a1.md
+20
View File
@@ -0,0 +1,20 @@
All gates green. Report:
**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.
**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).
**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).
**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).
**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).
**Next pending phase**: `186_git_log_streaming_dates`.
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__00_phase.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
....................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 864 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6872 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__01_fast_path_statement_probe.a1.err
View File
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__01_fast_path_statement_probe.a1.md
+18
View File
@@ -0,0 +1,18 @@
**Task 01 complete — probe written, green, and it pins the defect.**
**Implemented**
- `tests/integration/test_import_fast_path_columns.py` (new, test-only; no `app/` change): fixture KB of 4 × ~8 KB `.md` files seeded through the real `import_sources`, then a SECOND unchanged `import_sources` captured via a `sqlalchemy.event` `before_cursor_execute` listener (phase-183 technique, removed in teardown) with `(statement, params)` so every claim is attributable per file/PK.
- Helpers `doc_selects()` / `loads_content()` (projection-only, `\bdocuments\.content\b` token so `content_hash` can never match — stated in the comment), plus `row_loads()` (the `(source, path)` fast-path shape), `pk_followups()` (the deferred-column lazy-load shape) and a matcher self-test.
- `test_unchanged_run_hydrates_content_pre_fix`: 4 row loads, **all naming `documents.content`**, 0 PK follow-ups, +2 per-source image post-passes pinned apart; carries `# task 02 flips this`.
- `test_unchanged_file_is_one_select`: exactly **1** documents SELECT per summary-bearing file, 0 follow-ups — explicitly marked MUST-STAY-GREEN post-fix (the anti-lazy-load-storm guard).
- `test_summary_null_backfill_reads_stored_content`: staged `summary=NULL` row (stale `is_summary` chunk stamped) → backfilled from the stored markdown (the `lite` call receives `documents.content` verbatim), chunk replaced; `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 0` → 1 with the arithmetic in the comment.
- D4 reference test: unchanged-run counts (`unchanged=4`, `summary_backfilled=1`, `summaries=0`, `chunks=0`, `changed_paths=[]`) + verbatim log words (`import: unchanged` ×4, `import: summary-backfill` ×1, no `import: summary source=`/`date-refreshed`) and the intact KB.
- Module docstring records the baseline (17,407 rows · avg 15,217 chars · 252.6 MB), the mechanism (`_index_file` whole-entity load + `expire_on_commit=False`), the captured statement verbatim, the post-fix lazy-load shape measured via an ad-hoc `defer(Document.content)`, and the verdict **CONFIRMED — the fast path SELECTs `content` for every file inspected**.
**Gates** — `uv run pytest tests/integration/test_import_fast_path_columns.py -v` → 5 passed; `uv run pytest --cov=app --cov-report=term-missing` → **4511 passed, 0 failed, exit 0**, `app/` coverage **99%** (importer 99%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
**Docs** — no doc impact (test-only; the README/AGENTS sentences belong to tasks 02/03 per the phase plan).
**Notable decisions** — the two per-source image post-pass SELECTs (`_enrich_pending_images`, `_backfill_image_contexts`) still load `content` and are deliberately *out of scope* (once per source, not per file), so the flip target is the `row_loads()` subset, not "every documents SELECT" — documented so task 02 flips cleanly. Three earlier `test_sync_api.py`/`test_chat_api.py` errors were self-inflicted: I ran probe tests concurrently against the shared Postgres; the serial re-run is fully green, no pre-existing defects.
**Next pending task:** `.agents/phases/todo/185_sync_defer_content_fast_path/02_defer_content.md` (wire the shared `_doc_row_stmt`/`defer(Document.content)` helper into the four fast-path row loads, flip the probe).
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__01_fast_path_statement_probe.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__01_fast_path_statement_probe.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
............................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 862 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6870 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__02_defer_content.a1.err
View File
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__02_defer_content.a1.md
+15
View File
@@ -0,0 +1,15 @@
**Implemented (phase 185 task 02)**
- `app/rag/importer.py`: new module-level `_doc_row_stmt(source, rel)` (next to `_file_hash`) returning `select(Document).options(defer(Document.content)).where(source, path)`; `Select` + `defer` imports added. Only `content` is deferred — no `load_only`, no `raiseload`/`noload` (LOCKED A1/A2).
- All four fast-path row loads now share it: `_index_file` L1557, `_index_pdf_file` L2018, `_index_one_embedded_image` L2777 (`image_doc`), `_index_image_file` L3777. Surrounding logic untouched.
- Fifth-site note: the only other `select(Document)`s are the two per-source image post-passes (L3336/L3345, `_backfill_image_contexts` L3495) and `_prune`'s phase-184 `load_only` light row (L4076) — NOT fast-path loads, left as-is by D1.
- Docstrings: module phase-147 paragraph, `_index_file`, `_unchanged_doc` step-3 comment (the backfill is the ONE read that pulls deferred `content` back — one SELECT, one time).
- Probe flipped in `tests/integration/test_import_fast_path_columns.py`: `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 1`, `test_unchanged_run_hydrates_content_pre_fix` → `test_unchanged_run_defers_content_post_fix` (no `documents.content` in the fast-path SELECT, exactly 1 PK follow-up run-wide), docstring verdict **CONFIRMED → FIXED** with the post-fix captured statement (verified verbatim against a live capture) and the 6→7 statement arithmetic.
- New `tests/unit/test_importer_defer_content.py` (the D1 unit pins): `content` is the only column missing from the compiled projection vs the full mapper set; predicate stays `(source, path)`; all four functions call the helper and none builds its own `select(Document)`; exactly 4 call sites; AST guard that no `raiseload`/`noload` is called/imported; real-session read-back of the deferred column.
**Gates** — `uv run pytest` → **4517 passed, 0 failed**; `uv run pytest --cov=app --cov-report=term-missing` → **app/ 99%** (importer.py 99%), >90% met; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. `test_importer_unchanged_fast_path.py` green with **zero edited expectations** (no harness change was needed — those tests use the real Postgres session, so the lazy load works as-is). Extra regression runs: `test_sync_changed_only.py` (3), `test_document_viewer.py` + `test_document_summaries.py` (12), `test_sync_session_reset.py` (4) all pass.
**Docs** — `README.md`: fast-path row lookup is metadata-only + a new "A full walk reads row metadata, not the corpus (phase 185)" bullet under *Sync from the UI*. `AGENTS.md`: new operating rule 15 (share `_doc_row_stmt`, never a fifth whole-entity load, never `load_only`/`raiseload`/`noload`, pinned by the two test files).
**Deviations** — none functional; the phase's D1 "unit pin" landed in a new file so the phase-147 matrix stayed byte-untouched.
**Next pending task:** `.agents/phases/todo/185_sync_defer_content_fast_path/03_e2e_deferred_content.md`.
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__02_defer_content.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__02_defer_content.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
..................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 864 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6872 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__03_e2e_deferred_content.a1.err
View File
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__03_e2e_deferred_content.a1.md
+17
View File
@@ -0,0 +1,17 @@
**Task 03 complete — phase 185's dedicated Playwright suite is green in isolation and the whole project is 100% green.**
**Implemented**
- `tests/e2e/test_fast_path_deferred_content.py` (new, mock-only, dedicated module app on port 8202, dedicated `BOR_SOURCES_DIR`, per-test truncate before/after + settled-sync guard, fixture tree re-seeded to original bytes each test so single-test re-runs stay honest): **T1** hand-nulled `documents.summary` + deleted stale `is_summary` chunk → Sources-page re-sync settles `0 added · 2 unchanged` (`summaries`/`dates_updated` stay 0), summary comes back byte-identical from the STORED content through the one deferred-column lazy read, exactly one `is_summary` chunk, viewer shows it; **T2** untouched re-sync then viewer renders the FULL stored text (head/tail/middle sentinels + every stored line, whitespace-collapsed); **T3** rewritten file → `0 added · 1 updated · 1 unchanged`, stored content equals new bytes, chunk count moved, neighbour hashes byte-still, viewer shows only the new text, third sync all-`unchanged`. Docstring: incident, A1/A2 deferral, test→observable mapping, isolation command.
- Docs: README phase-185 operations note was already landed (task 02); extended the README and AGENTS (rule 15) pin lists to cite the new E2E suite; full review of both files surfaced no other inconsistencies (no test asserts doc contents).
**Gates (exact)**
- `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` → **3 passed** (in isolation)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6 passed**; `…test_sync_changed_only.py -v --no-cov` → **3 passed** (in isolation)
- `uv run pytest --cov=app --cov-report=term-missing` → **4517 passed, 0 failed**, `app/` coverage **99%** (importer 99%) — >90% met
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Docs** — `README.md` + `AGENTS.md`: phase-185 pin lists now include the E2E suite; no other doc impact.
**Notable** — Commit + phase-dir move left to the harness per pipeline rules (project per-phase-commit instruction overridden). Two transient E2E failures during development were my own fixture bugs (missing source registration per test), not app defects; the shared Postgres was left clean.
**Next pending task:** none in `todo/185_sync_defer_content_fast_path/` — the phase completes with this task (harness commit + move outstanding).
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__03_e2e_deferred_content.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/185_sync_defer_content_fast_path/185_sync_defer_content_fast_path__03_e2e_deferred_content.a1.validate
+143
View File
@@ -0,0 +1,143 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 98%]
..................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 292 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 318 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 494 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 864 3 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 162 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/rag/verbatim.py 160 0 100%
app/schemas.py 549 0 100%
--------------------------------------------------
TOTAL 6872 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/screenshots/166_pre_answer_window_1280.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 69 KiB

.agents/screenshots/doc_modal_scrollbar_1280.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 42 KiB

After

Width:  |  Height:  |  Size: 42 KiB

AGENTS.md
+25
View File
@@ -172,6 +172,31 @@
`tests/e2e/test_sync_session_reset.py` (the five-document prune at batch
`2` through the Sources page: both folders cleared, the survivors' chunks
intact, zero orphan chunk rows).
15. **The unchanged fast path reads metadata, never markdown:** the four
row lookups that share it (`_index_file`, `_index_pdf_file`,
`_index_one_embedded_image`, `_index_image_file`) all go through the ONE
statement `_doc_row_stmt` (`app/rag/importer.py`), which defers
`Document.content` and NOTHING ELSE (phase 185, LOCKED A1) — the row
load is the only place the fast path gets its row, and hydrating
`content` for every inspected file is what put ~252.6 MB of markdown in
front of a run that re-indexed 217 of ~27k files. Do not add a fifth
whole-entity `select(Document).where(Document.source == …, Document.path
== …)`, do not swap the `defer` for a `load_only` allowlist (a future
column would silently become lazy and resurface as an N+1), and do not
add `raiseload`/`noload` (LOCKED A2): `_unchanged_doc`'s phase-118
summary-NULL backfill genuinely reads `doc.content` and must pay one
per-row, one-time SELECT instead of raising. Pinned by
`tests/integration/test_import_fast_path_columns.py` (the captured
fast-path `SELECT` names no `documents.content`, exactly one `SELECT`
per unchanged file with a summary, exactly one PK follow-up for the
backfill), `tests/unit/test_importer_defer_content.py` (`content` is
the only deferred column, the four loads share the helper, no
`raiseload`/`noload` in the importer) and
`tests/e2e/test_fast_path_deferred_content.py` (the owner-visible
parity: the summary-NULL backfill survives the deferral with exactly
one `is_summary` chunk, an unchanged doc's viewer shows the full
stored text, a changed file re-indexes and the fast path still fires
after).
## Quick reference
```bash
README.md
+32 -1
View File
@@ -429,7 +429,12 @@ idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
streamed) and checked against the last index's stamp *before* any
extraction or vision work — an unchanged file costs one hash + one row
lookup (no re-extract, no vision, no re-embed), while a changed file
re-indexes fully. The one exception is a PDF row that still claims a
re-indexes fully. That row lookup reads **metadata only** (phase 185:
`Document.content` is deferred), so a full-resync walk's cost is
metadata-bound instead of corpus-bound; the rare unchanged file whose
`summary` is still NULL re-reads its stored content — one query, one time,
by design — because that backfill genuinely needs the text. The one
exception is a PDF row that still claims a
page strip: it is re-gated from its current bytes (no vision, no LLM —
see "The strip repairs itself" below). A refresh after a normal editing
session takes seconds — pictorial PDFs included (their expensive
@@ -683,6 +688,32 @@ only) runs the whole source refresh in one click:
`0`-cadence run of the same fixture is byte-identical to the `3`-cadence
one (per-path chunk counts, `content`, `summary`, both hashes, the counts
line).
- **A full walk reads row metadata, not the corpus** (phase 185). The four
row lookups the unchanged fast path shares (`_index_file` and the PDF /
PDF-embedded-image / standalone-image branches) go through ONE statement,
`_doc_row_stmt`, which defers the single column the fast path never
reads — `Document.content`. The 2026-10-15 sync inspected ~27k files and
re-indexed 217, yet hydrated all 17,407 rows whole (avg 15,217 chars =
**252.6 MB of markdown**, held for the whole run by `expire_on_commit=False`
and phase 184's stamp); now an inspected row carries file_hash, summary,
dates, page count — and no markdown. Exactly ONE column is deferred, and
there is deliberately **no `raiseload`/`noload`**: the phase-118
summary-NULL backfill genuinely reads the stored content, so that one row
pays one extra `SELECT documents.content … WHERE id = …` (per-row,
one-time, already rare) instead of crashing. Counts, log words, the D1
manual-date lock, the phase-163 strip re-gate and the backfill result are
byte-identical — pinned by
`tests/integration/test_import_fast_path_columns.py` (the captured
fast-path `SELECT` names no `documents.content`, one `SELECT` per
unchanged file with a summary, exactly one lazy follow-up for the
backfill), `tests/unit/test_importer_defer_content.py` (`content` is
the only deferred column, all four loads share the helper, no
`raiseload`/`noload` anywhere in the importer) and, through the UI,
`tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
backfills to its byte-identical text from the Sources-page re-sync with
exactly one `is_summary` chunk, an unchanged doc's viewer still shows
the full stored text, and a rewritten file re-indexes to the new text
with the next sync all-`unchanged` again).
- **Cancel**: while a run is in flight, a **Cancel** button appears
next to the sync button; clicking it posts `POST /api/sync/cancel`
(admin only) and registers a cooperative cancel (`202`) — the run
app/rag/importer.py
+55 -11
View File
@@ -84,6 +84,14 @@ pre-phase on the first post-upgrade sync (one full pass), then stamp
and go fast. The previous re-burn costs (a garbage PDF's fallback
vision re-issued each sync, a non-deterministic vision model flipping
a pictorial PDF's content digest) are gone: the file bytes decide.
Phase 185 (task 02): the fast path's ``(source, path)`` row comes from the
shared :func:`_doc_row_stmt`, which defers ``Document.content`` — the ONE
column the fast path never reads and the fat one of the row (252.6 MB of
markdown over 17,407 rows on the incident box) — so a full walk's memory
scales with row METADATA, not with the corpus. Exactly one column is
deferred (no ``load_only`` allowlist, no ``raiseload``/``noload``): the
phase-118 summary-NULL backfill genuinely reads ``doc.content`` and pays
ONE lazy SELECT for that one row, by design.
Standalone images (phase 122, LOCKED A3): with the ``images`` toggle
(``BOR_IMAGES``) ON, the walk also admits the image extension set
@@ -345,8 +353,8 @@ from pathlib import Path, PurePosixPath
from typing import Any, Protocol
import pymupdf as fitz # PyMuPDF (phase 126, locked A1); `fitz` = the stable API name
from sqlalchemy import select
from sqlalchemy.orm import Session, load_only
from sqlalchemy import Select, select
from sqlalchemy.orm import Session, defer, load_only
from app.config import Settings
from app.db import SessionLocal
@@ -705,6 +713,37 @@ def _file_hash(path: Path) -> str:
return h.hexdigest()
def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
"""The importer's single Document-row statement (phase 185).
Used by the FOUR loads that share the phase-147 unchanged fast path —
:func:`_index_file`, :func:`_index_pdf_file`,
:func:`_index_one_embedded_image` and :func:`_index_image_file` — so
the deferral below cannot drift between them.
``defer(Document.content)`` — the unchanged fast path (phase 147) reads
file_hash/is_image/page_count/summary/dates, never the stored markdown,
and ``app/db.py`` builds sessions with ``expire_on_commit=False``, so
hydrating ``content`` for EVERY inspected file is what took the
2026-10-15 run to ~250 MB on a 17,407-document KB (avg 15,217
chars/row): the walk inspected ~27k files and re-indexed 217.
ONE column is deferred on purpose (LOCKED A1): a ``load_only``
allowlist would silently defer every column a later phase adds, and the
deferred read would then surface as an unexplained N+1. No
``raiseload``/``noload`` either (LOCKED A2): the phase-118 summary-NULL
backfill (``_unchanged_doc`` step 3) genuinely reads ``doc.content`` and
must keep working — it costs one extra SELECT for that one row, a
per-row, one-time, already-rare cost, accepted instead of a rule that
would crash the branch where it should query.
"""
return (
select(Document)
.options(defer(Document.content))
.where(Document.source == source, Document.path == rel)
)
async def import_sources(
sources: list[Path],
llm: Embedder,
@@ -1194,7 +1233,10 @@ async def _unchanged_doc(
# branch paths — the next sync is then fast either way).
doc.file_hash = file_digest
# 3 — the phase-118 (A2) summary-NULL backfill, from the stored
# content (see the docstring — never a re-read/re-extract).
# content (see the docstring — never a re-read/re-extract). Phase 185:
# this is the ONE fast-path read that pulls the deferred ``content``
# back — one SELECT, one time, by design (LOCKED A2: no raiseload, the
# branch must query rather than crash).
if doc.summary is None:
await _store_summary(
session, doc=doc, source=source, rel=rel, content=doc.content,
@@ -1457,8 +1499,12 @@ async def _index_file(
Phase 147 (task 02): the UNCHANGED FAST PATH. BEFORE the
image/text/PDF dispatch, the file's RAW source bytes are hashed
(:func:`_file_hash` — streamed), the ``(source, path)`` row is
looked up, and the D2 mtime fallback for *raw_date* runs (once,
here — the branches receive the already-computed date) — a
looked up through the shared :func:`_doc_row_stmt` (phase 185: with
``Document.content`` DEFERRED — the fast path reads row METADATA only,
so a full walk no longer hydrates the KB's markdown; the phase-118
summary-NULL backfill is the one read that pulls the text back, one
SELECT for that row), and the D2 mtime fallback for *raw_date* runs
(once, here — the branches receive the already-computed date) — a
``doc.file_hash == file_digest`` match returns through the shared
unchanged path (:func:`_unchanged_doc`) with ZERO file read beyond
the streamed hash, ZERO PDF extraction, ZERO vision call, and
@@ -1508,7 +1554,7 @@ async def _index_file(
# ``read_text``/``read_bytes``, no ``extract_pdf_text``, no
# fallback/pictorial vision, no re-chunk/re-embed/re-summary.
file_digest = _file_hash(full_path)
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if raw_date is None:
# D2 fallback: no source date in the map → the file's mtime
# (one stat). Read before the unchanged early-return — the
@@ -1969,7 +2015,7 @@ async def _index_pdf_file(
else hashlib.sha256(b"").hexdigest()
)
title = _extract_pdf_title(full_path, fallback=full_path.stem)
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if doc is not None and doc.content_hash == digest:
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
# row, or a bytes-changed/content-same PDF (the content digest
@@ -2728,9 +2774,7 @@ async def _index_one_embedded_image(
return
image_rel = f"{rel}#page={img.page}#img={img.index}{ext}"
digest = hashlib.sha256(img.data).hexdigest()
image_doc = session.scalar(
select(Document).where(Document.source == source, Document.path == image_rel)
)
image_doc = session.scalar(_doc_row_stmt(source, image_rel))
if image_doc is not None and image_doc.content_hash == digest:
# Unchanged embedded image (byte digest) — no re-describe, no
# re-embed (the standalone image's unchanged rule).
@@ -3730,7 +3774,7 @@ async def _index_image_file(
# byte change.
data = full_path.read_bytes()
digest = hashlib.sha256(data).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
doc = session.scalar(_doc_row_stmt(source, rel))
if doc is not None and doc.is_image and doc.content_hash == digest:
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
tests/e2e/test_fast_path_deferred_content.py
+760
View File
@@ -0,0 +1,760 @@
"""Phase 185 E2E (Playwright, mock-only): the deferred column still shows up
where the owner reads it.
Source: owner incident 2026-10-15 (the same run phases 183/184 fixed). The
unchanged fast path looked each file up by a WHOLE-ENTITY ``select(Document)``
— ``documents.content`` included — and ``app/db.py`` builds sessions with
``expire_on_commit=False``, so the 2026-10-15 sync inspected ~27k files and
held ~252.6 MB of markdown (17,407 rows × avg 15,217 chars) resident to
discover that 217 files had changed.
The fix (``app/rag/importer.py``): the four fast-path row loads share the ONE
statement ``_doc_row_stmt``, which defers ``Document.content`` — exactly ONE
column (LOCKED A1: no ``load_only`` allowlist, so a future column can never
silently become lazy) and deliberately NO ``raiseload``/``noload`` (LOCKED
A2: the phase-118 summary-NULL backfill genuinely reads the stored markdown
and must pay one per-row, one-time SELECT instead of crashing). The
statement-level proof — the captured fast-path ``SELECT`` names no
``documents.content``, one ``SELECT`` per unchanged file with a summary, one
lazy follow-up for the backfill — lives in
``tests/integration/test_import_fast_path_columns.py``. What a browser can
see is what THIS module pins: the deferral is invisible to the owner.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is skipped — the backfilled summary is
compared BYTE-IDENTICAL against the summary the first import stored, which
is a property of the deterministic mock summarizer; against the real endpoint
the comparison would be a measurement, not a pin.
One module app on a dedicated port (the conftest session app owns 8123 in a
combined run), booting against the mock LLM with ``BOR_GIT_SOURCES`` forced
empty (an operator's local ``.env`` must never leak a real source into a
run) and its own ``BOR_SOURCES_DIR`` / ``BOR_PDF_PAGE_DIR`` under
``tmp_path_factory``. The fixture KB is ONE plain (non-git) directory named
:data:`SOURCE_NAME` (``kind='local'`` rows index under ``Path(path).name``,
so the source label is stable whatever temp root pytest picked) holding TWO
``.md`` files; both carry a sentinel phrase on their FIRST content line and
another on their LAST line, so "the viewer shows the FULL stored text" is
provable at both ends of a file.
Isolation: the shared E2E Postgres is truncated BEFORE **and** AFTER every
test (``chunks, documents, query_log, kb_overview, git_sources, sources_meta,
folder_summaries``), after waiting for the app to hold no running sync (the
run state lives in the app's memory — a leaked run would keep importing into
the next test's truncated KB). The fixture TREE is re-seeded to its original
bytes at the top of every test too (test 3 rewrites one file — a lone re-run
of this module must still see ``2 added``, not ``0 added · 2 unchanged``),
and each test registers its own source row.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_summary_null_backfill_survives_the_deferral`` — with one row's
``documents.summary`` nulled AND its stale ``is_summary`` chunk deleted
directly (the phase-118 staging, extended so the chunk's REAPPEARANCE
proves the backfill ran), the Sources-page re-sync still settles
``0 added · 2 unchanged`` (a backfill is not an ``updated``/``added``),
the document page shows a non-empty summary identical to the one run 1
stored — produced from the STORED content through the deferred-column
lazy read — and exactly ONE ``is_summary`` chunk exists for the doc
(no duplicate).
2. ``test_unchanged_document_viewer_shows_full_stored_text`` — a second
sync of untouched files settles ``0 added · 2 unchanged`` and the
document view then renders the FULL stored text: the head AND the tail
sentinel both present, every stored line present, no truncation, no
empty body. The row lookup the deferral thinned is the same lookup the
fast path answers with; the viewer reads its row separately and still
gets every byte.
3. ``test_changed_file_reindexes_under_the_deferral`` — one file is
rewritten on disk (new head/tail sentinels, more sections, different
length); the re-sync settles ``0 added · 1 updated · 1 unchanged``, the
viewer shows the NEW text with NO stale fragment, the stored content
equals the new file bytes, the content-chunk count moved, the
untouched neighbour row did not, and a THIRD sync reports all
``unchanged`` — the fast path still fires after a deferral-mediated
write.
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import delete, func, select, text
from app.db import SessionLocal
from app.models import Chunk, Document
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
#: Dedicated module port (the house env-overridable pattern): the conftest
#: session app owns 8123 in a combined run.
APP_PORT = int(os.environ.get("E2E_APP_PORT_DEFERCONTENT", "8202"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
SOURCES_URL = "/sources.html"
#: The fixture directory's basename — the ``source`` the importer records for
#: a ``kind='local'`` row (``Path(path).name``), stable whatever temp root
#: this run got.
SOURCE_NAME = "defer-fast-path"
ALPHA_PATH = "notes/alpha.md"
BETA_PATH = "notes/beta.md"
N_FILES = 2
#: Head/tail sentinels: the FIRST content line and the LAST line of each
#: fixture file, so "the viewer shows the FULL stored text" is proven at
#: both ends (test 2), "no stale fragment" is proven at both ends (test 3).
ALPHA_HEAD = "RESE-DEFER-ALPHA-HEAD-3c81"
ALPHA_TAIL = "RESE-DEFER-ALPHA-TAIL-7e52"
ALPHA_MIDDLE = "RESE-DEFER-ALPHA-MIDDLE-0f64"
BETA_HEAD = "RESE-DEFER-BETA-HEAD-5b19"
BETA_TAIL = "RESE-DEFER-BETA-TAIL-8a64"
#: What test 3 rewrites alpha.md into (and what must have replaced every
#: ALPHA_* marker in the KB and in the viewer).
REVISED_HEAD = "RESE-DEFER-ALPHA2-HEAD-e208"
REVISED_TAIL = "RESE-DEFER-ALPHA2-TAIL-41cb"
#: Test 1 stages THIS row (phase-118's summary-NULL backfill case).
BACKFILL_PATH = BETA_PATH
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: A two-file mock sync (walk + embeds + summary + KB overview) lands in
#: seconds; this is an order of magnitude of head-room with no client-side
#: hard timeout (the test_sync_session_reset.py budget).
SETTLE_TIMEOUT_MS = 90_000
# ---------------------------------------------------------------------------
# The fixture source
# ---------------------------------------------------------------------------
def _alpha_text() -> str:
"""The ORIGINAL bytes of ``notes/alpha.md`` (test 2's full-text subject,
test 3's rewrite subject). Two H2 sections — a section is the md chunker's
unit, so the rewrite's extra sections move the chunk count."""
return (
"# Deferred content alpha\n"
"\n"
f"Head marker {ALPHA_HEAD} opens this file.\n"
"\n"
f"Middle marker {ALPHA_MIDDLE} sits between the sections.\n"
"\n"
"## Alpha body\n"
"\n"
"Alpha prose that the unchanged fast path must never need to read: "
"the walk wants the row metadata, not these paragraphs. It keeps "
"growing so the file is unmistakably a document and not a stub.\n"
"\n"
"## Alpha notes\n"
"\n"
"More alpha prose so the content chunk count is more than one. Each "
"section is its own chunk, which is what test 3 counts when it "
"rewrites this file under the deferral.\n"
"\n"
f"Tail marker {ALPHA_TAIL} closes this file.\n"
)
def _revised_alpha_text() -> str:
"""The rewrite: NEW head/tail sentinels, two extra sections, clearly
different length — every old marker is gone from the bytes."""
return (
"# Deferred content alpha revised\n"
"\n"
f"Head marker {REVISED_HEAD} opens the revised file.\n"
"\n"
"## Alpha body\n"
"\n"
"The revised alpha prose. The deferral changed which columns the "
"unchanged fast path reads; a CHANGED file must still re-index its "
"full text, so this paragraph and its siblings must land in the KB "
"verbatim, and every marker of the original file must be gone.\n"
"\n"
"## Alpha notes\n"
"\n"
"Revised notes prose, longer than before on purpose so the stored "
"length and the viewer both prove the write path carried the new "
"text through the same session that read the row with content "
"deferred.\n"
"\n"
"## Alpha appendix one\n"
"\n"
"A brand new section that did not exist before the rewrite, added "
"so the content chunk count of this document must change when the "
"re-indexed file is re-chunked.\n"
"\n"
"## Alpha appendix two\n"
"\n"
"A second brand new section, because the point of the phase is "
"that NOTHING about what a run does changed — only which columns "
"travel on an unchanged row lookup.\n"
"\n"
f"Tail marker {REVISED_TAIL} closes the revised file.\n"
)
def _beta_text() -> str:
"""``notes/beta.md`` — test 1's backfill subject. Ordinary prose with
its own head/tail sentinels."""
return (
"# Deferred content beta\n"
"\n"
f"Head marker {BETA_HEAD} opens this file.\n"
"\n"
"## Beta body\n"
"\n"
"Beta is the document whose stored summary test 1 nulls directly, "
"so the Sources-page re-sync must backfill it from the stored "
"content — the ONE fast-path read that pulls the deferred column "
"back, one SELECT, one time, by design.\n"
"\n"
f"Tail marker {BETA_TAIL} closes this file.\n"
)
def _seed_source(root: Path) -> Path:
"""(Re-)write BOTH fixture files under ``root / SOURCE_NAME``.
Idempotent on purpose: every test calls it first, so the tree is at its
ORIGINAL bytes whatever an earlier test (test 3!) left behind — a lone
re-run of any single test still sees a fresh-KB ``2 added``.
"""
base = root / SOURCE_NAME
files = {ALPHA_PATH: _alpha_text(), BETA_PATH: _beta_text()}
for rel, body in files.items():
target = base / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
assert not (base / ".git").exists() # a plain directory — no git anywhere
return base
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The fixture tree the module app walks (the app server runs on the same
host, so the path is visible to it)."""
return _seed_source(tmp_path_factory.mktemp("bor_fast_path_deferred"))
# ---------------------------------------------------------------------------
# The module app
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def app_server(
mock_llm: int, source_dir: Path, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The real app under test — per-module env: the mock LLM, the
mock-calibrated gate (the conftest convention), NO git fallback, and a
dedicated ``BOR_SOURCES_DIR`` so no run writes into the operator's.
The phase's subject needs no knob: the ``defer(Document.content)`` fast
path is the code default (LOCKED A1/A2) — this app is the app every
other suite runs with, walked over a fixture small enough to read.
"""
if USE_REAL_LLM:
pytest.skip(
"mock-only suite — the byte-identical backfilled summary is a "
"property of the deterministic mock (E2E_REAL_LLM unsupported)"
)
assert mock_llm and source_dir
scratch = tmp_path_factory.mktemp("bor_fast_path_deferred_app")
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = f"http://127.0.0.1:{MOCK_PORT}/v1"
# Mock-calibrated gate (the conftest pattern): both floors must stay <=
# the threshold or the app refuses to boot (app/config.py).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# No env-fallback rows: the registry each test writes through the
# authenticated API is the only source list.
env["BOR_GIT_SOURCES"] = ""
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
proc = subprocess.Popen(
[
sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning",
],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
# ---------------------------------------------------------------------------
# Isolation
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern, the phase-184 list:
``sources_meta`` and ``folder_summaries`` reset with the KB)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No run may leak across a truncate: wait for a non-running status with
an own admin session (the endpoint is admin-only). Usually a no-op — the
tests settle before they end."""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
with httpx.Client(base_url=base_url, timeout=5.0) as client:
client.post("/api/login", json={"password": ADMIN_PASSWORD})
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation: the shared E2E Postgres is truncated before AND
after every test, the app holding no running sync at both boundaries."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_wait_sync_settled(app_url)
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers: the UI side
# ---------------------------------------------------------------------------
def _add_local_source(page: Page, base_url: str, path: Path) -> None:
"""Register the fixture directory through the authenticated API (the
phase-49 shape — the admin cookie rides ``page.request``)."""
r = page.request.post(
f"{base_url}/api/git-sources",
data={"kind": "local", "path": str(path)},
)
assert r.status == 201, f"expected 201 for {path}: {r.status} {r.text}"
def _started_at(body: dict[str, Any]) -> float:
"""The status body's ``started_at`` as a wall-clock epoch (the server
stamps it in UTC and this process runs on the same host)."""
started = body.get("started_at")
if not started:
return 0.0
try:
return datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return 0.0
def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]:
"""Click the Sources page **Sync sources** button and wait for THIS run's
terminal state (the module app is shared, so a terminal counts only after
a ``running`` tick or when its ``started_at`` post-dates the click).
Settles the UI on success: the button re-enabled, ``Synced HH:MM``, no
error banner (the test_sync_session_reset.py helper)."""
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(btn).to_be_enabled()
clicked_at = time.time()
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
deadline = time.monotonic() + SETTLE_TIMEOUT_MS / 1000
saw_running = False
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = page.request.get(f"{base_url}/api/sync/status")
assert r.status == 200, r.text
body = r.json()
state = body["state"]
if state == "running":
saw_running = True
elif state in ("success", "failed", "cancelled") and (
saw_running or _started_at(body) >= clicked_at - 1.0
):
break
time.sleep(0.2)
else:
raise AssertionError(f"sync never reached a terminal state: {body}")
assert body["state"] == "success", f"sync failed: {body}"
assert body["error"] is None, body
expect(btn).to_be_enabled(timeout=15_000)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
expect(page.locator("#sync-error-banner")).to_be_hidden()
return body
def _expect_counts_line(page: Page, expected: str) -> None:
"""The counts line — the owner-visible sync report on the single sync
text surface (phase 150; sources.js fmtSyncResult; the zero "updated"
part is omitted)."""
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
def _goto_sources(page: Page, base_url: str) -> None:
"""Back to the Sources page (the sync button's home) after a viewer
detour — the session cookie rides the navigation."""
page.goto(base_url + SOURCES_URL)
expect(page.locator("#sync-btn")).to_be_visible(timeout=30_000)
expect(page.locator("#sync-btn")).to_be_enabled()
def _open_document_page(page: Page, base_url: str, path: str) -> None:
"""Open the standalone viewer for one fixture document (the phase-10
surface; the session cookie from ``login`` rides the navigation)."""
page.goto(
f"{base_url}/document.html?source={SOURCE_NAME}"
f"&path={path.replace('/', '%2F')}"
)
expect(page.locator("#doc-content .doc-md")).to_have_count(1, timeout=30_000)
# ---------------------------------------------------------------------------
# Helpers: the DB side
# ---------------------------------------------------------------------------
def _summary_map() -> dict[str, str | None]:
"""``path`` → ``documents.summary`` for both fixture rows."""
with SessionLocal() as db:
rows = db.execute(
select(Document.path, Document.summary).where(
Document.source == SOURCE_NAME
)
).all()
return {str(path): (None if summary is None else str(summary)) for path, summary in rows}
def _summary_chunk_contents(path: str) -> list[str]:
"""The ``is_summary`` chunk contents of one document — the duplicate
check (phase 30: at most one per doc at a time)."""
with SessionLocal() as db:
return [
str(content)
for (content,) in db.execute(
select(Chunk.content)
.join(Document, Document.id == Chunk.document_id)
.where(
Document.source == SOURCE_NAME,
Document.path == path,
Chunk.is_summary.is_(True),
)
).all()
]
def _content_chunk_counts() -> dict[str, int]:
"""``path`` → content-chunk count per fixture doc (the ``position >= 0``
rows — the ``is_summary`` position −1 chunk stays out)."""
with SessionLocal() as db:
rows = db.execute(
select(Document.path, func.count(Chunk.id))
.join(Chunk, Chunk.document_id == Document.id)
.where(
Document.source == SOURCE_NAME,
Chunk.position >= 0,
)
.group_by(Document.path)
).all()
return {str(path): int(count) for path, count in rows}
def _stored_content(path: str) -> str:
with SessionLocal() as db:
content = db.scalar(
select(Document.content).where(
Document.source == SOURCE_NAME, Document.path == path
)
)
assert content is not None, f"{path} is not in the KB"
return str(content)
def _row_stamp(path: str) -> tuple[Any, Any]:
"""``(content_hash, file_hash)`` of one fixture row — the neighbour's
byte-still proof (a re-index would re-stamp both)."""
with SessionLocal() as db:
row = db.execute(
select(Document.content_hash, Document.file_hash).where(
Document.source == SOURCE_NAME, Document.path == path
)
).one()
return row[0], row[1]
def _stage_null_summary(path: str) -> None:
"""The phase-118 backfill case, staged directly (the way phase 118's
tests null the column), extended for this phase: the stale
``is_summary`` CHUNK goes too, so the chunk's REAPPEARANCE after the
re-sync proves the backfill ran rather than merely survived."""
with SessionLocal() as db:
doc_id = db.scalar(
select(Document.id).where(
Document.source == SOURCE_NAME, Document.path == path
)
)
assert doc_id is not None, f"{path} is not in the KB"
db.execute(delete(Chunk).where(Chunk.document_id == doc_id, Chunk.is_summary.is_(True)))
db.execute(
text(
"UPDATE documents SET summary = NULL "
"WHERE source = :s AND path = :p"
),
{"s": SOURCE_NAME, "p": path},
)
db.commit()
# The stage landed — otherwise every later assertion would pass vacuously.
assert _summary_map()[path] is None
assert _summary_chunk_contents(path) == []
# ---------------------------------------------------------------------------
# 1. The summary-NULL backfill survives the deferral
# ---------------------------------------------------------------------------
def test_summary_null_backfill_survives_the_deferral(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""Run 1 indexes both docs and summarizes them (phase 118 A2: markdown
docs are summarized too). One row is then staged by hand —
``documents.summary = NULL`` AND its ``is_summary`` chunk deleted — and
the Sources-page re-sync runs WITH the deferral in place:
* the run reports the doc as ``unchanged``, not ``added``/``updated``
(the settle line and the terminal detail both say so — the backfill
must not masquerade as a re-index, and ``dates_updated``/``summaries``
stay 0);
* the document page shows a NON-EMPTY summary again, byte-identical to
the one run 1 stored — produced from the STORED content through the
one lazy read the deferral leaves on this branch (LOCKED A2);
* exactly ONE ``is_summary`` chunk exists for the doc (it was recreated,
not duplicated)."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent) # idempotent — the tree starts at its original bytes
login(page, app_url, next=SOURCES_URL)
_add_local_source(page, app_url, source_dir)
# --- run 1: everything is new, both docs get a summary ----------------
body1 = _sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"{N_FILES} added")
assert body1["detail"]["added"] == N_FILES, body1["detail"]
originals = _summary_map()
assert set(originals) == {ALPHA_PATH, BACKFILL_PATH}, originals
stored_summary = originals[BACKFILL_PATH]
assert stored_summary, "run 1 must have summarized the markdown doc (phase 118 A2)"
# --- stage the phase-118 case directly --------------------------------
_stage_null_summary(BACKFILL_PATH)
# --- run 2: the unchanged re-sync backfills from the STORED content ----
body2 = _sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
detail = body2["detail"]
assert detail["added"] == 0 and detail["updated"] == 0, detail
assert detail["unchanged"] == N_FILES, detail
assert detail["errors"] == 0, detail
# The counts stay honest: a backfill is neither a re-summary of changed
# content (``summaries``) nor a date refresh (``dates_updated``).
assert detail["summaries"] == 0, detail
assert detail["dates_updated"] == 0, detail
assert detail["pruned"] == 0, detail
# --- the DB side: the summary is back, byte-identical, one chunk ------
after = _summary_map()
assert after[BACKFILL_PATH] == stored_summary, (
"the backfilled summary must equal what run 1 stored from the same "
"content (the deterministic mock makes byte-equality a pin)"
)
chunks = _summary_chunk_contents(BACKFILL_PATH)
assert len(chunks) == 1, f"exactly one is_summary chunk, got {len(chunks)}"
assert chunks[0] == stored_summary
# --- the owner's surface: the document page shows the summary ---------
_open_document_page(page, app_url, BACKFILL_PATH)
expect(page.locator("#doc-title")).to_have_text("Deferred content beta")
summary_text = page.locator("#doc-content .doc-summary-text")
expect(summary_text).to_have_count(1)
expect(summary_text).not_to_be_empty()
expect(summary_text).to_have_text(stored_summary)
# The deferral did not thin the doc for the viewer either.
expect(page.locator("#doc-content")).to_contain_text(BETA_HEAD)
expect(page.locator("#doc-content")).to_contain_text(BETA_TAIL)
# ---------------------------------------------------------------------------
# 2. An unchanged document's viewer still shows the FULL stored text
# ---------------------------------------------------------------------------
def test_unchanged_document_viewer_shows_full_stored_text(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""Run 1 indexes, run 2 changes NOTHING (untouched bytes, so every file
walks the deferred-row fast path) — and the document view then renders
the FULL stored text: the sentinel at the START and the sentinel at the
END are both present, every stored line is present, no truncation, no
empty body. The deferral must not have made the viewer's source row
thinner."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent)
login(page, app_url, next=SOURCES_URL)
_add_local_source(page, app_url, source_dir)
_sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"{N_FILES} added")
_sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
stored = _stored_content(ALPHA_PATH)
assert stored == _alpha_text(), "the stored content is the fixture's original bytes"
_open_document_page(page, app_url, ALPHA_PATH)
expect(page.locator("#doc-title")).to_have_text("Deferred content alpha")
body = page.locator("#doc-content .doc-md")
expect(body).not_to_be_empty()
# Both ENDS of the file, in the rendered view (the task's pin): a viewer
# fed by a truncated row would be missing one of them.
expect(body).to_contain_text(ALPHA_HEAD)
expect(body).to_contain_text(ALPHA_TAIL)
expect(body).to_contain_text(ALPHA_MIDDLE)
# The strong form: EVERY stored line renders (headings render without
# their leading '#', the fixture carries no list/inline markup, so the
# comparison is line-for-line on whitespace-collapsed text — a viewer
# fed by a thinner row, or truncated anywhere in the middle, drops one
# of these).
rendered = " ".join(body.inner_text().split())
for line in stored.splitlines():
if not line.strip():
continue
expected = " ".join(line.lstrip("#").split())
assert expected in rendered, f"stored line not rendered: {line!r}"
# ---------------------------------------------------------------------------
# 3. A changed file re-indexes under the deferral
# ---------------------------------------------------------------------------
def test_changed_file_reindexes_under_the_deferral(
page: Page, app_url: str, source_dir: Path, db_ready: None
) -> None:
"""One file is rewritten on disk (new head/tail sentinels, two extra
sections, clearly different length). The re-sync runs with the row read
deferred and must behave exactly as before the phase: ``1 updated`` with
the neighbour ``unchanged``, the viewer shows the NEW text with NO stale
fragment, the stored content equals the new bytes, the content-chunk
count moved, and the untouched neighbour's hashes did not. A THIRD sync
then reports everything ``unchanged`` — the fast path still fires after
a deferral-mediated write."""
page.set_default_timeout(30_000)
_seed_source(source_dir.parent)
login(page, app_url, next=SOURCES_URL)
_add_local_source(page, app_url, source_dir)
_sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"{N_FILES} added")
chunks_before = _content_chunk_counts()
stamps_before = _row_stamp(BETA_PATH)
assert set(chunks_before) == {ALPHA_PATH, BETA_PATH}, chunks_before
# --- rewrite alpha on disk (new sentinels, longer, more sections) -----
(source_dir / ALPHA_PATH).write_text(_revised_alpha_text(), encoding="utf-8")
body = _sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"0 added · 1 updated · {N_FILES - 1} unchanged")
detail = body["detail"]
assert detail["added"] == 0 and detail["updated"] == 1, detail
assert detail["unchanged"] == N_FILES - 1, detail
assert detail["errors"] == 0, detail
# The stored content is the NEW file bytes — the write path carried the
# full text through the same session whose row read deferred it.
assert _stored_content(ALPHA_PATH) == _revised_alpha_text()
# The content-chunk count moved (the rewrite added two whole sections).
chunks_after = _content_chunk_counts()
assert chunks_after[ALPHA_PATH] != chunks_before[ALPHA_PATH], (
f"chunk count must move on a re-index: {chunks_before} -> {chunks_after}"
)
assert chunks_after[ALPHA_PATH] > 0 and chunks_before[ALPHA_PATH] > 0
# The untouched neighbour row did not move (both hashes byte-still).
assert _row_stamp(BETA_PATH) == stamps_before
# The viewer shows the NEW text and NO stale fragment.
_open_document_page(page, app_url, ALPHA_PATH)
expect(page.locator("#doc-title")).to_have_text("Deferred content alpha revised")
rendered = page.locator("#doc-content .doc-md")
expect(rendered).to_contain_text(REVISED_HEAD)
expect(rendered).to_contain_text(REVISED_TAIL)
expect(rendered).not_to_contain_text(ALPHA_HEAD)
expect(rendered).not_to_contain_text(ALPHA_TAIL)
expect(rendered).not_to_contain_text(ALPHA_MIDDLE)
# --- run 3: the fast path still fires after the deferral-mediated write
_goto_sources(page, app_url)
body3 = _sync_from_sources_page(page, app_url)
_expect_counts_line(page, f"0 added · {N_FILES} unchanged")
assert body3["detail"]["updated"] == 0, body3["detail"]
assert body3["detail"]["unchanged"] == N_FILES, body3["detail"]
tests/e2e/test_modal_back_button.py
+23 -6
View File
@@ -99,6 +99,7 @@ Test → observable mapping (Playwright Mapping Rule):
from __future__ import annotations
import asyncio
import time
from pathlib import Path
from threading import Thread
from typing import Any
@@ -229,13 +230,29 @@ def _assert_crumb(page: Page, links: list[str], current: str) -> None:
``.kb-crumb-link`` per ancestor (in order — the first is always the
``Knowledge base`` jump-to-top segment), then the last segment as
the ``.kb-crumb-current`` span with ``aria-current="page"``.
Hidden = the TOP level (no drill)."""
Hidden = the TOP level (no drill).
The chain is POLLED, not read once: phase 167 made a drill LAZY (the
level and its breadcrumb render when ``/api/docs/tree/node`` resolves),
so the single read that sufficed while the whole tree shipped at boot
now races the fetch and intermittently returns the level the user came
FROM.
"""
expected = [*links, current]
expect(page.locator("#kb-crumb")).to_be_visible()
texts = page.eval_on_selector_all(
"#kb-crumb :is(.kb-crumb-link, .kb-crumb-current)",
"(els) => els.map((el) => el.textContent)",
)
assert texts == [*links, current], texts
def _crumb_texts() -> list[str]:
return page.eval_on_selector_all(
"#kb-crumb :is(.kb-crumb-link, .kb-crumb-current)",
"(els) => els.map((el) => el.textContent)",
)
deadline = time.monotonic() + 30.0
texts = _crumb_texts()
while texts != expected and time.monotonic() < deadline:
page.wait_for_timeout(50)
texts = _crumb_texts()
assert texts == expected, texts
cur = page.locator("#kb-crumb .kb-crumb-current")
expect(cur).to_have_count(1)
expect(cur).to_have_attribute("aria-current", "page")
tests/e2e/test_nav_rename_sources.py
+4 -3
View File
@@ -112,13 +112,14 @@ CURRENT_LINK: dict[str, str | None] = {
#: The four primary nav links, in their physical DOM order — the labels
#: after the phase-48 swap (ids/hrefs unchanged).
NAV_LABELS = ("Chat", "RAG", "Sources", "Tuning")
#: The FIFTH–SEVENTH a.nav-link in every page header (phase 53
#: The FIFTH–EIGHTH a.nav-link in every page header (phase 53
#: saved-chat history + phase 79 task 06 access tokens + phase 91
#: theme tab — all ship-hidden, revealed by header.js for admins).
#: theme tab + phase 168 admin settings panel — all ship-hidden,
#: revealed by header.js for admins).
#: The DOM enumeration below therefore always sees them (pre-existing;
#: the list matches the real nav — the phase-34 one-bar contract ships
#: the SAME nav, incl. the Tokens and Theme links, on every page).
NAV_TAIL = ("History", "Tokens", "Theme")
NAV_TAIL = ("History", "Tokens", "Theme", "Settings")
#: The login.js script — route pattern for the redirect suppression.
LOGIN_JS_ROUTE = re.compile(r"/assets/login\.js(\?.*)?$")
tests/e2e/test_navbar_refresh.py
+18 -1
View File
@@ -71,6 +71,7 @@ from collections.abc import Callable
from pathlib import Path
from threading import Thread
from typing import Any
from urllib.parse import urlsplit
import httpx
from playwright.sync_api import Locator, Page, Request, expect
@@ -510,7 +511,23 @@ def test_all_four_data_views_refetch_on_reshow(
log = _get_logger(page)
def count(endpoint: str) -> int:
return sum(1 for u in log if u.endswith(endpoint))
"""Requests to ONE endpoint: exact path, plus every query pair the
pin names. Phase 175 paginated the History list
(``/api/chats?limit=25&offset=0``, ``history.js``), so the plain
``endswith`` this suite used to count with stopped seeing the
view's own fetch altogether — the mount baseline never arrived and
the loop timed out on the FIRST view."""
path, _, query = endpoint.partition("?")
wanted = [pair for pair in query.split("&") if pair]
total = 0
for url in log:
parts = urlsplit(url)
if parts.path != path:
continue
params = parts.query.split("&") if parts.query else []
if all(pair in params for pair in wanted):
total += 1
return total
for nav_sel, view_path, endpoint in DATA_VIEWS:
# First show (the mount) — wait until the mount's own load
tests/e2e/test_source_chip_quality.py
+68 -20
View File
@@ -114,6 +114,46 @@ def _ask(page: Page, message: str) -> None:
page.click("#send-btn")
def _assert_ranked(recorded: str, groups: list[list[str]]) -> None:
"""Compare a ``query_log.sources`` string against an expected rank list
whose trailing entries are TIE-GROUPS (order-free sets).
WHY the groups exist: the E2E mock embedder builds real token-overlap
vectors, so several tiny fixture documents end up on EXACTLY the same
fused score. ``fuse`` then sorts by ``(-score, -cosine, path,
position)``… but that only orders chunks whose SCORES differ; between
two documents tied in both lists the vector rank itself comes out of
``ORDER BY embedding <=> q``, which is not a total order, so Postgres
(a parallel seq scan handing its partial sorts back) may return the
tied pair the other way round between two runs of the SAME fixture.
That was this suite's flake. The rank of every block, and the exact
SET inside it, are still pinned — what the story pins is that the
record carries the FULL retrieval (LOCKED A3), not the luck of a tie.
"""
docs = [d.strip() for d in recorded.split(",")]
expected = [d for group in groups for d in group]
assert len(docs) == len(expected), f"recorded {docs}, expected {expected}"
at = 0
for group in groups:
window = docs[at : at + len(group)]
assert sorted(window) == sorted(group), (
f"rank block {at + 1}..{at + len(group)}: got {window}, "
f"expected the set {sorted(group)}"
)
at += len(group)
def _assert_related_pair(links: object, titles: set[str]) -> None:
"""The two related-doc links carry EXACTLY these two documents, in
either order (the same tie the record pins as a group — see
:func:`_assert_ranked`)."""
found = {
links.nth(i).get_attribute("title") # pyright: ignore[reportAttributeAccessIssue]
for i in range(2)
}
assert found == titles, f"related pair is {found}, expected {titles}"
def test_single_source_question_shows_exactly_one_citation_chip(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
@@ -179,15 +219,18 @@ def test_single_source_question_shows_exactly_one_citation_chip(
# (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036);
# re-pinned against the post-phase walk (the record's CONTENT —
# suggested + related + read, deduped — is LOCKED A3, untouched).
assert row_log.sources == (
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/backups.md, "
"docs/homelab/kubernetes.md, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/networking/static-dns.json"
), row_log.sources
_assert_ranked(
row_log.sources,
[
["docs/homelab/ssh/ssh_aliases.txt"],
["docs/homelab/container_gitlab/gitlab-compose.yaml"],
["docs/homelab/scripts/uptime_probe.py"],
# The tied pair this record used to pin in an arbitrary order.
["docs/homelab/backups.md", "docs/homelab/kubernetes.md"],
["docs/homelab/quadlet/compose.container"],
["docs/homelab/networking/static-dns.json"],
],
)
def test_deflected_question_shows_zero_citation_chips(
@@ -225,8 +268,9 @@ def test_deflected_question_shows_zero_citation_chips(
# record, but a deflected turn cites nothing, so they never render
# as chips.
expect(links).to_have_count(2)
expect(links.nth(0)).to_contain_text("backups.md")
expect(links.nth(1)).to_contain_text("kubernetes.md")
# The pair, as a SET: the two are tied in the fused score (see
# :func:`_assert_ranked`), so only their presence is a contract.
_assert_related_pair(links, {"docs/homelab/backups.md", "docs/homelab/kubernetes.md"})
# ZERO citation chips under the bubble — the weak hits are scored
# docs, not citations (the phase-112/113 contract on the wire).
@@ -243,12 +287,16 @@ def test_deflected_question_shows_zero_citation_chips(
assert row_log.deflected is True
assert 0.0 < row_log.top_score < get_settings().relevance_threshold
assert row_log.fts_hits == 0
assert row_log.sources == (
"docs/deployments/new-service.md, "
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/quadlet/lan.network, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/backups.md, "
"docs/homelab/kubernetes.md"
), row_log.sources
_assert_ranked(
row_log.sources,
[
["docs/deployments/new-service.md"],
["docs/homelab/ssh/ssh_aliases.txt"],
["docs/homelab/quadlet/lan.network"],
["docs/homelab/scripts/uptime_probe.py"],
["docs/homelab/quadlet/compose.container"],
# The related row IS this pair, and the row's order is the same
# tie — pinned as a set below (``_assert_related_pair``).
["docs/homelab/backups.md", "docs/homelab/kubernetes.md"],
],
)
tests/e2e/test_steering.py
+7 -5
View File
@@ -48,11 +48,13 @@ NOTE = "STEEER-MARKER be concise"
XSS_NOTE = "<script>window.__xss = true; alert('xss')</script>"
#: The shell (index.html — served for BOTH / and /tuning.html since
#: phase 76 task 01, when the Tuning view folded into it) ships exactly
#: FIVE classic/module script tags: the phase-39 brand.js classic layer
#: + markdown.js + the chat module (app.js) + the shell router
#: (router.js, which lazy-imports the tuning.js view module) + the
#: phase-79 token gate (token-gate.js, phase 79 task 05).
BASE_SCRIPT_COUNT = 5
#: SIX classic/module script tags: the phase-39 brand.js classic layer
#: + markdown.js + the phase-182 verbatim.js hydrator (a classic script
#: loaded AFTER markdown.js, which is what put it in this count) + the
#: chat module (app.js) + the shell router (router.js, which
#: lazy-imports the tuning.js view module) + the phase-79 token gate
#: (token-gate.js, phase 79 task 05).
BASE_SCRIPT_COUNT = 6
async def _import_fixtures(mock_port: int) -> ImportSummary:
tests/e2e/test_suggestion_chips.py
+7 -3
View File
@@ -463,10 +463,14 @@ def test_chips_keyboard_accessible(
assert chip_text
# Tab from the page start: the first chip must be reachable on the
# keyboard, and before the composer input (skip-link + 2 nav links come
# first). Track tab stops until we land on a chip.
# keyboard, and before the composer input. What comes FIRST has grown
# with the nav itself — skip-link + the eight admin links (Chat, RAG,
# Sources, Tuning, History, Tokens, Theme, and the phase-168 Settings
# panel) + the header controls — so the walk allowance is the whole
# header, not the phase-05 two-link bar. The contract under test is the
# ORDER (a chip before the composer), pinned by the input check below.
reached_chip_at: int | None = None
for step in range(1, 11):
for step in range(1, 25):
page.keyboard.press("Tab")
state = page.evaluate(
"""() => {
tests/integration/test_import_fast_path_columns.py
+749
View File
@@ -0,0 +1,749 @@
"""Integration: phase 185 — what the UNCHANGED fast path SELECTs (the probe).
**VERDICT: CONFIRMED (task 01) → FIXED (phase 185 task 02,
``defer(Document.content)`` in ``app/rag/importer.py::_doc_row_stmt``).**
The fast path used to SELECT ``content`` for every
file it inspects. Each inspected file is looked up by
``app/rag/importer.py::_index_file`` (``select(Document).where(
Document.source == source, Document.path == rel)``), which is a WHOLE-ENTITY
load: its select list names ``documents.content`` — the full stored markdown,
**15,217** chars on average — next to the metadata the fast path actually
reads. The three sibling loads that share the phase-147 unchanged path (the
PDF branch, the PDF-embedded-image branch, the standalone-image branch)
compile to the same statement, so the whole KB walk pays this for EVERY file
it looks at, not only for the ones that changed. Nothing is released
afterwards: ``app/db.py:30`` builds every session with
``expire_on_commit=False``, so a row the run read — and then dirtied with the
``file_hash`` stamp (phase 184) — stays resident with its markdown until the
run ends.
**What the fix changed (measured by this file, post-fix).** The four fast-path
row loads now share ``_doc_row_stmt``, whose projection is every ``Document``
column EXCEPT ``content``, and the unchanged run issues **7** documents
SELECTs for the same four files: the **4** metadata-only row loads, **1**
primary-key follow-up (``SELECT documents.content AS documents_content …
WHERE documents.id = …``) — paid by the ONE summary-NULL row whose phase-118
backfill genuinely reads the stored markdown — and the same **2** per-source
post-passes. A file that already has a summary still costs exactly **1**
documents SELECT and zero follow-ups (``test_unchanged_file_is_one_select``).
**The incident (owner, 2026-10-15 — the same run phases 183/184 fixed).**
``documents`` held **17,407** rows at ``avg(length(content)) = 15,217``
characters = **252.6 MB of markdown**. The sync inspected ~27k files and
re-indexed **217**; the remaining ~26.8k took the fast path and still hydrated
their markdown to discover that nothing had changed. Phase 183 took ``content``
out of the prune's row (six light columns), phase 184 bounded what the walk
RETAINS (``session.expunge_all()`` every ``BOR_SYNC_SESSION_RESET_FILES``
files); this phase takes the markdown out of what the walk READS per unchanged
file, so a full re-sync's memory scales with row metadata instead of with the
corpus.
**Measured BY THIS FILE (2026-10-15, the dev Postgres; fixture: 4 markdown
files of ~8 KB seeded through the real ``import_sources``, then a SECOND
``import_sources`` over the same tree — the unchanged run — captured with
``before_cursor_execute``, the phase-183 task-01 technique).** PRE-FIX the
unchanged run issued **6** statements over ``documents``:
* **4** per-file ``(source, path)`` row loads — exactly one per inspected
file, the fast path's ONE lookup, every one of them naming
``documents.content`` (the load-bearing pin, flipped by task 02);
* **0** primary-key follow-up SELECTs over ``documents`` — nothing was
deferred yet, so nothing could be re-read; every file that already has a
summary costs exactly **1** documents SELECT in total;
* **2** per-source post-pass walks (``_enrich_pending_images`` +
``_backfill_image_contexts``, both ``WHERE documents.source = … AND
documents.is_image IS true``) — twice per RUN, independent of the file
count, and outside this phase's scope (D1 names exactly the four fast-path
row loads);
* the one staged row (``documents.summary = NULL``, its stale ``is_summary``
chunk stamped so replacement is observable) cost **1** documents SELECT
total — the row load, whose already-hydrated ``content`` the phase-118
backfill consumed directly. Post-task-02 that row is the ONLY one that pays
a second, single-column SELECT.
The captured fast-path statement, verbatim for one file **POST-FIX** — every
column of the PRE-FIX list below EXCEPT ``documents.content`` (which sat
between ``title`` and ``content_hash``), so no markdown travels with an
inspected row; the fast path reads ``file_hash``, then ``summary`` /
``created_at`` / ``page_count`` / ``path`` / ``full_path`` — never the text
this projection drops:
```
SELECT documents.id, documents.source, documents.path, documents.full_path,
documents.title, documents.content_hash, documents.file_hash,
documents.indexed_at, documents.created_at, documents.created_at_manual,
documents.summary, documents.is_image, documents.image_path,
documents.image_context, documents.page_count, documents.page_strip_mode
FROM documents
WHERE documents.source = %(source_1)s::VARCHAR AND documents.path =
%(path_1)s::VARCHAR
```
**The PRE-FIX shape** (task 01's evidence — the statement above WITH
``documents.content`` between ``title`` and ``content_hash``, which is what
put ~250 MB of markdown in front of a run that re-indexed 217 of ~27k files)
and **the follow-up a deferred row pays when it genuinely needs the
markdown** — the phase-118 summary-NULL backfill, ``_unchanged_doc`` step 3,
and nothing else on the fast path (measured 2026-10-15 by putting the same
row behind an ad-hoc ``defer(Document.content)`` load and then reading
``doc.content``)::
SELECT documents.content AS documents_content
FROM documents
WHERE documents.id = %(pk_1)s::UUID
That is the whole trade (phase 185 D2 / LOCKED A2): a doc WITH a stored
summary pays ZERO extra queries, a summary-NULL backfill pays ONE per-row
one-time SELECT, and deliberately NO ``raiseload``/``noload`` — those would
turn the backfill into a crash instead of a query.
**What this file does NOT touch:** the two per-source post-pass walks above,
``_prune``'s light row (phase 184), the identity-map cadence (phase 184),
``app/db.py:30``, ``iter_importable_files`` (phase 185 LOCKED A3) and the
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
"""
from __future__ import annotations
import asyncio
import logging
import re
from collections.abc import Iterator
from pathlib import Path
from typing import Any, NamedTuple
import pytest
from sqlalchemy import event, func, select, text
from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal, engine
from app.models import Chunk, Document
from app.rag.importer import ImportSummary, import_sources
from tests.fakes import FakeEmbedder
# ---------------------------------------------------------------------------
# The fixture: 4 markdown files of ~8 KB, one seed run + one unchanged run
# ---------------------------------------------------------------------------
#: The probe's own source name — nothing else in the KB uses it.
PROBE_SOURCE = "fastpath-probe"
#: Four files: enough that "one row load per file" is arithmetic rather than
#: a single sample, few enough that the seed run is instant.
FILE_COUNT = 4
#: The row whose stored ``summary`` is nulled before the second run — the
#: phase-118 backfill case, the ONE fast-path branch that genuinely needs the
#: stored markdown.
BACKFILL_PATH = "probe-02.md"
#: What the pre-existing ``is_summary`` chunk of that row is stamped with, so
#: "the backfill replaced the chunk" is observable instead of assumed (the
#: replacement is the only thing that can make the chunk read ``expected``).
STALE_SUMMARY = "STALE SUMMARY CHUNK — the backfill must replace this"
#: One repeated sentence, so every file has the same shape (and therefore the
#: same chunk count) as every other.
_SENTENCE = (
"The homelab runs podman quadlets on the hnb host with a static DNS "
"zone, nightly restic backups and a systemd timer that probes uptime. "
)
#: ~8 KB per file — production ``content`` averages 15.2 KB, so a whole-entity
#: row load is tens of kilobytes per inspected file and ``content`` is
#: unmistakably the fat column of the row (and stays under
#: ``BOR_SUMMARY_MAX_CHARS``, so the backfill's summary call is uncapped).
CONTENT_CHARS_PER_FILE = 8_000
#: The documents SELECTs of the two per-source post-passes, measured
#: 2026-10-15 — asserted so a future per-file extra SELECT cannot hide among
#: them (and so a post-pass that stopped querying is noticed too).
POST_PASS_DOC_SELECTS = 2
#: **0 pre-phase-185** — the whole-entity row load already carries
#: ``content``, so the phase-118 backfill needs no follow-up query. **1
#: **0 pre-phase-185** — the whole-entity row load already carried
#: ``content``, so the phase-118 backfill needed no follow-up query. **1
#: post-phase-185 (the value now)** — the row load defers ``content``, so the
#: ONE backfill pays exactly ONE lazy ``SELECT documents.content … WHERE
#: documents.id = …``. The arithmetic: the backfilled file costs ``1 row load
#: + EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL`` documents SELECTs, every other
#: unchanged file costs exactly 1 (``test_unchanged_file_is_one_select``).
#: Everything else about the backfill is identical across the flip: the stored
#: summary, its ``is_summary`` chunk, the run's counts and its log words
#: (``test_unchanged_run_keeps_every_row_intact``).
EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 1
class Captured(NamedTuple):
"""One executed statement plus its bind parameters.
The parameters make the probe attributable: the fast-path row load carries
``path_1`` (the file it looked up) while the deferred column's lazy
follow-up carries only the primary key — so "one SELECT per unchanged
file" and "one follow-up for the backfill" are two different, separately
asserted claims.
"""
text: str
params: dict[str, Any]
class UnchangedRun(NamedTuple):
"""The unchanged run, snapshotted at the moment it ENDED.
``captured`` is a copy: the listener stays installed for the rest of the
test, so the assertions must not race the probe's own read queries.
"""
captured: list[Captured]
summary: ImportSummary
llm: FakeEmbedder
@property
def lite_prompts(self) -> list[str]:
"""The user-message text of every ``lite`` call the run made."""
return [
str(message.get("content"))
for call in self.llm.chat_calls
for message in call
if message.get("role") == "user"
]
def _file_body(n: int) -> str:
"""One ~8 KB markdown document, carrying a per-file sentinel per section."""
parts = [f"# Probe document {n}\n"]
section = 0
while sum(len(p) for p in parts) < CONTENT_CHARS_PER_FILE:
parts.append(f"\n## Section {section} of probe {n} (fastpath-sentinel-{n})\n\n")
parts.extend(_SENTENCE for _ in range(4))
parts.append("\n")
section += 1
return "".join(parts)
def _seed_files(tmp: Path) -> Path:
"""``FILE_COUNT`` markdown files under a fresh source root; returns it."""
root = tmp / PROBE_SOURCE
root.mkdir(parents=True, exist_ok=True)
for n in range(FILE_COUNT):
body = _file_body(n)
assert 8_000 <= len(body) <= 9_000, len(body)
(root / f"probe-{n:02d}.md").write_text(body)
return root
def _probe_llm() -> FakeEmbedder:
"""The deterministic fake embedder/summarizer on the SHIPPED defaults
(``sync_session_reset_files=200`` — no identity-map release lands inside a
four-file walk, so nothing but the row loads is in play here)."""
llm = FakeEmbedder()
llm.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
return llm
def _run_import(root: Path, llm: FakeEmbedder) -> ImportSummary:
"""One real ``import_sources`` pass over *root*, in its own session."""
session = SessionLocal()
try:
return asyncio.run(
import_sources([root], llm, prune=False, session=session)
)
finally:
session.close()
def _doc_id(db: Session, path: str) -> Any:
"""The id of one probe row (lazy follow-ups are keyed by primary key)."""
doc_id = db.scalar(
select(Document.id).where(
Document.source == PROBE_SOURCE, Document.path == path
)
)
assert doc_id is not None
return doc_id
def _stage_the_backfill(db: Session, path: str) -> None:
"""The phase-118 state: a row whose stored ``summary`` is NULL while its
content and ``file_hash`` are current (the direct UPDATE of
``tests/integration/test_importer_dates.py`` — exactly what a pre-phase-118
import left behind).
The stale ``is_summary`` chunk is stamped rather than deleted: after the
run a chunk carrying the backfill text can only have got there through
:func:`app.rag.importer._store_summary`'s replacement rule, which is what
makes "the backfill ran" a fact rather than an assumption.
"""
doc_id = _doc_id(db, path)
db.execute(
text(
"UPDATE documents SET summary = NULL WHERE source = :s AND path = :p"
),
{"s": PROBE_SOURCE, "p": path},
)
db.execute(
text("UPDATE chunks SET content = :stale WHERE document_id = :d AND is_summary"),
{"stale": STALE_SUMMARY, "d": doc_id},
)
db.commit()
db.expunge_all()
def _two_runs(db: Session, tmp_path: Path, captured: list[Captured]) -> UnchangedRun:
"""Seed the KB, then run the UNCHANGED pass with the capture active.
The seed run's SQL is cleared away at the boundary: this file pins what
the unchanged walk costs, not what first indexing costs.
"""
root = _seed_files(tmp_path)
first = _run_import(root, _probe_llm())
assert (first.files, first.added, first.unchanged, first.errors) == (
FILE_COUNT,
FILE_COUNT,
0,
0,
)
_stage_the_backfill(db, BACKFILL_PATH)
captured.clear()
llm = _probe_llm()
second = _run_import(root, llm)
run = UnchangedRun(list(captured), second, llm)
assert (second.files, second.added, second.unchanged, second.errors) == (
FILE_COUNT,
0,
FILE_COUNT,
0,
)
assert second.chunks == 0 # the fast path re-embeds no content
assert second.summary_backfilled == 1 # the staged row was backfilled
return run
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(autouse=True)
def clean_kb(db: Session) -> Iterator[None]:
"""``documents`` / ``chunks`` are global state: reset around every test
(the ``test_sync_api.py`` truncate pattern) and drop whatever the session
cached, so the next read is the database's truth."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
db.expunge_all()
yield
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
db.expunge_all()
@pytest.fixture()
def statement_capture() -> Iterator[list[Captured]]:
"""Every SQL the engine executes for the duration of ONE test.
The ``before_cursor_execute`` technique of phase 183 task 01
(``tests/integration/test_prune_cascade_statements.py``): the column list
a SELECT uses is invisible in the ORM's Python API and plain to see in the
statement it compiles, so a capture is the only deterministic way to pin
what the fast path's row actually contains. Installed on the engine and
REMOVED in teardown, so no other test of the session ever sees the
listener (verified by the full ``uv run pytest`` run being green with this
file in it).
"""
captured: list[Captured] = []
def _listener(
conn: object, # noqa: ARG001
cursor: object, # noqa: ARG001
statement: str,
parameters: object,
context: object, # noqa: ARG001
executemany: bool, # noqa: ARG001
) -> None:
params = dict(parameters) if isinstance(parameters, dict) else {}
captured.append(Captured(statement, params))
event.listen(engine, "before_cursor_execute", _listener)
try:
yield captured
finally:
event.remove(engine, "before_cursor_execute", _listener)
# ---------------------------------------------------------------------------
# Helpers — reading a captured statement
# ---------------------------------------------------------------------------
#: ``… FROM documents`` — case-insensitive, word-bounded.
_FROM_DOCUMENTS_RE = re.compile(r"\bfrom\s+documents\b", re.IGNORECASE)
#: Where the projection ends: SQLAlchemy compiles ``FROM`` onto its own line,
#: and everything after it is a predicate, not a column the row hydrates.
_FROM_RE = re.compile(r"\bFROM\b")
#: The ``content`` COLUMN TOKEN, aliased or plain. The trailing ``\\b`` is what
#: keeps the neighbours out: ``_`` is a WORD character, so the ``documents.content``
#: inside ``documents.content_hash`` has no boundary after the ``t`` and never
#: matches. That is why this is a token regex and not a substring test —
#: ``content_hash`` is in EVERY one of these select lists, and a substring
#: match would report a content load for every row of every run, pre- and
#: post-fix alike.
_CONTENT_TOKEN_RE = re.compile(r"\bdocuments\.content\b")
def doc_selects(captured: list[Captured]) -> list[Captured]:
"""The captured statements that SELECT the ``documents`` table."""
return [
c
for c in captured
if c.text.lstrip().lower().startswith("select")
and _FROM_DOCUMENTS_RE.search(c.text)
]
def loads_content(statement: Captured | str) -> bool:
"""``True`` when the statement's SELECT LIST names the ``content`` column.
Only the projection is inspected (see :data:`_FROM_RE`), and only the
column token matches (see :data:`_CONTENT_TOKEN_RE`).
"""
text = statement.text if isinstance(statement, Captured) else statement
from_at = _FROM_RE.search(text)
projection = text[: from_at.start()] if from_at else text
return _CONTENT_TOKEN_RE.search(projection) is not None
def test_the_content_matcher_names_the_column_and_nothing_else() -> None:
"""The matcher's own pin — every other assertion here rests on it.
``SELECT … documents.content, documents.content_hash …`` is a content load
because of the FIRST token; ``documents.content_hash`` alone is not. If
this ever stops holding, the pre-/post-fix assertion above is meaningless
in BOTH directions, so it is asserted directly rather than trusted.
"""
assert loads_content(
"SELECT documents.id, documents.content, documents.content_hash\n"
"FROM documents\nWHERE documents.id = %(pk_1)s"
)
assert loads_content(
"SELECT documents.content AS documents_content\nFROM documents\n"
"WHERE documents.id = %(pk_1)s"
)
assert not loads_content(
"SELECT documents.id, documents.content_hash, documents.file_hash\n"
"FROM documents\nWHERE documents.source = %(source_1)s"
)
assert not loads_content(
"SELECT documents.summary, documents.is_image, documents.page_count\n"
"FROM documents\nWHERE documents.id = %(pk_1)s"
)
# A predicate mention is not a hydration.
assert not loads_content(
"SELECT documents.id\nFROM documents\nWHERE documents.content = %(c_1)s"
)
def _predicate(statement: str) -> str:
"""Everything after the first ``FROM`` — the statement's predicate."""
from_at = _FROM_RE.search(statement)
return statement[from_at.end():] if from_at else ""
def row_loads(captured: list[Captured]) -> list[Captured]:
"""The ``(source, path)`` row loads — the fast path's ONE lookup per file.
Matched on the PREDICATE, not the projection: the four sites of phase 185
D1 (``_index_file`` plus the PDF / PDF-embedded-image / image branch loads)
all compile to ``WHERE documents.source = … AND documents.path = …``. The
two per-source post-pass walks filter on ``documents.is_image`` instead
and are therefore excluded here — they are not the fast path's row load.
"""
return [
c
for c in doc_selects(captured)
if "documents.source" in _predicate(c.text)
and "documents.path" in _predicate(c.text)
]
def pk_followups(captured: list[Captured]) -> list[Captured]:
"""Documents SELECTs keyed by PRIMARY KEY — the deferred-column shape.
A ``defer``-red column loads as ``SELECT documents.content … WHERE
documents.id = …``: no ``source``/``path`` predicate at all. Pre-fix this
list is EMPTY (the row was loaded whole, so nothing is deferred); post-fix
it holds the ONE query the phase-118 backfill pays.
"""
return [
c
for c in doc_selects(captured)
if "documents.source" not in _predicate(c.text)
and "documents.path" not in _predicate(c.text)
]
def _summary_chunks(db: Session, path: str) -> list[Chunk]:
"""The ``is_summary`` chunks of one document, read by ``document_id`` so
the parent row's ``content`` is never what this probe reads."""
return list(
db.scalars(
select(Chunk)
.where(
Chunk.document_id == _doc_id(db, path),
Chunk.is_summary.is_(True),
)
.order_by(Chunk.id)
)
)
# ---------------------------------------------------------------------------
# The probe
# ---------------------------------------------------------------------------
def test_unchanged_run_defers_content_post_fix(
db: Session, tmp_path: Path, statement_capture: list[Captured]
) -> None:
"""THE PROBE (post-fix): no file the unchanged walk inspects SELECTs its
markdown.
Four unchanged files → four ``(source, path)`` row loads, and NONE of them
names ``documents.content`` in its projection — the fast path reads row
metadata only (phase 185, ``_doc_row_stmt``). The only follow-up the run
may pay is the backfill row's, attributed per primary key in
:func:`test_summary_null_backfill_reads_stored_content`, while
:func:`test_unchanged_file_is_one_select` pins that every
summary-bearing file pays none. The rest of the run's documents traffic
is the two per-source image post-passes, pinned apart so a future
per-file extra SELECT cannot hide inside the total.
(Task 01's PRE-fix assertion was the inverse — ``all(loads_content(c))``
with zero follow-ups — and flipped here exactly as planned. A deferred
column that is genuinely READ still shows up, as a PK follow-up, so
neither a re-appeared ``content`` nor a new lazy reader hides here.)
"""
run = _two_runs(db, tmp_path, statement_capture)
selects = doc_selects(run.captured)
loads = row_loads(run.captured)
assert len(loads) == FILE_COUNT, (
f"a {FILE_COUNT}-file unchanged walk issued {len(loads)} (source, "
"path) row loads — the fast path looks each file up exactly once"
)
assert len(selects) >= FILE_COUNT
# Post-fix (phase 185 task 02): no content in the fast-path SELECT.
assert not any(loads_content(c) for c in loads), (
"a fast-path row load SELECTs documents.content again — the deferral "
"in _doc_row_stmt is gone or bypassed, which re-hydrates the corpus "
"(252.6 MB on the incident box) for a walk that reads metadata only"
)
followups = pk_followups(run.captured)
assert len(followups) == EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL, (
f"an unchanged run over {FILE_COUNT} files (one with summary IS NULL) "
f"paid {len(followups)} primary-key follow-up SELECTs over documents "
f"while EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL is "
f"{EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL} — the backfill's single "
"deferred-column read is the only one the fast path may need"
)
post_passes = [c for c in selects if c not in loads and c not in followups]
assert len(post_passes) == POST_PASS_DOC_SELECTS, (
f"the unchanged run did {len(post_passes)} documents SELECTs beyond "
f"the {FILE_COUNT} row loads — expected the {POST_PASS_DOC_SELECTS} "
"per-source image post-passes and nothing else"
)
def test_unchanged_file_is_one_select(
db: Session, tmp_path: Path, statement_capture: list[Captured]
) -> None:
"""One documents SELECT per unchanged file — MUST STAY GREEN POST-FIX.
A file WITH a stored summary reads nothing beyond what its row load
already carries: the ``file_hash`` comparison, then ``_unchanged_doc``'s
``summary`` / ``created_at`` / ``page_count`` bookkeeping. Pre-fix that is
one whole-entity load; post-fix it must still be exactly ONE — this is the
guard that ``defer(Document.content)`` does not decay into a lazy-load
storm (``summary``, then ``created_at``, then ``page_count`` … one SELECT
apiece), which would trade 15 KB of residency for five round trips.
"""
run = _two_runs(db, tmp_path, statement_capture)
for n in range(FILE_COUNT):
path = f"probe-{n:02d}.md"
if path == BACKFILL_PATH:
continue # asserted separately, with its documented extra
one_file = [c for c in row_loads(run.captured) if c.params.get("path_1") == path]
assert one_file, (
f"{path} was walked with no (source, path) row load carrying "
"path_1 — the row-load statement shape changed, so this probe is "
"attributing nothing"
)
assert len(one_file) == 1, (
f"{path} cost {len(one_file)} documents row loads — the fast path "
"looks a file up ONCE"
)
doc_id = str(_doc_id(db, path))
followups = [
c for c in pk_followups(run.captured) if str(c.params.get("pk_1")) == doc_id
]
assert followups == [], (
f"{path} paid {len(followups)} deferred-column follow-up SELECT(s) "
"despite having a stored summary — nothing on its fast path may "
"read content, so a follow-up means a new reader crept in"
)
def test_summary_null_backfill_reads_stored_content(
db: Session, tmp_path: Path, statement_capture: list[Captured]
) -> None:
"""The ONE branch that genuinely needs the markdown: ``summary IS NULL``.
``_unchanged_doc`` step 3 (phase 118, A2) backfills the summary FROM THE
STORED ``doc.content`` — never by re-reading the file, re-extracting or
re-describing. So the probe pins the behaviour AND its cost:
* the run backfills exactly one summary (``summary_backfilled=1`` while
``summaries`` stays ``0`` — a backfill is not a KB-change signal), the
stored summary comes back and its single ``is_summary`` chunk is the
REPLACED one (the pre-existing chunk still carried :data:`STALE_SUMMARY`
when the run started, and replacement is the only way it changes);
* the text the ``lite`` call was given IS the stored markdown (~8 KB, with
the file's own sentinel in every section) — the bytes travelled from
``documents.content``, not from a re-read of the file;
* the documents SELECT count for that file is ``1 +
EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL``: the row load plus, once task
02 defers ``content``, exactly one lazy follow-up keyed by primary key
(D2 — a per-row, one-time, already-rare cost, accepted precisely instead
of a ``raiseload`` that would crash this branch where it should query).
"""
run = _two_runs(db, tmp_path, statement_capture)
n = int(BACKFILL_PATH.removeprefix("probe-").removesuffix(".md"))
expected = f"Summary of #\nSource: {PROBE_SOURCE}/{BACKFILL_PATH}"
# The backfill happened, from the stored content.
assert run.summary.summary_backfilled == 1
assert run.summary.summaries == 0
stored = db.scalar(
select(Document.content).where(
Document.source == PROBE_SOURCE, Document.path == BACKFILL_PATH
)
)
assert stored is not None and len(stored) >= 8_000
assert f"fastpath-sentinel-{n}" in stored
doc = db.scalar(
select(Document).where(
Document.source == PROBE_SOURCE, Document.path == BACKFILL_PATH
)
)
assert doc is not None
assert doc.summary == expected
assert doc.file_hash is not None
chunks = _summary_chunks(db, BACKFILL_PATH)
assert [c.content for c in chunks] == [expected], (
"the summary chunk was not replaced by the backfill — a stale chunk "
"survived, or more than one is_summary chunk exists"
)
assert chunks[0].position == -1
assert chunks[0].embedding is not None
# The lite call got the STORED markdown, sentinel and all — uncapped.
assert run.lite_prompts, "the backfill made no lite-model call"
assert any(stored == prompt for prompt in run.lite_prompts), (
"the summary model was not fed documents.content verbatim — the "
"phase-118 backfill reads something other than the stored markdown"
)
# And the cost of that branch, as a documented constant.
doc_id = str(_doc_id(db, BACKFILL_PATH))
loads = [c for c in row_loads(run.captured) if c.params.get("path_1") == BACKFILL_PATH]
followups = [c for c in pk_followups(run.captured) if str(c.params.get("pk_1")) == doc_id]
assert len(loads) == 1
assert len(followups) == EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL, (
f"the summary-NULL backfill paid {len(followups)} lazy content "
f"SELECT(s) while EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL is "
f"{EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL} — 0 while the row load "
"carries content (pre task 02), 1 once content is deferred"
)
assert all(loads_content(c) for c in followups)
def test_unchanged_run_keeps_every_row_intact(
db: Session, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The observable contract the deferral must not move (phase 185 D4).
The reference for task 02's byte-identity claim: the unchanged run's whole
``ImportSummary`` shape and the whole KB after it — four unchanged files,
one backfill, no re-embed, no date refresh, every file still holding its
full ~8 KB of content, its summary and all of its chunks. ``changed_paths``
stays empty (the backfill is not a document change, so the folder-summary
regeneration never sees it) and no source is pruned.
"""
root = _seed_files(tmp_path)
first = _run_import(root, _probe_llm())
assert (first.files, first.added, first.summary_backfilled) == (
FILE_COUNT,
FILE_COUNT,
0,
)
content_chunks_after_seed = db.scalar(
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(False))
)
assert content_chunks_after_seed is not None and content_chunks_after_seed > FILE_COUNT
_stage_the_backfill(db, BACKFILL_PATH)
caplog.clear()
with caplog.at_level(logging.INFO, logger="app.importer"):
second = _run_import(root, _probe_llm())
messages = [r.getMessage() for r in caplog.records if r.name == "app.importer"]
# The phase-147/118 log words, verbatim (D4: task 02 may change WHICH
# COLUMNS travel, never what the run says it did). ``import: summary
# source=`` is the FRESH-summary line — it must not appear on an unchanged
# run, where only the backfill's own wording is allowed.
assert sum(m.startswith("import: unchanged ") for m in messages) == FILE_COUNT
assert sum(m.startswith("import: summary-backfill ") for m in messages) == 1
assert not [m for m in messages if m.startswith("import: summary source=")]
assert not [m for m in messages if m.startswith("import: date-refreshed ")]
assert not [m for m in messages if m.startswith("import: added ")]
assert not [m for m in messages if m.startswith("import: updated ")]
assert (second.files, second.added, second.updated) == (FILE_COUNT, 0, 0)
assert (second.unchanged, second.pruned, second.errors) == (FILE_COUNT, 0, 0)
assert second.summary_backfilled == 1
assert second.summaries == 0
assert second.summary_errors == 0
assert second.dates_updated == 0
assert second.chunks == 0
assert second.changed_paths == []
assert second.formats == {"md": FILE_COUNT}
assert second.format_counts() == "md:4"
# Nothing was re-embedded, nothing was lost.
assert db.scalar(
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(False))
) == content_chunks_after_seed
assert db.scalar(
select(func.count())
.select_from(Document)
.where(Document.source == PROBE_SOURCE, Document.summary.is_not(None))
) == FILE_COUNT
assert all(
len(c) >= 8_000
for c in db.scalars(
select(Document.content).where(Document.source == PROBE_SOURCE)
)
)
# Exactly one summary chunk per document (the backfill replaced, not added).
assert db.scalar(
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(True))
) == FILE_COUNT
tests/integration/test_import_session_retention.py
+210 -88
View File
@@ -136,6 +136,7 @@ lazy SELECT per row and fail that test, not get papered over.
``scripts/git_sync.py`` (phase 186), ``app/db.py:30``, and the
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
"""
from __future__ import annotations
import asyncio
@@ -189,9 +190,7 @@ EXPECTED_SUMMARY_CHUNKS = 12
#: Every ORM object a full ADD run creates: the 12 documents, their content
#: chunks and their summary chunks. What the identity map would have to
#: hold if the ADDED path retained what the phase-184 baseline describes.
OBJECTS_CREATED_BY_ADD_RUN = (
FILE_COUNT + EXPECTED_CONTENT_CHUNKS + EXPECTED_SUMMARY_CHUNKS
)
OBJECTS_CREATED_BY_ADD_RUN = FILE_COUNT + EXPECTED_CONTENT_CHUNKS + EXPECTED_SUMMARY_CHUNKS
#: The measured number of identity-map entries at the end of an UNCHANGED
#: run with the cadence OFF — one per file walked, each a full ``Document``
@@ -322,9 +321,7 @@ def _kb_snapshot(db: Session) -> dict[str, tuple[str, str | None, str, int]]:
}
snapshot: dict[str, tuple[str, str | None, str, int]] = {}
for doc in db.scalars(
select(Document)
.where(Document.source == PROBE_SOURCE)
.order_by(Document.path)
select(Document).where(Document.source == PROBE_SOURCE).order_by(Document.path)
):
assert doc.file_hash is not None # the phase-147 stamp is always set
snapshot[doc.path] = (
@@ -373,9 +370,7 @@ def test_the_added_run_holds_its_objects_only_until_a_collection(
root = _seed_files(tmp_path)
session = SessionLocal()
try:
summary = asyncio.run(
import_sources([root], _probe_llm(), prune=False, session=session)
)
summary = asyncio.run(import_sources([root], _probe_llm(), prune=False, session=session))
assert (summary.files, summary.added, summary.unchanged) == (
FILE_COUNT,
FILE_COUNT,
@@ -405,9 +400,7 @@ def test_the_added_run_holds_its_objects_only_until_a_collection(
session.close()
def test_the_unchanged_run_keeps_every_document_it_walks(
db: Session, tmp_path: Path
) -> None:
def test_the_unchanged_run_keeps_every_document_it_walks(db: Session, tmp_path: Path) -> None:
"""THE PROBE: a repeat sync holds one full ``Document`` per file walked.
The production-shaped case — the second sync of the same source, every
@@ -446,9 +439,7 @@ def test_the_unchanged_run_keeps_every_document_it_walks(
staircase.append(_map_size(session))
summary = asyncio.run(
import_sources(
[root], _probe_llm(), prune=False, session=session, progress=progress
)
import_sources([root], _probe_llm(), prune=False, session=session, progress=progress)
)
assert (summary.files, summary.added, summary.unchanged) == (
FILE_COUNT,
@@ -459,8 +450,7 @@ def test_the_unchanged_run_keeps_every_document_it_walks(
# One row resident per file already walked — a staircase, never a
# plateau and never a release.
assert staircase == list(range(FILE_COUNT)), (
"expected exactly one document retained per file already walked,"
f" got {staircase}"
f"expected exactly one document retained per file already walked, got {staircase}"
)
assert _map_size(session) == MEASURED_UNCHANGED_MAP_SIZE
@@ -643,8 +633,7 @@ def test_reset_cadence_bounds_the_unchanged_walk(db: Session, tmp_path: Path) ->
FILE_COUNT,
)
assert staircase == [n % RESET_FILES for n in range(FILE_COUNT)], (
f"the bounded walk sampled {staircase} — expected a release every"
f" {RESET_FILES} files"
f"the bounded walk sampled {staircase} — expected a release every {RESET_FILES} files"
)
assert max(staircase) < MEASURED_UNCHANGED_MAP_SIZE
assert _map_size(session) == 0, (
@@ -682,14 +671,10 @@ def test_reset_cadence_survives_an_embedding_error(db: Session, tmp_path: Path)
session = SessionLocal()
try:
summary = asyncio.run(
import_sources([root], llm, prune=False, session=session)
)
summary = asyncio.run(import_sources([root], llm, prune=False, session=session))
assert (summary.files, summary.errors) == (FILE_COUNT, 1)
assert summary.added == FILE_COUNT - 1
assert summary.chunks == EXPECTED_CONTENT_CHUNKS - (
EXPECTED_CONTENT_CHUNKS // FILE_COUNT
)
assert summary.chunks == EXPECTED_CONTENT_CHUNKS - (EXPECTED_CONTENT_CHUNKS // FILE_COUNT)
assert summary.summaries == EXPECTED_SUMMARY_CHUNKS - 1
assert _map_size(session) <= MAP_BOUND
finally:
@@ -743,9 +728,7 @@ def test_a_clean_row_leaves_the_map_and_a_dirty_one_never_does(db: Session) -> N
assert _map_size(session) == 0, "a clean committed row must be weakly held"
row = session.scalar(
select(Document).where(
Document.source == PROBE_SOURCE, Document.path == "mechanism.md"
)
select(Document).where(Document.source == PROBE_SOURCE, Document.path == "mechanism.md")
)
assert row is not None
row.file_hash = row.file_hash # the stamp: same value, dirty state
@@ -765,9 +748,7 @@ def test_a_clean_row_leaves_the_map_and_a_dirty_one_never_does(db: Session) -> N
assert _map_size(session) == 0
assert (
db.scalar(
select(func.count())
.select_from(Document)
.where(Document.source == PROBE_SOURCE)
select(func.count()).select_from(Document).where(Document.source == PROBE_SOURCE)
)
== 1
)
@@ -801,9 +782,7 @@ PRUNE_BATCH = 10
#: The SIX columns the prune loads — the ``load_only`` set of ``_prune``
#: (docstring: the light-row contract). An eighth name here means the walk
#: got heavier; one missing means a guard lost the column it reads.
PRUNE_COLUMNS = frozenset(
{"id", "source", "path", "is_image", "image_path", "page_count"}
)
PRUNE_COLUMNS = frozenset({"id", "source", "path", "is_image", "image_path", "page_count"})
#: ``… FROM documents`` — the prune's own walk AND (that is the pin) any
#: per-document follow-up lazy load of a deferred column. The word boundary
@@ -862,9 +841,7 @@ def _projected_columns(statement: str) -> frozenset[str]:
return frozenset(re.findall(r"documents\.\w+", head))
def _seed_prune_docs(
session: Session, *, page_dir: Path | None = None
) -> list[UUID]:
def _seed_prune_docs(session: Session, *, page_dir: Path | None = None) -> list[UUID]:
"""``PRUNE_DOC_COUNT`` indexed docs of :data:`PRUNE_SOURCE`, each with
``PRUNE_CHUNKS_PER_DOC`` embedded chunks and a full-fat ``content``.
@@ -924,9 +901,7 @@ def _arm_commit_counter(session: Session) -> list[int]:
return commits
def test_prune_holds_no_content(
db: Session, tmp_path: Path, statement_capture: list[str]
) -> None:
def test_prune_holds_no_content(db: Session, tmp_path: Path, statement_capture: list[str]) -> None:
"""THE PRUNE PROBE: the walk loads SIX columns and never ``content``.
30 documents × 5 chunks are pruned through the importer's own ``_prune``
@@ -971,10 +946,7 @@ def test_prune_holds_no_content(
"prune — a deferred column is being read per row, so the light row "
"is incomplete (add the column to load_only)"
)
projected = {
col.removeprefix("documents.")
for col in _projected_columns(selects[0])
}
projected = {col.removeprefix("documents.") for col in _projected_columns(selects[0])}
assert projected == PRUNE_COLUMNS, (
f"the prune's row is not the six-column light row: {sorted(projected)}"
)
@@ -985,12 +957,11 @@ def test_prune_holds_no_content(
# Everything really went, children included (no orphans — phase 183 D4,
# asserted here too so a light row cannot quietly change it).
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
) == 0
assert db.scalar(
select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids))
) == 0
assert (
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
== 0
)
assert db.scalar(select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids))) == 0
assert sorted(summary.changed_paths) == sorted(
(PRUNE_SOURCE, f"docs/doc-{n:04d}.md") for n in range(PRUNE_DOC_COUNT)
)
@@ -1001,9 +972,7 @@ def test_prune_holds_no_content(
assert _map_classes(db) == {}
def test_prune_without_sources_touches_nothing(
db: Session, statement_capture: list[str]
) -> None:
def test_prune_without_sources_touches_nothing(db: Session, statement_capture: list[str]) -> None:
"""The empty-source guard is a real short-circuit, not a walk for naught.
``_prune`` with no source names returns ``0`` before the SELECT — so a
@@ -1017,9 +986,10 @@ def test_prune_without_sources_touches_nothing(
assert _prune(db, set(), set(), summary=summary, prune_batch=10) == 0
assert statement_capture == []
assert summary.changed_paths == []
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
) == PRUNE_DOC_COUNT
assert (
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
== PRUNE_DOC_COUNT
)
def test_prune_batch_commits(db: Session) -> None:
@@ -1059,11 +1029,10 @@ def test_prune_batch_commits(db: Session) -> None:
session.close()
# The FK cascade did the child work in EVERY transaction the
# prune opened, not just the last one.
assert db.scalar(
select(func.count())
.select_from(Chunk)
.where(Chunk.document_id.in_(ids))
) == 0
assert (
db.scalar(select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids)))
== 0
)
pruned, summary, commits = run(PRUNE_BATCH)
assert pruned == PRUNE_DOC_COUNT
@@ -1074,9 +1043,10 @@ def test_prune_batch_commits(db: Session) -> None:
assert sorted(summary.changed_paths) == sorted(
(PRUNE_SOURCE, f"docs/doc-{n:04d}.md") for n in range(PRUNE_DOC_COUNT)
)
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
) == 0
assert (
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
== 0
)
# …and the pre-phase run does the SAME work in ONE transaction.
_truncate(db)
@@ -1087,9 +1057,10 @@ def test_prune_batch_commits(db: Session) -> None:
"reproduce the pre-phase single transaction exactly"
)
assert sorted(summary_0.changed_paths) == sorted(summary.changed_paths)
assert db.scalar(
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
) == 0
assert (
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
== 0
)
def test_prune_batch_setting_reaches_the_prune(
@@ -1137,9 +1108,7 @@ def test_prune_batch_setting_reaches_the_prune(
monkeypatch.setattr(importer, "_prune", spy)
second = asyncio.run(import_sources([root], llm, prune=True, session=db))
assert seen_kwargs == [3], (
"import_sources did not pass Settings.sync_prune_batch to _prune"
)
assert seen_kwargs == [3], "import_sources did not pass Settings.sync_prune_batch to _prune"
assert (second.files, second.pruned, second.errors) == (1, 3, 0)
assert sorted(path for (_src, path) in second.changed_paths) == [
"keep-0.md",
@@ -1152,16 +1121,99 @@ def test_prune_batch_setting_reaches_the_prune(
select(func.count()).select_from(Chunk).where(Chunk.embedding.is_not(None))
)
assert embedded is not None and embedded > 0
assert db.scalar(
select(func.count())
.select_from(Chunk)
.where(Chunk.document_id.not_in(select(Document.id)))
) == 0
assert (
db.scalar(
select(func.count())
.select_from(Chunk)
.where(Chunk.document_id.not_in(select(Document.id)))
)
== 0
)
def test_prune_guards_unaffected_with_light_rows(
db: Session, tmp_path: Path
) -> None:
def test_the_summary_backfill_survives_the_reset_cadence(db: Session, tmp_path: Path) -> None:
"""The constraint that fixes WHERE the cadence may sit, in ONE file.
``_unchanged_doc`` (phase 118) backfills a NULL summary by calling
:func:`app.rag.summarizer.summarize_document`, which reads
``doc.content`` AFTER the row was loaded and its ``file_hash`` stamped.
That local is live across statements: a
``session.expunge_all()`` moved into ``_unchanged_doc`` — or a
``noload``/``raiseload`` added to the row it walks — would discard or
break the very row the backfill reads. Phase 185 task 01 chose
``defer`` over ``noload``/``raiseload`` (LOCKED A2) for the same reason,
and task 05 of THIS phase pinned the choice on the statement itself
(``tests/unit/test_importer_defer_content.py``); this pins the SAME
constraint on the run.
The fixture is a document whose ``is_summary`` chunk is missing while
its ``file_hash`` matches (a re-sync with a NULL summary — exactly the
row the backfill exists for), imported at the tightest cadence the
setting allows that still resets inside a 12-file walk. Three
observables, none of them a memory proxy:
* the backfill RUNS — one and only one ``is_summary`` chunk appears,
carrying the fake-embedder's summary text;
* every other file is still ``unchanged`` and the summary count is
exactly the file count afterwards, i.e. the reset cadence destroyed
no summary on the way through;
* the doc's own text is still intact afterwards (``content`` was never
rewritten by a re-index).
"""
root = _seed_files(tmp_path)
seed = SessionLocal()
try:
first = asyncio.run(import_sources([root], _probe_llm(), session=seed))
assert first.added == FILE_COUNT
finally:
seed.close()
# Hand the next sync ONE document with a NULL summary: it takes
# ``_unchanged_doc``'s backfill branch instead of the early return.
probe_rel = "probe-00.md"
before = _kb_snapshot(db)[probe_rel]
target = db.execute(
select(Document).where(Document.source == PROBE_SOURCE, Document.path == probe_rel)
).scalar_one()
db.query(Chunk).filter(Chunk.document_id == target.id, Chunk.is_summary.is_(True)).delete()
target.summary = None
db.commit()
db.expunge_all()
assert _kb_snapshot(db)[probe_rel][1] is None
session = SessionLocal()
try:
summary = asyncio.run(
import_sources(
[root], _probe_llm(reset_files=RESET_FILES), prune=False, session=session
)
)
finally:
session.close()
# Everything stayed unchanged — the backfill never re-indexed.
assert (summary.files, summary.added, summary.updated, summary.unchanged) == (
FILE_COUNT,
0,
0,
FILE_COUNT,
)
assert (summary.errors, summary.summary_backfilled) == (0, 1)
snap = _kb_snapshot(db)
backfilled = snap[probe_rel]
assert backfilled[1] is not None, "the summary-NULL backfill did not run"
# The one doc that needed the deferred read still has its OWN text and
# its own hash: a summary was added, the document was not rewritten.
assert backfilled[0] == before[0]
assert backfilled[2] == before[2]
# And the cadence destroyed nothing on its way through: every file is
# back to 5 content chunks + 1 summary chunk.
assert {p: c for p, (_, _, _, c) in snap.items()} == {
f"probe-{n:02d}.md": 6 for n in range(FILE_COUNT)
}
def test_prune_guards_unaffected_with_light_rows(db: Session, tmp_path: Path) -> None:
"""The guards run off SIX columns — the light row is COMPLETE.
A column missing from ``load_only`` does not announce itself: the guard
@@ -1216,9 +1268,7 @@ def test_prune_guards_unaffected_with_light_rows(
seen = {(PRUNE_SOURCE, "notes/kept.md"), (PRUNE_SOURCE, "deck.pdf")}
session = SessionLocal()
try:
pruned_toggle_off = _prune(
session, {PRUNE_SOURCE}, seen, images=False, prune_batch=0
)
pruned_toggle_off = _prune(session, {PRUNE_SOURCE}, seen, images=False, prune_batch=0)
finally:
session.close()
# Toggle OFF: BOTH image docs are invisible to the walk, not deleted —
@@ -1228,9 +1278,7 @@ def test_prune_guards_unaffected_with_light_rows(
session = SessionLocal()
try:
pruned_toggle_on = _prune(
session, {PRUNE_SOURCE}, seen, images=True, prune_batch=0
)
pruned_toggle_on = _prune(session, {PRUNE_SOURCE}, seen, images=True, prune_batch=0)
finally:
session.close()
# Toggle ON: the deleted image file prunes its row AND its copy (the
@@ -1240,8 +1288,82 @@ def test_prune_guards_unaffected_with_light_rows(
assert not gone_copy.exists()
assert kept_copy.exists()
assert sorted(
path
for (path,) in db.execute(
select(Document.path).where(Document.source == PRUNE_SOURCE)
)
path for (path,) in db.execute(select(Document.path).where(Document.source == PRUNE_SOURCE))
) == ["deck.pdf", "deck.pdf#page=1#img=0.png", "notes/kept.md"]
def test_both_prune_guards_cost_the_same_statement_budget(
db: Session, tmp_path: Path, statement_capture: list[str]
) -> None:
"""The guards add NO hidden read — proved by statement count, both ways.
:func:`test_prune_guards_unaffected_with_light_rows` above proves the
guards DECIDE correctly off the six light columns. This pins the other
half of the same contract, the half that is invisible in behaviour: a
guard reaching for a deferred attribute would still decide correctly and
only show up as ONE MORE SELECT PER ROW. So the assertion here is a
statement count, not an outcome:
* the SAME projection runs under both toggles — one candidate walk
apiece, naming the same six columns (images OFF only skips rows
through ``is_image``, a column the walk already reads);
* neither walk projects ``documents.content``;
* deleting an image row + copying its file costs no extra document
SELECT (the ``image_path`` the copy cleanup uses is in the projection).
Total: exactly TWO document SELECTs for the two syncs, whatever the
guards do.
"""
image_dir = tmp_path / "image_dir"
image_dir.mkdir()
gone_copy = image_dir / "gone.png"
gone_copy.write_bytes(b"\x89PNG\r\n\x1a\n gone")
def add(path: str, *, is_image: bool, copy: Path | None = None) -> None:
db.add(
Document(
source=PRUNE_SOURCE,
path=path,
full_path=f"/srv/brain/{PRUNE_SOURCE}/{path}",
title=f"Budget probe {path}",
content=PRUNE_CONTENT,
content_hash=f"{abs(hash(path)):064x}"[:64],
is_image=is_image,
image_path=str(copy) if copy else None,
)
)
add("notes/kept.md", is_image=False)
add("notes/gone.md", is_image=False)
add("img/gone.png", is_image=True, copy=gone_copy)
add("deck.pdf#page=1#img=0.png", is_image=True)
db.commit()
seen = {(PRUNE_SOURCE, "notes/kept.md")}
projections: list[frozenset[str]] = []
for toggle in (False, True):
since = len(statement_capture)
session = SessionLocal()
try:
_prune(session, {PRUNE_SOURCE}, seen, images=toggle, prune_batch=0)
finally:
session.close()
# Only what THIS sync emitted — the capture is cumulative.
walks = [
s for s in _doc_selects(statement_capture[since:]) if "WHERE documents.source" in s
]
assert len(walks) == 1, (
f"the images={toggle} prune ran {len(walks)} candidate walks; "
"one per sync is the budget (a second is a per-row lazy load)"
)
projections.append(_projected_columns(walks[0]))
assert projections[0] == projections[1], (
"the two toggles read DIFFERENT columns — a guard grew a deferred "
f"read:\n off: {sorted(projections[0])}\n on: {sorted(projections[1])}"
)
assert "documents.content" not in projections[0]
assert len(_doc_selects(statement_capture)) == 2, (
"two syncs must cost exactly two document SELECTs in total, guards "
"included — any extra is a per-row read the light row must cover"
)
tests/unit/test_importer_defer_content.py
+258
View File
@@ -0,0 +1,258 @@
"""Phase 185, task 02 — the ONE fast-path row statement (unit pins).
``app/rag/importer.py::_doc_row_stmt`` is the single place the importer's
unchanged fast path (phase 147) loads a ``Document`` row, and it defers
exactly ONE column: ``Document.content``. That shape is load-bearing twice
over — it is what keeps a full re-sync's memory bound to row METADATA
instead of to the corpus (the 2026-10-15 incident: 17,407 rows × avg
15,217 chars = 252.6 MB of markdown hydrated for a run that re-indexed
217 files), AND it is what must NOT grow into a lazy-load landmine six
phases from now.
Pinned here (the behaviour itself — the captured SQL, the one-SELECT-per-file
count and the backfill's single lazy read — lives in
``tests/integration/test_import_fast_path_columns.py``):
* the statement defers ``content`` and NOTHING ELSE: its projection names
every other mapped column of ``Document``, so a column a later phase adds
keeps today's whole-row semantics (LOCKED A1 — no ``load_only`` allowlist,
which would silently defer every future column and resurface as an
unexplained N+1);
* the statement is the ``(source, path)`` lookup the fast path has always
used — the deferral changed the projection, never the predicate;
* all FOUR fast-path row loads call the helper (``_index_file``,
``_index_pdf_file``, ``_index_one_embedded_image``,
``_index_image_file``) and none of them builds its own
``select(Document)`` — the four sites cannot drift;
* the importer contains no ``raiseload``/``noload`` (LOCKED A2): the
phase-118 summary-NULL backfill genuinely reads ``doc.content`` and must
pay one SELECT, not crash;
* reading the deferred column through a real session still yields the stored
text (the honest "plain ``defer`` keeps today's behaviour" claim).
"""
from __future__ import annotations
import ast
import inspect
import re
import uuid
from collections.abc import Iterator
import pytest
from sqlalchemy import inspect as sa_inspect
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
from sqlalchemy.orm import Session
from app.models import Document
from app.rag import importer
from app.rag.importer import _doc_row_stmt
#: The four loads that share the phase-147 unchanged fast path (phase 185
#: D1) — the ONLY call sites of :func:`_doc_row_stmt`.
FAST_PATH_FUNCTIONS = (
"_index_file",
"_index_pdf_file",
"_index_one_embedded_image",
"_index_image_file",
)
#: The helper call as it appears at an importer call site (``session.scalar(
#: _doc_row_stmt(source, rel))``) — counted, not trusted.
_HELPER_CALL = "_doc_row_stmt("
#: ``documents.<column>`` inside a compiled projection (everything the
#: statement SELECTs before ``FROM``). The loader options are invisible in
#: the ORM statement's own column list — only the COMPILED projection shows
#: what actually travels to the database, which is what this phase is about.
_PROJECTION_COLUMN_RE = re.compile(r"documents\.([a-z_]+)")
def _projected_columns() -> set[str]:
"""The ``documents`` columns the fast-path statement actually SELECTs."""
compiled = str(_doc_row_stmt("s", "p").compile(dialect=postgresql.dialect()))
projection = compiled.split("\nFROM")[0].split(" FROM ")[0]
return set(_PROJECTION_COLUMN_RE.findall(projection))
def test_helper_defers_content_and_nothing_else() -> None:
"""``content`` is the ONLY column missing from the fast-path projection.
The compiled projection is compared against the mapper's FULL column set,
so a future ``Document`` column must appear there — if someone swaps the
``defer`` for a ``load_only`` allowlist, the new column goes missing and
this test fails (which is exactly the drift LOCKED A1 forbids).
"""
mapped = {c.name for c in sa_inspect(Document).columns}
projected = _projected_columns()
assert "content" in mapped, "Document.content vanished — this pin needs a new shape"
assert projected, "the fast-path statement projects no documents columns at all"
assert "content" not in projected, (
"the fast-path row statement no longer defers Document.content — a "
"full walk would hydrate the corpus markdown again (252.6 MB on the "
"incident box)"
)
assert projected == mapped - {"content"}, (
"the fast-path row statement defers more (or other) columns than "
f"content: missing={sorted(mapped - projected - {'content'})} "
f"unexpected={sorted(projected - mapped)} — only Document.content "
"may be deferred (LOCKED A1)"
)
def test_helper_keeps_the_fast_path_predicate() -> None:
"""The predicate stays the ``(source, path)`` lookup of phase 147."""
compiled = str(_doc_row_stmt("the-source", "the/path.md").compile()).lower()
assert "documents.source" in compiled
assert "documents.path" in compiled
assert "where" in compiled
# The bind names the integration probe attributes statements by.
assert "source_1" in compiled and "path_1" in compiled
def test_every_fast_path_load_calls_the_helper() -> None:
"""All four fast-path row loads go through ONE statement (phase 185 D1).
Source-level on purpose: the deferral must be impossible to bypass, and
a fifth whole-entity load would silently undo the phase for one document
class. The two per-source post-pass walks
(``_enrich_pending_images``, ``_backfill_image_contexts``) and the prune
(``load_only`` of six light columns, phase 184) are NOT fast-path loads
and are deliberately excluded — see the module docstring of
``tests/integration/test_import_fast_path_columns.py``.
"""
for name in FAST_PATH_FUNCTIONS:
fn = getattr(importer, name)
src = inspect.getsource(fn)
assert _HELPER_CALL in src, (
f"{name} no longer loads its row through importer.{_HELPER_CALL} — "
"the four fast-path row loads must share the ONE deferring "
"statement so they cannot drift"
)
assert "select(Document)" not in src, (
f"{name} builds its own select(Document) row again — a whole-entity "
"fast-path load re-hydrates Document.content (phase 185)"
)
def test_helper_has_exactly_the_four_fast_path_call_sites() -> None:
"""The importer calls the helper exactly four times (one per fast path).
A fifth ``session.scalar(_doc_row_stmt(…)`` would mean a new consumer was
wired to the fast-path statement (it is the ``(source, path)`` deferring
lookup, not a general-purpose loader), and a surviving whole-entity
``(source, path)`` ``select(Document)`` would mean a bypass.
"""
src = inspect.getsource(importer)
calls = src.count(f"session.scalar({_HELPER_CALL}")
assert calls == len(FAST_PATH_FUNCTIONS), (
f"importer.py calls session.scalar({_HELPER_CALL}…) {calls} times — "
f"expected exactly {len(FAST_PATH_FUNCTIONS)} (the four fast-path "
"row loads)"
)
# No fast-path-shaped whole-entity load survives next to the helper.
assert src.count("select(Document).where(Document.source ==") == 0, (
"a (source, path) whole-entity Document load is back — route it "
"through _doc_row_stmt or give it its own explicit projection"
)
def test_no_raiseload_or_noload_anywhere_in_the_importer() -> None:
"""LOCKED A2: the deferred column must QUERY, never raise or come back None.
``raiseload``/``noload`` would turn the phase-118 summary-NULL backfill
(``_unchanged_doc`` step 3, ``content=doc.content``) into a crash or an
empty summary — a behaviour regression on the one branch that genuinely
needs the stored markdown. The accepted cost is one SELECT for that row.
Checked through the AST, not the text: the module's own docstrings argue
against both options by NAME, so a substring scan would fail on the
prose. Only real call/import names count here.
"""
tree = ast.parse(inspect.getsource(importer))
called: set[str] = set()
imported: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Call):
if isinstance(node.func, ast.Name):
called.add(node.func.id)
elif isinstance(node.func, ast.Attribute):
called.add(node.func.attr)
elif isinstance(node, ast.ImportFrom | ast.Import):
imported.update(alias.name for alias in node.names)
for forbidden in ("raiseload", "noload"):
assert forbidden not in called | imported, (
f"importer.py calls/imports {forbidden} — phase 185 LOCKED A2 "
"keeps plain defer() so the summary-NULL backfill reads the "
"stored content instead of raising"
)
assert "defer" in called, "the fast path no longer calls defer() at all"
# The importer's ONLY load_only is _prune's six-light-column row
# (phase 184) — it must never reach the fast path, whose shape is
# "everything except content" (LOCKED A1).
prune_src = inspect.getsource(importer._prune)
assert "load_only" in inspect.getsource(importer)
assert "defer" not in prune_src, (
"_prune picked up a defer() — its light row is an explicit "
"load_only allowlist by phase 184, not a deferral"
)
@pytest.fixture()
def probe_row(db: Session) -> Iterator[tuple[str, str]]:
"""One seeded row (raw SQL — this file pins a SELECT, not the importer)
removed again after the test."""
source = f"phase185-unit-{uuid.uuid4().hex[:8]}"
rel = "notes/deferred.md"
db.execute(
text(
"INSERT INTO documents (id, source, path, full_path, title, content, "
"content_hash, file_hash, indexed_at, created_at, created_at_manual, "
"summary, is_image) VALUES (:id, :source, :path, :full_path, :title, "
":content, :content_hash, :file_hash, now(), now(), false, :summary, "
"false)"
),
{
"id": uuid.uuid4(),
"source": source,
"path": rel,
"full_path": f"/tmp/{source}/{rel}",
"title": "Deferred",
"content": "the stored markdown body — phase 185",
"content_hash": "ch",
"file_hash": "fh",
"summary": None,
},
)
db.commit()
db.expunge_all()
try:
yield source, rel
finally:
db.execute(text("DELETE FROM documents WHERE source = :s"), {"s": source})
db.commit()
db.expunge_all()
def test_deferred_content_still_reads_back(db: Session, probe_row: tuple[str, str]) -> None:
"""A deferred column is LAZY, not gone: the row reads back whole.
The trade phase 185 accepted (D2): the metadata arrives with the row, and
the markdown arrives on demand for the one reader that asks for it. With
``raiseload`` this would raise; with ``noload`` it would be ``None`` —
both would break the phase-118 backfill, and both are pinned out by
:func:`test_no_raiseload_or_noload_anywhere_in_the_importer`.
"""
source, rel = probe_row
doc = db.scalar(_doc_row_stmt(source, rel))
assert doc is not None
assert doc.file_hash == "fh"
assert doc.summary is None
assert doc.content == "the stored markdown body — phase 185"
# And the second read is served from the identity map, not the database.
assert doc.content == "the stored markdown body — phase 185"