Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 127_sync_incremental_folder_summaries
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Details
Build and Push Containers / build-and-push-db (push) Successful in 15s
Details

Browse Source 
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
ducoterra
2026-09-27 21:12:25 -04:00
parent f50d4c8bc3
commit 6e654df387
38 changed files with 2741 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/127_sync_incremental_folder_summaries/00_phase.md → .agents/phases/complete/127_sync_incremental_folder_summaries/00_phase.md
View File
.agents/phases/todo/127_sync_incremental_folder_summaries/01_importer_changed_paths.md → .agents/phases/complete/127_sync_incremental_folder_summaries/01_importer_changed_paths.md
View File
.agents/phases/todo/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md → .agents/phases/complete/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md
View File
.agents/phases/todo/127_sync_incremental_folder_summaries/03_sync_wiring.md → .agents/phases/complete/127_sync_incremental_folder_summaries/03_sync_wiring.md
View File
.agents/phases/todo/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md → .agents/phases/complete/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.md
+10
View File
@@ -0,0 +1,10 @@
All verification complete. Final report:
**Phase 127 — final verification pass: all green** (all 4 tasks were already in `complete/`; no code changes were needed this pass)
- **Verified implementation:** `ImportSummary.changed_paths` recorded at all 5 added/updated sites + `_prune` (before delete, guards intact); `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys = candidates ∩ affected, prune pass both modes, hook total = affected count); sync's changed-KB branch passes `changed_docs=summary.changed_paths` with the `changed=N` log line; CLI (`scripts/import_docs.py`) passes no change set — full-regen contract preserved
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** ① one-file change rewrites only the folder chain (E2E pins sibling rows byte-identical, text AND `updated_at`) ✅ ② coverage >90% ✅ ③ E2E isolated green ✅ ④ no completed-phase behavior change (full suite + first-sync full-regen E2E) ✅ ⑤ atomic commit — delegated to the harness per this pass's rules (I make no git changes)
- **Docs:** README.md — fixed 7 pre-existing inconsistencies: missing Theme row in the pages table; stale `ls` (now drill-down tree, phase 94) and `read` (now capped at `BOR_READ_MAX_CHARS`, phase 95) tool descriptions; missing phase-124 answer images (new section + `show_image` tool row + `BOR_ANSWER_IMAGES` env row); "Document summaries (non-markdown)" → all docs incl. markdown (phase 118); theme palette 8 → 17 colors (phases 92/93); `BOR_LLM_SUMMARY_MODEL` now names folder summaries; CLI section notes overview + folder-summary regen. AGENTS.md — verified consistent, no changes needed
- **Deviations:** none. Untracked pre-existing `NOTES.md` left untouched (owner's notes, will ride in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.md
+10
View File
@@ -0,0 +1,10 @@
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.
**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)
- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.md
+16
View File
@@ -0,0 +1,16 @@
Everything is green. Final state verified:
- **`app/rag/importer.py`**: `ImportSummary.changed_paths` field (task-specified docstring, `field(default_factory=list)`); `changed_paths.append` beside **every** `added`/`updated` increment — all 9 sites across the 5 verb blocks (text, PDF, PDF-embedded image, image-context enrichment, standalone image) — and in `_prune` (new `summary: ImportSummary | None = None` param, appended before `session.delete`, only for docs actually deleted past both guards); only the `import_sources` call site updated; `log()` untouched.
- **`tests/unit/test_importer.py`**: 8 new tests — mixed 2-added/1-updated/1-pruned run with exact order + `len == added+updated+pruned`, unchanged re-run empty, backfill re-run empty, date-only re-run empty, changed image (added+updated sites), changed PDF (both sites), `_prune` with/without summary, and default-construction `== []`.
**Results:**
- `uv run pytest --cov=app --cov-report=term-missing` → 3005 passed, exit 0, TOTAL 99% (gate >90% ✅; `app/rag/importer.py` 98%, remaining misses pre-existing)
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors (fixed two `# pyright: ignore[reportCallIssue]` placements)
- Grep audit: 9 increment sites ↔ 9 matching appends ✅
**Docs:** no doc impact (internal record-tracking only — no commands, config, env vars, or API surface changed; README/AGENTS.md untouched).
**Decisions:** appended right after each verb if/else (single line, same encounter order as the counter); enrichment path records `(source, image_doc.path)` since its docstring already states it fires the KB-change gate; PNG/PDF fixtures generated in-test with PyMuPDF (house no-binary-fixtures convention).
**Next pending task:** `02_folder_summaries_affected_mode.md`
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 98%]
..................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5122 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.md
+12
View File
@@ -0,0 +1,12 @@
Everything is green. The unit run shows all dots through [100%] with exit 0.
**Task 02 complete — report**
- **Implemented** in `app/rag/folder_summaries.py`:
- `affected_folder_keys(changed)` pure helper (next to `folder_of`/`group_by_folder`) — per changed doc: source-root key + `folder_of` prefix walk; fragment paths fold in naturally; empty input → empty set
- `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg: `None` = byte-identical full regen (CLI stays on it, verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys)`; manual skip, fail-soft, `on_progress` (total = affected count), and the prune pass are untouched code; docstrings (module + function + progress) updated
- **Tests** (`tests/unit/test_folder_summaries.py`, +380 lines): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts (call log + fresh rows), unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs
- **Results:** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed**, exit 0; `app/` coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit -q` → exit 0, all green; `uv run ruff check .` → passed; `uv run pyright` → 0 errors
- **Docs:** no doc impact (README sync description remains accurate; no new env/API/commands)
- **Decisions:** intersection with `candidates` (per task ASSUMPTION) — vanished affected folders are the prune pass's job; no caller combines `changed_docs` + `only_missing` (documented in the docstring)
- **Next pending task:** `03_sync_wiring.md` (sync's changed-KB branch passes `summary.changed_paths`)
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.md
+12
View File
@@ -0,0 +1,12 @@
All gates are green. The previous rejection was an infrastructural one (the child run ended without emitting its final report) — no test failures existed; the working tree already contained the completed task. I re-verified everything end-to-end this run: full suite with coverage, the target unit files, and lint + types.
**Task 02 complete — report**
- **Implemented** in `app/rag/folder_summaries.py` (unchanged from prior run, verified intact):
- `affected_folder_keys(changed)` pure helper — per changed doc: source-root key `(source, "")` + `folder_of` prefix walk (same concept as `group_by_folder` pass 1); fragment paths (`x.pdf#page=…`) fold into parent chain; empty input → empty set
- `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg — `None` = byte-identical full regen (CLI verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys(changed_docs))`; manual skip / fail-soft / `on_progress` (total = affected count) / prune pass untouched; module + function docstrings updated
- **Tests** (`tests/unit/test_folder_summaries.py`, 14 new): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts, unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs
- **Results (this run):** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed, exit 0**; `app/` TOTAL coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit/test_folder_summaries.py tests/unit/test_importer.py -q` → exit 0; `uv run ruff check .` → all passed; `uv run pyright` → 0 errors, 0 warnings
- **Docs:** no doc impact (README sync section stays accurate; no new env/API/commands)
- **Decisions:** intersection with `candidates` per task ASSUMPTION — vanished affected folders are the prune pass's job; `changed_docs` + `only_missing` documented as never combined by any caller
- **Next pending task:** `03_sync_wiring.md` — the sync's changed-KB branch passes `summary.changed_paths`; docstrings + logging
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.md
+20
View File
@@ -0,0 +1,20 @@
Everything is verified green. The previous run completed all of task 03's work but its validation was interrupted by a KeyboardInterrupt; I re-verified every gate from scratch and confirmed the implementation matches the task spec exactly.
**Task 03 — Sync wiring: complete (verified)**
- `app/api/sync.py`: changed-KB branch calls `generate_folder_summaries(fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook)`; else-arm (gap probe, `only_missing=True`) byte-identical; log line now `sync: folder_summaries changed=%d stats=%s`
- Docstrings updated to the incremental contract: module docstring step-5 + phase-machine paragraph (incl. stale "at least 2 docs" → "at least 1 doc" fix) and `SyncStatus` `summaries_done/total` note (total = affected count on changed KB, missing count on gap-fill)
- `detail` dict (12 keys) and `GET /api/sync/status` shape unchanged — asserted in tests
- Fakes extended to accept/record `changed_docs=`: `tests/integration/test_sync_api.py` (`FakeFolderSummaries`, `_GatedFolderSummaries`, `_HookThenFail`) + `tests/unit/test_sync_button.py`
- 3 new tests in `tests/integration/test_sync_api.py`: change-set wiring + `changed=N` log + detail shape; unchanged-walk gap probe passes `only_missing=True` with `changed_docs=None`; mid-span status `summaries_total == len(affected_folder_keys(...))` — all 3 pass individually (3 passed, 22 deselected)
**Results:**
- `uv run pytest tests/unit -q` → 2326 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, app/ **99%** (sync.py 100%, folder_summaries.py 100%, importer.py 98%) — gate >90% ✅
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
**Docs:** no doc impact (no setup/commands/config/env/API changes; README never described the regeneration mode, AGENTS.md untouched).
**Note:** prune-only syncs still take the gap-probe branch (pre-existing `added+updated>0` gate untouched, per task scope). `NOTES.md` (owner's untracked note) left in place.
**Next pending task:** `04_e2e_incremental_sync.md` (dedicated Playwright E2E in isolation)
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.md
+18
View File
@@ -0,0 +1,18 @@
All green. Final report:
**Implemented**
- New `tests/e2e/test_incremental_folder_summaries.py` — dedicated mock-only Playwright suite (phase 127 task 04), house pattern from `test_sync_summary_visibility.py`: module app on port 8147 (env-overridable), host temp-dir `kind=local` source `incsum/` (4 folders × 2 docs + 1 root doc = 9 files, 5 candidates), autouse truncate + settle-wait cleanup, stale-terminal freshness gate in `_run_sync`.
- 3 tests: (1) first full sync → exactly 5 canned rows, terminal keeps 5/5; (2) one `alpha/` file changed → only `alpha` + source root re-stamped (strictly newer `updated_at`), `bravo`/`charlie`/`delta` byte-identical (text AND `updated_at`), terminal keeps `summaries_done == summaries_total == 2`; (3) `bravo/` vanishes → its row pruned by the prune pass, `charlie` + root re-stamped, `alpha`/`delta` byte-identical, terminal 2/2.
**Gates (all 100% green)**
- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed, in isolation (run twice; DB up)
- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%)
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
**Docs**
- README.md: added step 4 to the "Sync from the UI" pipeline (folder summaries: incremental on changed KB, gap-fill on unchanged) — it predated folder summaries entirely (stale since phase 94). AGENTS.md: no impact.
**Decisions**
- "bravo's last doc" read as *the docs whose deletion makes the folder vanish* (0 docs → not a candidate); both `bravo/` files deleted, since a 1-doc folder is still a candidate and would not be pruned. All other assertions match the task verbatim.
**Next pending task:** none — task 04 is the last task of phase 127.
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.err
View File
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.md
+20
View File
@@ -0,0 +1,20 @@
All checks are green and consistent. The documentation is accurate:
- **README.md** — the step-4 addition from the prior run is intact and matches the implemented behavior (incremental on changed KB via the changed set's folder chains, gap-fill on unchanged walks). The "Updating the documents" CLI section makes no regeneration-scope claims (CLI keeps full regen — consistent).
- **AGENTS.md** — no references to the sync pipeline internals; operating rules, test gates, and conventions all still accurate. No changes needed.
All gates re-verified from scratch this run (the prior a2 attempt was an infrastructural interruption — it died mid-run without a report, leaving no port or DB residue):
**What was verified/fixed this run**
- Confirmed working tree intact from the successful a1 run: `tests/e2e/test_incremental_folder_summaries.py` (3 tests: first-full-sync 5 canned rows @ terminal 5/5; one `alpha/` file → only `alpha`+root re-stamped, siblings byte-identical text AND `updated_at`, terminal 2/2; `bravo/` vanishes → row pruned, `charlie`+root re-stamped, terminal 2/2) plus tasks 01–03 code/tests.
- No code changes required — re-ran every gate fresh; found no defects, no leftovers (ports clean, DB truncated).
- Doc consistency check: README step-4 accurate; AGENTS.md untouched.
**Results (this run)**
- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%)
- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed in isolation (run twice)
- `uv run ruff check .` → all passed; `uv run pyright` → 0 errors
**Notable decision** (unchanged from a1): "bravo's last doc" = the docs whose deletion makes the folder vanish (both deleted; a 1-doc folder is still a candidate and wouldn't prune).
**Next pending task:** none — task 04 is the last task of phase 127.
.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.validate
+113
View File
@@ -0,0 +1,113 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 244 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5133 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
README.md
+54 -12
View File
@@ -65,6 +65,7 @@ client-side router. Direct URLs deep-link to the matching view.
| **Tuning** | `/tuning.html` | Admin only |
| **Saved chats** | `/history.html` | Admin only |
| **Access tokens** | `/tokens.html` | Admin only |
| **Theme** | `/theme.html` | Admin only |
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
| **Login** | `/login.html` | Everyone |
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
@@ -207,6 +208,10 @@ reflect the new files.
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
bmp`) are indexed as documents too — see
[Image support](#image-support).
- A run that changed the KB also regenerates the **KB overview** and the
**folder summaries** (the drill-down catalog's stored per-folder
descriptions) with the `lite` model — best-effort: a summary failure
never fails the import.
### Git-based sources
@@ -262,6 +267,12 @@ whole git-source refresh in one click:
2. **re-import with prune** — files deleted upstream leave the index.
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
chat turn injects) — only when the import changed the knowledge base.
4. **refresh the folder summaries** (the drill-down catalog's stored
per-folder descriptions) — a changed knowledge base rewrites ONLY the
folders the changed files touch (each file's folder chain up to its
source root — phase 127: one changed file re-stamps a handful of
rows, not the whole table); an unchanged re-sync instead gap-fills
any missing rows (`only_missing`).
- States: the button shows **Syncing…** with a spinner while polling
`GET /api/sync/status` every 2 s. No client-side timeout — a clone +
@@ -288,13 +299,16 @@ To hide it, set `BOR_STREAM_THINKING=0`.
Retrieval only puts the top documents in context. When an answer depends on
a file a note *references* ("the exact JSON shape is in
`example-record-file.json`"), the model can extend its own context with
three server-side tools — on **grounded** (high-relevance) turns only:
server-side tools — on **grounded** (high-relevance) turns only:
| Tool | Description |
|------|-------------|
| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source |
| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string |
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
| **`read(path)`** | Appends one indexed document's text to context by its combined `source/path`; very large documents are truncated at `BOR_READ_MAX_CHARS` with a notice that points at `grep` to find the rest |
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
Each call is executed against Postgres only (no extra LLM round trip) and
streamed as an SSE `tool` frame. In the chat, each call shows a transient
@@ -361,12 +375,14 @@ docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
---
## Document summaries (non-markdown)
## Document summaries
Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
retrieval can miss exactly the documents that are all configuration. At
import time, every **non-markdown** A9 document is summarized by the aipi
`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`):
import time, **every** new or changed document gets a `lite`-model summary —
**markdown included**: the summary is a natural-language retrieval target
and the seed text for the onboarding suggestion blocks (phase 118 retired
the old "markdown is already natural language" exclusion):
* The summary is stored on `documents.summary` **and** indexed as one extra
embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
@@ -452,6 +468,30 @@ image:
A question image is **never indexed as a document** — it is turn-local
storage, not a source.
### Answer images (phase 124)
A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
on, the grounded chat model may show a standalone image document as part
of its answer:
- **The model decides.** The agent is offered a fourth tool,
`show_image(path)`, and calls it only when the image itself contains
the answer (a diagram, screenshot, or photo of the thing asked about) —
never for mere topical relevance. At most **one** image per answer
(enforced server-side); the text answer is always given in full — the
image supplements, never replaces.
- **Rendered as an answer figure.** A shown image appears as a
first-class figure below the answer text and above the source chips —
on the live turn, after a page reload, and on the shared chat page; a
failed load collapses the figure (never a broken-image icon).
- **Image chips on zero-read turns.** A retrieved image document in the
suggested tier is auto-cited: its source chip (with figure) appears
even when the model never `read`s it — the model's seed block already
carries the image's full description, so a real model has no reason to
read it (this applies with `BOR_IMAGES=1`, any `BOR_ANSWER_IMAGES`).
- **Off = byte-identical.** No fourth tool offered, no figure, no
`answer_image` key in any SSE frame or saved record.
---
## Caching & deploys
@@ -515,7 +555,7 @@ to `.env` and adjust.
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
@@ -578,7 +618,7 @@ to `.env` and adjust.
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
### Image support (phases 122/123)
### Image support (phases 122/123/124)
| Env | Default | Meaning |
|-----|---------|---------|
@@ -587,6 +627,7 @@ to `.env` and adjust.
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |
### Docs push
@@ -615,10 +656,11 @@ Every identity string is an env var: `BOR_APP_NAME` (display name),
applied by `assets/brand.js` at boot.
The colors are set from the admin **Theme tab** (`/theme.html`,
admin-only — phase 91): the 8 identity colors plus the three strings
above are edited with text fields and color pickers, persisted in the
`ui_settings` table, and injected into every served page as an inline
`<style>` BEFORE first paint (no red flash, no pop-in). The old
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
the 8 semantic state colors — success/error/accent families) and the
three strings above are edited with text fields and color pickers,
persisted in the `ui_settings` table, and injected into every served page
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The old
CSS-file theming is retired — a leftover `BOR_THEME` line in a
deployment's `.env` is simply ignored. Leave everything unset and the
app renders the built-in dark-tech palette byte-identically.
app/api/sync.py
+52 -23
View File
@@ -56,9 +56,16 @@ decisions):
(phase 31 trigger, best-effort inside) — and ``generate_
folder_summaries`` (phase 94, task 02) regenerates the stored folder
summaries (the drill-down ``ls``'s per-level descriptions): its gate
is the same change trigger (full regeneration) **plus**, on an
unchanged re-sync, a GAP probe — a candidate folder (at least 2
docs) with no stored row (``missing_folder_summaries``, phase 96,
is the same change trigger — INCREMENTAL since phase 127 (task 03):
the run's changed documents (``ImportSummary.changed_paths``, the
import's added/updated/pruned ``(source, path)`` set) are passed as
``changed_docs=`` and ONLY their folder chains (each document's
folder up to its source root — ``affected_folder_keys``) are
rewritten, so a one-file change burns a handful of rows, not the
whole table (the ``scripts/import_docs.py`` CLI keeps the
byte-identical full regeneration — it passes no change set) **plus**,
on an unchanged re-sync, a GAP probe — a candidate folder (at least
1 doc) with no stored row (``missing_folder_summaries``, phase 96,
task 03): a gap fills ONLY the missing rows (``only_missing=True``;
the old table-empty first-sync trigger is subsumed exactly — an
empty table leaves every candidate missing), a complete table burns
@@ -91,13 +98,16 @@ reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
(set immediately before ``import_sources``), ``"overview"`` (set
before ``regenerate_overview`` — a changed KB only), and
``"summaries"`` (set before a folder-summary generation — the
changed-KB full regeneration or the unchanged-walk gap-fill; the
no-gap skip sets NO phase and the run stays ``"import"``). While in
changed-KB incremental regeneration (phase 127, task 03 — the changed
set's folder keys only) or the unchanged-walk gap-fill; the no-gap
skip sets NO phase and the run stays ``"import"``). While in
``"summaries"`` the generator's ``on_progress`` hook fills
``current_summary`` (the ``source`` / ``source/folder_path`` being
summarized — the bare source name for the source-root row) plus
``summaries_done`` / ``summaries_total`` — the per-folder position
through the long summary span where the file count sits still.
through the long summary span where the file count sits still (the
hook's ``total`` is the run's key count — the AFFECTED count on a
changed KB, the missing count on a gap-fill).
Terminal states (success AND failed) clear ``phase`` +
``current_summary`` but keep the run's final ``summaries_done`` /
``summaries_total`` (the phase-64 keep-final-counts convention).
@@ -161,9 +171,13 @@ class SyncStatus:
``"summaries"`` phase, ``current_summary`` is the source /
``source/folder_path`` the generator is summarizing right now and
``summaries_done`` / ``summaries_total`` carry the generator's
progress hook's position. Terminal states clear ``phase`` +
``current_summary`` but keep the run's final summary counts (the
phase-64 keep-final-counts convention).
progress hook's position — the hook's ``total`` is the run's key
count: the AFFECTED count on a changed KB (phase 127, task 03 —
the changed set's folder keys, NOT the table's candidate count) or
the missing count on an unchanged-walk gap-fill (as before).
Terminal states clear ``phase`` + ``current_summary`` but keep the
run's final summary counts (the phase-64 keep-final-counts
convention).
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -358,13 +372,19 @@ async def _run_sync() -> None:
if summary.added + summary.updated > 0:
_status.phase = "overview" # phase 98 (task 01)
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03): the folder
# summaries — the drill-down ls's per-level descriptions. A
# changed KB (added + updated > 0) is a full regeneration
# (today's behavior, byte-identical); an unchanged re-sync
# takes the GAP probe instead of the old table-empty one — a
# candidate folder (at least 1 doc — every existing folder)
# with no stored row: a gap
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
# descriptions. A changed KB (added + updated > 0) is an
# INCREMENTAL regeneration (phase 127, task 03): the run's
# changed documents (summary.changed_paths — the import's
# added/updated/pruned set) are passed as changed_docs= and
# ONLY their folder chains (each document's folder up to its
# source root) are rewritten — a one-file change burns a
# handful of rows, not the whole table (the CLI keeps the
# byte-identical full regeneration — it passes no change set);
# an unchanged re-sync takes the GAP probe instead of the old
# table-empty one — a candidate folder (at least 1 doc — every
# existing folder) with no stored row: a gap
# fills ONLY the missing rows (only_missing=True — the
# subsumed table-empty first-sync trigger included, where
# every candidate is missing), a complete table burns zero
@@ -378,8 +398,9 @@ async def _run_sync() -> None:
# Phase 98 (task 01): the folder-summary progress hook — the
# closure-captures-``_status`` convention the file's ``_hook``
# above already uses; the generator's ``total`` (the loop-start
# candidate count) and the bare source name for the root row
# come straight from the hook's arguments (D5's shape).
# key count — the affected count on a changed KB, the missing
# count on a gap-fill) and the bare source name for the root
# row come straight from the hook's arguments (D5's shape).
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
_status.current_summary = (
source if folder_path == "" else f"{source}/{folder_path}"
@@ -390,15 +411,23 @@ async def _run_sync() -> None:
fs_db = SessionLocal()
try:
if summary.added + summary.updated > 0:
# Changed KB: full regeneration — the summaries phase
# starts now, and the hook reports the per-folder
# position through the (long) span.
# Changed KB: incremental regeneration (phase 127,
# task 03) — ONLY the changed set's folder keys
# (changed_docs=summary.changed_paths: each document's
# folder chain up to its source root); the summaries
# phase starts now, and the hook reports the per-folder
# position through the (long) span (its total is the
# affected count).
_status.phase = "summaries"
folder_stats = await generate_folder_summaries(
fs_db, llm, on_progress=_summary_hook
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
)
fs_db.commit()
logger.info("sync: folder_summaries stats=%s", folder_stats)
logger.info(
"sync: folder_summaries changed=%d stats=%s",
len(summary.changed_paths), folder_stats,
)
else:
missing = missing_folder_summaries(fs_db)
if missing:
app/rag/folder_summaries.py
+75 -6
View File
@@ -59,11 +59,22 @@ the stored keys — one concept, as with :func:`group_by_folder`), and
:func:`generate_folder_summaries`'s ``only_missing=True`` fills
EXACTLY those on the next sync — existing rows stay
byte-identical (text AND ``updated_at``), the prune pass still runs.
Restricted regeneration (phase 127, task 02): a changed-KB sync no
longer rewrites EVERY folder — the sync path passes the run's changed
documents (``ImportSummary.changed_paths``) as ``changed_docs=`` and
the generator then rewrites ONLY the candidate folders the change
touches — each document's folder chain up to the source root
(:func:`affected_folder_keys`, the SAME prefix walk
:func:`group_by_folder` pass 1 uses — one concept); every other
stored row keeps its text AND ``updated_at``. The
``scripts/import_docs.py`` CLI keeps the byte-identical full
regeneration (``changed_docs=None``).
"""
from __future__ import annotations
import logging
from collections.abc import Callable, Sequence
from collections.abc import Callable, Iterable, Sequence
from datetime import UTC, datetime
from typing import Protocol
@@ -209,6 +220,32 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
return groups
def affected_folder_keys(
changed: Iterable[tuple[str, str]],
) -> set[tuple[str, str]]:
"""The candidate folder keys a changed-document set invalidates.
For each ``(source, path)``: the source-root key ``(source, "")``
plus every folder prefix of *path* — :func:`folder_of` walked from
the document's nearest folder up to ``""`` — the SAME prefix walk
:func:`group_by_folder` pass 1 uses (one concept): a doc under
``a/b/`` affects ``a/b``, ``a``, and the source root. A
root-level file (``folder_of(path) == ""``) affects ONLY the
source root. Embedded-image fragment paths (``x.pdf#page=…#img=…``,
phase 126) fold into their parent's folder chain with no special
case — ``folder_of`` never sees a ``/`` inside the fragment.
Returns a set; the generator sorts it (deterministic order).
"""
keys: set[tuple[str, str]] = set()
for source, path in changed:
keys.add((source, ""))
folder = folder_of(path)
while folder:
keys.add((source, folder))
folder = folder_of(folder)
return keys
def _first_summary_line(summary: str | None) -> str:
"""First line of a stored summary, stripped; ``''`` when absent.
@@ -406,13 +443,34 @@ async def generate_folder_summaries(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
"""Regenerate the stored folder summaries for the current catalogue.
The sync-path orchestrator (phase 94, task 02 calls it
change-gated, like the KB overview; ``--limit`` debug runs pass
``skip=True``). Steps, in order:
``skip=True``).
Modes (phase 127, task 02 — the *changed_docs* kwarg):
``changed_docs=None`` (the DEFAULT) is the FULL regeneration —
every candidate folder, byte-identical to the pre-127 behavior;
the ``scripts/import_docs.py`` CLI stays on it. A GIVEN change set
(the sync path passes ``ImportSummary.changed_paths``) restricts
the generation to the CANDIDATE folders the changed documents
touch — :func:`affected_folder_keys`: a document's change
invalidates exactly its folder chain + source root, the folders
whose subtree summary text CAN change; every other stored row
keeps its summary text AND ``updated_at`` (an unaffected folder's
documents — and hence its summary — are untouched). An affected
folder with no stored row (a gap in the chain) is a candidate and
is generated; an affected folder that dropped below the minimum
is the prune pass's job (step 4), not a generation's. The two
restriction kwargs are ORTHOGONAL and no caller combines them:
``only_missing=True`` (the unchanged-sync gap fill, phase 96)
keeps its exact pre-127 semantics.
Steps, in order:
1. ``skip=True`` → a no-op: the zero stats dict is returned, the
LLM is never called, and no rows are touched.
@@ -461,10 +519,12 @@ async def generate_folder_summaries(
BEFORE the attempt — so an instant manual skip and a failed folder
BOTH advance the counter (the UI's position moves on either), and
``total`` is ``len(keys)`` at loop start (under ``only_missing
=True`` that is the MISSING count, not the full candidate count).
``None`` — the ``scripts/import_docs.py`` CLI path — is a
zero-cost no-op (guarded at the call site; nothing observable
changes without it).
=True`` that is the MISSING count, not the full candidate count;
under a ``changed_docs`` restriction that is the AFFECTED count —
the sync status's ``summaries_total`` reports it through this
existing hook). ``None`` — the ``scripts/import_docs.py`` CLI
path — is a zero-cost no-op (guarded at the call site; nothing
observable changes without it).
Only flushes — the CALLER commits (the phase-53
``bump_sources_version`` convention: the sync path owns the
@@ -490,6 +550,15 @@ async def generate_folder_summaries(
for row in db.execute(select(FolderSummary)).scalars()
}
keys = sorted(candidates)
if changed_docs is not None:
# Phase 127 (task 02): restricted regeneration — exactly the
# CANDIDATE folders the change set touches (each document's
# folder chain + source root). Unaffected candidates keep
# their rows byte-identical (text AND ``updated_at``); a gap
# in an affected folder is a candidate and gets generated;
# an affected folder that lost its last document is no
# candidate — the prune pass below drops its row.
keys = sorted(set(candidates) & affected_folder_keys(changed_docs))
if only_missing:
keys = [key for key in keys if key not in existing]
app/rag/importer.py
+23 -1
View File
@@ -244,6 +244,12 @@ class ImportSummary:
#: Fail-soft: the sync continues, this counter + the warning line
#: are the signal.
images_failed: int = 0
#: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
#: tuples in encounter order (added/updated in walk order, pruned in
#: _prune order). Drives the sync's incremental folder-summary
#: regeneration (phase 127): len(changed_paths) == added + updated +
#: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
changed_paths: list[tuple[str, str]] = field(default_factory=list)
#: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
formats: dict[str, int] = field(default_factory=dict)
@@ -590,7 +596,8 @@ async def import_sources(
logger.warning("import: --prune ignored because --limit was given")
else:
summary.pruned = _prune(
session, source_names, seen, images=llm.settings.images
session, source_names, seen, images=llm.settings.images,
summary=summary,
)
summary.embed_batches = llm.embed_batches
summary.log()
@@ -794,6 +801,7 @@ async def _index_file(
summary.added += 1
else:
summary.updated += 1
summary.changed_paths.append((source, rel))
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
@@ -1068,6 +1076,7 @@ async def _index_pdf_file(
summary.added += 1
else:
summary.updated += 1
summary.changed_paths.append((source, rel))
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
@@ -1328,6 +1337,7 @@ async def _index_one_embedded_image(
summary.added += 1
else:
summary.updated += 1
summary.changed_paths.append((source, image_rel))
summary.chunks += chunks_n
logger.info(
"import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
@@ -1728,6 +1738,7 @@ async def _enrich_pending_images(
)
continue
summary.updated += 1
summary.changed_paths.append((source, image_doc.path))
summary.chunks += chunks_n
logger.info(
"import: image-context-enriched source=%s path=%s chunks=%d",
@@ -1994,6 +2005,7 @@ async def _index_image_file(
summary.added += 1
else:
summary.updated += 1
summary.changed_paths.append((source, rel))
summary.chunks += chunks_n
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
@@ -2017,6 +2029,7 @@ def _prune(
source_names: set[str],
seen: set[tuple[str, str]],
images: bool = False,
summary: ImportSummary | None = None,
) -> int:
"""Delete documents of *source_names* whose file is no longer in *seen*.
@@ -2040,6 +2053,13 @@ def _prune(
toggle states (toggle off, the phase-122 guard already skips every
``is_image`` doc; this one is the toggle-on half of the same
"invisible, not deleted" contract).
*summary* (phase 127, task 01): when given, every document that is
ACTUALLY deleted (i.e. not skipped by either guard above) records
its ``(doc.source, doc.path)`` on ``summary.changed_paths`` —
appended BEFORE the ``session.delete`` — so the sync's incremental
folder-summary regeneration (phase 127) sees the pruned docs' folders
too. Callers passing nothing keep the pre-phase count-only contract.
"""
if not source_names:
return 0
@@ -2059,6 +2079,8 @@ def _prune(
# invisible to the walk, not deleted.
continue
_delete_image_copy(doc.image_path)
if summary is not None:
summary.changed_paths.append((doc.source, doc.path))
session.delete(doc)
pruned += 1
logger.info("import: pruned source=%s path=%s", doc.source, doc.path)
tests/e2e/test_incremental_folder_summaries.py
+619
View File
@@ -0,0 +1,619 @@
"""Phase 127 task 04 E2E (Playwright, mock-only): a one-file change
rewrites ONLY that folder's summary chain.
The dedicated story suite for ``127_sync_incremental_folder_summaries``
(A16 — one Playwright file per phase, run in isolation): the owner's
report was that "on sync brain of reese rewrites all the summaries for
every directory — so even if just one file changed this can result in
700+ directory summary rewrites in production which can take a very
long time" (``TODO.md`` L3). The fix under test: a KB-changing sync
passes the import's changed-document set
(``ImportSummary.changed_paths``) into
``generate_folder_summaries(changed_docs=…)``, so ONLY the folders a
changed document touches — each document's folder chain up to its
source root (``affected_folder_keys``) — are re-stamped; every other
stored row keeps its summary text AND ``updated_at`` byte-identical.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock (the phase-94 ``FOLDER_SUMMARY_MODE`` canned
``Fixture folder summary for <folder>.`` lines): the stored rows are a
pure function of the request, so the byte-identical assertions are
exact only against the mock.
No slow-LLM proxy: the assertions are on the POST-terminal DB state
(the rows' ``summary`` AND ``updated_at``) plus the terminal status
(the keep-final-counts convention — the terminal keeps
``summaries_done == summaries_total == <affected count>``), never on
in-flight sampling, so the fast mock is dense enough by construction.
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
the same host) registered as a ``kind=local`` source (the
``test_local_directory_sources.py`` / phase-94 API-registration
pattern; no git anywhere): ``incsum/`` with FOUR ≥ 2-doc folders —
``alpha/``, ``bravo/``, ``charlie/``, ``delta/`` (2 docs each) — plus
ONE root-level doc (9 documents total). The generator's candidate set
is therefore exactly 5: the source root (``folder_path ""``) + the
four folders (the recursive-subtree ≥ 1 rule).
Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the module app, then truncate the shared registry + KB
tables (documents/chunks/query_log/steering_notes/kb_overview/
git_sources/folder_summaries) — each test seeds its own source and
runs its own sync(s); after each test the same truncate. (The temp
tree itself persists for the module; the tests' disk mutations are
ordered by test definition and each test re-syncs the state it finds.)
Test → observable mapping (Playwright Mapping Rule):
1. ``test_first_sync_generates_all_rows`` — a first (full) sync settles
``success``; the ``folder_summaries`` table holds EXACTLY the 5
rows (source root + the four folders) with the mock's canned text
naming each folder, each ``updated_at`` non-null, and the terminal
keeps ``summaries_done == summaries_total == 5`` (the full
regeneration — a changed KB with no prior rows, the pre-127
behavior preserved end to end).
2. ``test_one_file_change_restamps_only_that_folders_chain`` — re-seed
(autouse truncated) + sync #1, capture every row's
``(summary, updated_at)``; change ONE doc inside ``alpha/`` and
sync #2 settles ``success``: the ``alpha`` AND the source-root rows
carry a NEW ``updated_at`` (strictly after the captured stamp) with
the canned text, while the ``bravo``/``charlie``/``delta`` rows are
BYTE-IDENTICAL (same ``summary`` AND same ``updated_at``); the
terminal keeps ``summaries_done == summaries_total == 2`` (the
AFFECTED count — alpha + root, NOT the table's candidate count —
the keep-final-counts convention) and the detail shows the
single-file change (1 updated, 8 unchanged, 0 added/pruned).
3. ``test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_
folder`` — re-seed + sync #1, capture; delete ``bravo/``'s last
docs (the folder VANISHES — 0 documents, no longer a candidate) and
change one doc in ``charlie/``; the sync settles ``success``: the
``bravo`` row is PRUNED (gone — the prune pass owns vanished
folders' rows), the source-root AND ``charlie`` rows are re-stamped
(new ``updated_at`` + canned text), the ``alpha``/``delta`` rows
are byte-identical, and the terminal keeps
``summaries_done == summaries_total == 2`` (the affected CANDIDATE
count — charlie + root; bravo is no longer a candidate).
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pytest
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import FolderSummary
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_INCSUM", "8147"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------
#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "incsum"
#: The four ≥ 2-doc folders (path order).
FOLDER_A = "alpha"
FOLDER_B = "bravo"
FOLDER_C = "charlie"
FOLDER_D = "delta"
FOLDERS = (FOLDER_A, FOLDER_B, FOLDER_C, FOLDER_D)
ROOT_NOTE = "root-note.md" # the one root-level doc
N_FILES = 9 # four folders × 2 docs + the root-level doc
N_CANDIDATES = 5 # the source root + the four folders (the ≥ 1-doc rule)
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
#: tail: ``<source>`` for the root, ``<source>/<folder>`` for a folder).
SUMMARY_FOR = "Fixture folder summary for {}."
EXPECTED: dict[str, str] = {
"": SUMMARY_FOR.format(SOURCE),
FOLDER_A: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_A}"),
FOLDER_B: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_B}"),
FOLDER_C: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_C}"),
FOLDER_D: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_D}"),
}
#: The single-file change (test 2) — a revised body whose hash differs
#: from the fixture's (the importer's sha256 delta reports ``updated``)
#: and which no other fixture text contains.
ALPHA_A_REVISED = (
"Incsum alpha fixture note A, REVISED: the only file test 2 changed "
"— the incremental regeneration must touch exactly this file's chain "
"(alpha + the source root), and nothing else. (RESE- INCSUM-01)"
)
#: The charlie change (test 3) — the same shape on the other side.
CHARLIE_A_REVISED = (
"Incsum charlie fixture note A, REVISED: the one changed file of "
"test 3's sync, while bravo vanishes entirely. (RESE- INCSUM-02)"
)
def _md(title: str, body: str) -> str:
return f"# {title}\n\n{body}\n"
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def incsum_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it. The
directory NAME is the source name (``kind=local``, phase 38)."""
root = tmp_path_factory.mktemp("bor_incsum")
src = root / SOURCE
for folder in FOLDERS:
(src / folder).mkdir(parents=True)
for letter in ("a", "b"):
(src / folder / f"{folder}-{letter}.md").write_text(
_md(
f"Incsum {folder.title()} {letter.upper()}",
f"Incsum {folder} fixture note {letter.upper()}: covers "
f"topic {letter.upper()} of the {SOURCE} source tree.",
),
encoding="utf-8",
)
(src / ROOT_NOTE).write_text(
_md(
"Incsum Root Note",
f"This file sits directly under the {SOURCE} source, not in "
"any folder.",
),
encoding="utf-8",
)
assert len(list(src.rglob("*.md"))) == N_FILES
return src
def _app_env(
base_url: str, tmp_path_factory: pytest.TempPathFactory
) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_kb_tree.py`` / ``test_sync_summary_visibility.py``): the
``base_url`` is the mock LLM directly (the fast leg is the only
leg — see the module docstring), ``BOR_GIT_SOURCES`` forced empty
(the sync source is this suite's own DB-registered local row), and
the leak-guarded code defaults."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated threshold (conftest pattern): this suite never
# asks the chat model anything — the gate is never on a path.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
return env
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
incsum_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The app under test (all three tests): ``BOR_LLM_BASE_URL`` is
the mock directly — the canned ``FOLDER_SUMMARY_MODE`` lines are
byte-stable there, and the byte-identity assertions key on that
exact text (no slow proxy: the assertions are post-terminal, see
the module docstring). ``incsum_dir`` is a dependency only for the
fixture ordering (the temp tree exists before the app boots — the
sync reads it)."""
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the candidate count and the counts the syncs and the
folder-rows assertions pin exactly."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test seeds its own source and runs its own sync(s), so the
KB rows + folder_summaries rows of the temp source are exactly
this test's own. The pre-wait keeps a leaked run from the
previous test (the module app is shared) from racing the
truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": path}
)
assert r.status_code == 201, r.text
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
run reaches a terminal state (the ``test_local_directory_sources``
pattern, over plain httpx — with the stale-terminal freshness gate:
the module app is shared, so the PREVIOUS run's terminal lives in
the app's memory until this run supersedes it; a terminal is
accepted only after a running tick was observed or its
``started_at`` post-dates this POST)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
t0 = time.time()
r = client.post("/api/sync")
assert r.status_code == 202, r.text
saw_running = False
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
state = body["state"]
if state == "running":
saw_running = True
elif state in ("success", "failed") and (
saw_running
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
):
return body
time.sleep(0.1)
raise AssertionError(f"sync did not reach a terminal state: {body}")
def _started_at(body: dict[str, Any]) -> float:
"""The status body's ``started_at`` as a wall-clock epoch (the
server stamps it in UTC; this process runs on the same host —
same clock, the phase-64 recorder's freshness convention)."""
started = body.get("started_at")
if not started:
return 0.0
try:
return datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return 0.0
def _folder_rows() -> dict[str, tuple[str, bool, datetime]]:
"""The source's stored folder summaries
``{folder_path: (summary, manually_edited, updated_at)}``
(``""`` = the source root) — the test process's direct DB access
(the E2E's other established lens)."""
with SessionLocal() as db:
rows = db.execute(
select(
FolderSummary.folder_path,
FolderSummary.summary,
FolderSummary.manually_edited,
FolderSummary.updated_at,
).where(FolderSummary.source == SOURCE)
).all()
return {
folder: (summary, manually, updated)
for folder, summary, manually, updated in rows
}
# ---------------------------------------------------------------------------
# 1. The first sync (a changed KB with no prior rows) is the FULL
# regeneration: exactly the 5 candidate rows, canned text, the
# terminal keeps the full candidate count
# ---------------------------------------------------------------------------
def test_first_sync_generates_all_rows(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""A first sync of the temp source settles ``success`` with all 9
docs added; the ``folder_summaries`` table holds EXACTLY the 5
candidate rows (the source root + the four folders — the
recursive-subtree ≥ 1 rule) with the mock's canned text naming
each folder, every row AI-written and freshly stamped, and the
terminal keeps ``summaries_done == summaries_total == 5`` (the
full-regeneration count — a first sync passes no prior rows, so
this is the pre-127 behavior preserved end to end)."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == N_FILES, detail
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the FULL candidate
# count — the pre-127 shape a first sync still reports.
assert body["summaries_done"] == N_CANDIDATES, body
assert body["summaries_total"] == N_CANDIDATES, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
# The DB pins the full regeneration: exactly one AI row per
# candidate, the deterministic canned line, a fresh stamp.
rows = _folder_rows()
assert set(rows) == {""} | set(FOLDERS), rows
for folder, (summary, manually, updated_at) in rows.items():
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at is not None, (folder, updated_at)
# ---------------------------------------------------------------------------
# 2. ONE changed file in alpha/ → exactly alpha's chain (alpha + the
# source root) is re-stamped; every other row is byte-identical
# (text AND updated_at); the terminal keeps the AFFECTED count
# ---------------------------------------------------------------------------
def test_one_file_change_restamps_only_that_folders_chain(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""Re-seed (the autouse truncate) + sync #1 (full), capturing every
row's ``(summary, updated_at)``; then ONE doc inside ``alpha/``
changes content and sync #2 settles ``success``: the ``alpha`` AND
the source-root rows carry a NEW ``updated_at`` (strictly after
the captured stamp) with the canned text, while the
``bravo``/``charlie``/``delta`` rows are byte-identical — same
``summary`` AND same ``updated_at`` (an unaffected folder's
documents, and hence its summary, are untouched — no re-stamp).
The terminal keeps ``summaries_done == summaries_total == 2`` (the
AFFECTED count — alpha + root, NOT the table's candidate count —
the keep-final-counts convention), and the detail shows the
single-file change."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == N_FILES, body["detail"]
before = _folder_rows()
assert set(before) == {""} | set(FOLDERS), before
# The single-file change: ONE alpha doc gets new content (the
# importer's sha256 delta reports ``updated`` — no add, no prune).
target = incsum_dir / FOLDER_A / f"{FOLDER_A}-a.md"
target.write_text(
_md("Incsum Alpha A (revised)", ALPHA_A_REVISED), encoding="utf-8"
)
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == 0, detail
assert detail["updated"] == 1, detail
assert detail["unchanged"] == N_FILES - 1, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the AFFECTED count —
# alpha + the source root — the phase-127 observable at the wire.
assert body["summaries_done"] == 2, body
assert body["summaries_total"] == 2, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
after = _folder_rows()
assert set(after) == {""} | set(FOLDERS), after
# The chain (alpha + the source root): RE-STAMPED — a new stamp
# strictly after the captured one, the canned text, still
# AI-written (the mock labels from the folder header only, so the
# text is the same deterministic line).
for folder in ("", FOLDER_A):
summary, manually, updated_at = after[folder]
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at > before[folder][2], (
folder,
before[folder],
after[folder],
)
# The siblings: BYTE-IDENTICAL — same summary text AND same
# updated_at (never re-stamped, never even re-fetched for
# generation).
for folder in FOLDERS:
if folder == FOLDER_A:
continue
assert after[folder] == before[folder], (
folder,
before[folder],
after[folder],
)
# ---------------------------------------------------------------------------
# 3. A vanished folder (bravo/ loses its last docs) has its row PRUNED
# by the prune pass; the changed sibling's chain (charlie + the
# source root) is re-stamped; the other rows are byte-identical
# ---------------------------------------------------------------------------
def test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_folder(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""Re-seed + sync #1 (full), capturing every row; then ``bravo/``
loses its last docs (the folder VANISHES — 0 documents, no longer
a candidate) and ONE doc in ``charlie/`` changes content; the sync
settles ``success``: the ``bravo`` row is PRUNED (gone — the
prune pass owns vanished folders' rows, independent of the
generation's affected set), the source-root AND ``charlie`` rows
are re-stamped (new ``updated_at`` + canned text), the
``alpha``/``delta`` rows are byte-identical (text AND
``updated_at``), and the terminal keeps
``summaries_done == summaries_total == 2`` (the affected
CANDIDATE count — charlie + root; bravo is no longer a candidate,
so it is neither generated nor counted)."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == N_FILES, body["detail"]
before = _folder_rows()
assert set(before) == {""} | set(FOLDERS), before
# bravo/ loses its last docs — the folder vanishes from the
# catalogue (0 documents → below the ≥ 1 minimum → not a
# candidate → the prune pass owns its row).
for letter in ("a", "b"):
(incsum_dir / FOLDER_B / f"{FOLDER_B}-{letter}.md").unlink()
# One charlie doc gets new content (``updated``).
target = incsum_dir / FOLDER_C / f"{FOLDER_C}-a.md"
target.write_text(
_md("Incsum Charlie A (revised)", CHARLIE_A_REVISED), encoding="utf-8"
)
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == 0, detail
assert detail["updated"] == 1, detail
assert detail["unchanged"] == N_FILES - 3, detail
assert detail["pruned"] == 2, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the affected
# CANDIDATE count — charlie + the source root (bravo is gone from
# the candidates, so it is not in the affected set's intersection).
assert body["summaries_done"] == 2, body
assert body["summaries_total"] == 2, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
after = _folder_rows()
# The vanished folder's row is PRUNED (the prune pass — it runs in
# both modes, manual rows exempt — bravo's row is AI-written).
assert FOLDER_B not in after, after
assert set(after) == ({""} | set(FOLDERS)) - {FOLDER_B}, after
# charlie + the source root: RE-STAMPED — a new stamp strictly
# after the captured one, the canned text, still AI-written.
for folder in ("", FOLDER_C):
summary, manually, updated_at = after[folder]
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at > before[folder][2], (
folder,
before[folder],
after[folder],
)
# alpha + delta: BYTE-IDENTICAL — same summary text AND same
# updated_at (neither folder is in the changed set's chains).
for folder in (FOLDER_A, FOLDER_D):
assert after[folder] == before[folder], (
folder,
before[folder],
after[folder],
)
tests/integration/test_sync_api.py
+275 -2
View File
@@ -64,6 +64,17 @@ background sync task lives on the app's event loop, so the loop must
survive across requests — exactly how the app runs under uvicorn.
(A TestClient without the context manager starts a fresh loop per
request and would cancel the task on request exit.)
Phase 127 (task 03): the changed-KB branch passes the import's change
set to the generator (``changed_docs=summary.changed_paths`` —
incremental regeneration over the changed set's folder keys only),
the ``changed=N`` log line names the change count, and the status's
``summaries_total`` is the affected count (the hook's total) rather
than the table's candidate count. The unchanged-walk gap probe is
orthogonal: it still calls the generator exactly as before
(``only_missing=True``, no change set). The fakes above record the
``changed_docs=`` kwarg (``FakeFolderSummaries.changed_docs_sets``) so
the wiring is asserted, not inferred.
"""
from __future__ import annotations
@@ -71,7 +82,7 @@ import asyncio
import logging
import threading
import time
from collections.abc import Callable, Iterator
from collections.abc import Callable, Iterator, Sequence
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -85,8 +96,9 @@ from app.api import sync as sync_api
from app.config import Settings
from app.db import SessionLocal, db_available
from app.main import app as fastapi_app
from app.models import GitSource
from app.models import Document, GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.folder_summaries import affected_folder_keys
from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
from app.rag.sources_meta import current_sources_version
@@ -330,6 +342,11 @@ class FakeFolderSummaries:
hook when given — the same way the import's file counter is driven
through its hook, so the status's summary counters are testable
deterministically.
Phase 127 (task 03): records the ``changed_docs=`` change set the
runner passes on the CHANGED-KB branch (the import's
``changed_paths`` — None when the runner regresses to the
change-set-less full regeneration; the unchanged-walk gap-fill
branch never passes it).
"""
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
@@ -346,6 +363,7 @@ class FakeFolderSummaries:
self.skip_flags: list[bool] = []
self.only_missing_flags: list[bool] = []
self.progress_hooks: list[Callable[[int, int, str, str], None] | None] = []
self.changed_docs_sets: list[Sequence[tuple[str, str]] | None] = []
async def __call__(
self,
@@ -354,10 +372,12 @@ class FakeFolderSummaries:
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
self.skip_flags.append(skip)
self.only_missing_flags.append(only_missing)
self.changed_docs_sets.append(changed_docs)
self.progress_hooks.append(on_progress)
if skip:
return dict(self.ZERO)
@@ -663,6 +683,7 @@ class _GatedFolderSummaries:
self.steps_before = steps_before
self.steps_after = steps_after
self.hook: Callable[[int, int, str, str], None] | None = None
self.changed_docs: Sequence[tuple[str, str]] | None = None
async def __call__(
self,
@@ -671,10 +692,12 @@ class _GatedFolderSummaries:
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
assert not skip, "these runs take a generation branch, never --limit"
self.hook = on_progress
self.changed_docs = changed_docs
for done, total, source, folder_path in self.steps_before:
if on_progress is not None:
on_progress(done, total, source, folder_path)
@@ -822,6 +845,7 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
self.hook = on_progress
@@ -845,6 +869,255 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
assert failing.hook is not None # the wiring held
# --- phase 127 (task 03): the changed-KB branch's incremental wiring ----
def test_changed_sync_passes_the_change_set_to_the_generator(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Phase 127 (task 03): a changed-KB sync passes the import's
change set into the generator (``changed_docs=summary.
changed_paths`` — the incremental-regression restriction), NOT the
gap-fill flag, the success ``detail`` keeps its exact pre-phase key
set (the stats stay log-only), and the log line names the change
count (``changed=N``)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
# One updated document — the import's change set carries exactly
# its (source, path) identity (task 01's contract).
change_set = [("repo", "notes/deep.md")]
summary = ImportSummary(
files=4, added=0, updated=1, unchanged=3, changed_paths=change_set
)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
fake_folders = FakeFolderSummaries()
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The CHANGED-KB branch fired the generator exactly once ... with
# the change set, NOT the gap-fill flag (the two restriction
# kwargs are orthogonal — no caller combines them).
assert len(fake_folders.changed_docs_sets) == 1
assert fake_folders.changed_docs_sets == [change_set]
assert fake_folders.only_missing_flags == [False]
assert fake_folders.skip_flags == [False]
# The success detail is byte-identical in shape to pre-phase 127 —
# no new key (the stats stay log-only, the phase-98 convention).
assert set(body["detail"]) == {
"files", "added", "updated", "unchanged", "pruned", "errors",
"chunks", "summaries", "summary_errors", "dates_updated",
"overview", "sources_version",
}
assert body["detail"]["added"] == 0 and body["detail"]["updated"] == 1
assert body["detail"]["overview"] is True
assert body["detail"]["sources_version"] == 1 # changed KB → one bump
# The ample-logging line names the change count (PLAN §9).
assert any(
r.getMessage() == (
"sync: folder_summaries changed=1 "
"stats={'generated': 0, 'failed': 0, 'pruned': 0}"
)
for r in caplog.records
)
def test_unchanged_walk_gap_probe_passes_no_change_set(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 127 (task 03): the unchanged-walk gap probe is
ORTHOGONAL to the change set — a walk that changed nothing with a
real gap (seeded catalogue, empty ``folder_summaries`` table) still
calls the generator exactly as before: ``only_missing=True`` and
NO ``changed_docs`` (the gap probe's total is the missing count,
as before)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
# The gap: two docs under notes/ (candidates root + notes), the
# folder_summaries table empty (the autouse fixture truncated it).
db.add_all([
Document(
source="repo", path="notes/one.md", full_path="/tmp/one.md",
title="One", content="# One\nfirst\n", content_hash="a" * 64,
),
Document(
source="repo", path="notes/two.md", full_path="/tmp/two.md",
title="Two", content="# Two\nsecond\n", content_hash="b" * 64,
),
])
db.commit()
summary = ImportSummary(files=2, added=0, updated=0, unchanged=2)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
fake_folders = FakeFolderSummaries()
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The gap probe fired the generator exactly once — the gap-fill
# shape, byte-identical to pre-phase 127: only_missing=True, and
# NO change set (the gap is a stored-rows gap, not a document
# change — passing changed_paths here would restrict the fill to
# nothing).
assert len(fake_folders.changed_docs_sets) == 1
assert fake_folders.changed_docs_sets == [None]
assert fake_folders.only_missing_flags == [True]
assert body["detail"]["overview"] is False # unchanged → no overview
assert body["detail"]["sources_version"] == 0 # unchanged → no bump
class _AffectedKeysFolderSummaries:
"""A folder-summary step that mirrors the REAL generator's hook
contract on a changed set (phase 127, task 02): the loop runs over
the sorted AFFECTED keys (``affected_folder_keys(changed_docs)``),
the hook fires once per key with ``total == len(keys)``, and the
step parks mid-span after the first key — the (long) phase the
user reported. The test reads the status mid-span and asserts the
status's ``summaries_total`` IS the affected count (the hook's
total), not the table's candidate count."""
def __init__(
self,
started: threading.Event,
release: threading.Event,
) -> None:
self.started = started
self.release = release
self.changed_docs: Sequence[tuple[str, str]] | None = None
self.keys: list[tuple[str, str]] = []
self.hook: Callable[[int, int, str, str], None] | None = None
async def __call__(
self,
db: Session,
llm: LLMClient,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
assert not skip, "these runs take a generation branch, never --limit"
self.hook = on_progress
self.changed_docs = changed_docs
assert on_progress is not None, "the runner must wire the hook"
# The real generator's step 2: sorted(candidates & affected) —
# the fake takes the affected side (the candidates are a
# superset here: every affected folder holds a doc).
self.keys = sorted(affected_folder_keys(changed_docs or []))
assert self.keys, "the change set must affect at least the root"
on_progress(1, len(self.keys), self.keys[0][0], self.keys[0][1])
self.started.set()
await asyncio.to_thread(self.release.wait)
for i, (source, folder_path) in enumerate(self.keys[1:], start=2):
on_progress(i, len(self.keys), source, folder_path)
return {"generated": len(self.keys), "failed": 0, "pruned": 0}
def test_summaries_phase_reports_the_affected_count_as_total(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 127 (task 03): mid-span, the status's ``summaries_total``
is the AFFECTED count — ``len(affected_folder_keys(changed set))``
(the hook's total), not the table's candidate count — and the
terminal keeps that final count (the phase-64 convention, now over
the affected span). The change set: one nested-folder doc (a
three-key chain) + one root-level doc (shares the root key) →
exactly 3 affected keys."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
change_set = [("repo", "notes/a/one.md"), ("repo", "top.md")]
fake_import = FakeImportSources(
ImportSummary(
files=6, added=1, updated=2, unchanged=3, changed_paths=change_set
)
)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
gated_folders = _AffectedKeysFolderSummaries(
threading.Event(), threading.Event()
)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
try:
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
s = sync_client.get("/api/sync/status").json()
assert s["state"] == "running"
assert s["phase"] == "summaries"
# The hook's first key: the source root (the bare source name —
# folder_path "" never gets a slash) ...
assert s["current_summary"] == "repo"
assert s["summaries_done"] == 1
# ... and the total IS the affected count (the task's pin):
# notes/a/one.md → ("repo", ""), ("repo", "notes"),
# ("repo", "notes/a"); top.md → ("repo", "") — 3 keys.
expected = sorted(affected_folder_keys(change_set))
assert expected == [("repo", ""), ("repo", "notes"), ("repo", "notes/a")]
assert s["summaries_total"] == len(expected)
assert s["summaries_total"] == 3
gated_folders.release.set()
finally:
gated_folders.release.set()
body = _poll(sync_client, "success")
# Terminal: phase + current_summary cleared, the hook's FINAL
# (affected) counts kept (the phase-64 keep-final-counts rule, now
# over the affected span).
assert body["phase"] is None and body["current_summary"] is None
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
# Wiring: the runner passed the change set, not the gap-fill flag.
assert gated_folders.changed_docs == change_set
assert gated_folders.hook is not None
# --- admin: failures -------------------------------------------------------
tests/unit/test_folder_summaries.py
+380
View File
@@ -29,6 +29,7 @@ from app.rag.folder_summaries import (
FOLDER_SUMMARY_MODE,
MIN_DOCS_PER_FOLDER,
SYSTEM_PROMPT,
affected_folder_keys,
build_folder_summary_prompt,
folder_of,
generate_folder_summaries,
@@ -202,6 +203,56 @@ def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
assert "top.md" not in [folder for _source, folder in groups]
# ---------- affected_folder_keys (phase 127, task 02) ----------
def test_affected_folder_keys_root_level_file_is_only_the_root() -> None:
"""A root-level file (``folder_of == ""``) affects ONLY the source
root — there is no folder prefix to walk."""
assert affected_folder_keys([("S", "a.md")]) == {("S", "")}
def test_affected_folder_keys_nested_file_walks_the_folder_chain() -> None:
"""A doc under ``a/b/`` affects ``a/b``, ``a``, and the source
root — the SAME prefix walk ``group_by_folder`` pass 1 uses (one
concept)."""
assert affected_folder_keys([("S", "a/b/c.md")]) == {
("S", ""),
("S", "a"),
("S", "a/b"),
}
def test_affected_folder_keys_multi_source_set_semantics() -> None:
"""Keys are per source (the PK is ``(source, folder_path)``), and
the result is a SET — a repeated chain adds no new keys, and an
unrelated source keeps its own root."""
keys = affected_folder_keys(
[
("S", "a/b/c.md"),
("T", "x.md"),
("S", "a/d.md"), # the same chain as the first entry — no new keys
]
)
assert keys == {("S", ""), ("S", "a"), ("S", "a/b"), ("T", "")}
def test_affected_folder_keys_fragment_paths_fold_into_their_parent_folder() -> None:
"""Embedded-image fragment paths (phase 126: ``x.pdf#page=…#img=…``)
get NO special case: the fragment never holds a ``/``, so
``folder_of`` folds them into the parent folder's chain (a
root-level fragment affects only the source root)."""
assert affected_folder_keys([("S", "dir/img.pdf#page=2#img=0")]) == {
("S", ""),
("S", "dir"),
}
assert affected_folder_keys([("S", "img.pdf#page=1#img=0")]) == {("S", "")}
def test_affected_folder_keys_empty_input_is_an_empty_set() -> None:
assert affected_folder_keys([]) == set()
# ---------- build_folder_summary_prompt: system ----------
@@ -1093,3 +1144,332 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
("FSU", "a/b"): REPLY,
("FSU-solo", ""): REPLY,
}
# ---------- generate_folder_summaries(changed_docs=…) (phase 127, task 02) ----------
def _seed_three_sources(db: Session) -> None:
"""The restricted-mode catalogue (three sources): A is nested
(``a/`` and ``a/b/``), B is a sibling single-folder source, and C
is a single-doc root — six candidate keys total: A ×
{``""``, ``a``, ``a/b``}, B × {``""``, ``b``}, C × {``""``}.
"""
_add_doc(db, "A", "a/b/one.md", "A One", "A one lead.")
_add_doc(db, "A", "a/b/two.md", "A Two")
_add_doc(db, "A", "a/three.md", "A Three")
_add_doc(db, "B", "b/one.md", "B One")
_add_doc(db, "C", "solo.md", "C Solo")
def _stamps(
db: Session, keys: set[tuple[str, str]]
) -> dict[tuple[str, str], object]:
"""The stored rows' ``updated_at`` for *keys* (raw SQL — the
byte-identity comparisons stay honest)."""
return {
(source, folder): _updated_at(db, source, folder) for source, folder in keys
}
def _freeze(
db: Session, values: dict[tuple[str, str], str]
) -> dict[tuple[str, str], object]:
"""Overwrite the stored summaries for *values*' keys with DISTINCT
frozen text (a "regenerated" row is then distinguishable from an
"untouched" one), and return the pre-run stamps."""
for (source, folder_path), value in values.items():
db.execute(
text(
"UPDATE folder_summaries SET summary = :v"
" WHERE source = :s AND folder_path = :f"
),
{"v": value, "s": source, "f": folder_path},
)
db.commit()
return _stamps(db, set(values))
def test_changed_docs_restricted_regeneration_only_affected_keys(
db: Session, clean_tables
) -> None:
"""One changed file in A/``a/b/`` → exactly that file's folder
chain (``a/b``, ``a``, and the A root) regenerates: one lite call
per affected non-manual key, in sorted order, fresh rows. Every
OTHER stored row — same-source siblings and other sources — keeps
its summary text AND ``updated_at`` byte-identical (zero lite
calls for them)."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
frozen = {
("A", ""): "A root frozen",
("A", "a"): "A/a frozen",
("A", "a/b"): "A/a/b frozen",
("B", ""): "B root frozen",
("B", "b"): "B/b frozen",
("C", ""): "C root frozen",
}
stamps = _freeze(db, frozen)
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3, "one lite call per AFFECTED non-manual key — nothing else"
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a",
"Folder: A/a/b",
], "the affected chain in sorted (source, folder_path) order"
stored = _rows(db)
assert stored[("A", "")] == REPLY
assert stored[("A", "a")] == REPLY
assert stored[("A", "a/b")] == REPLY, "the changed file's own folder regenerates"
assert stored[("B", "")] == "B root frozen"
assert stored[("B", "b")] == "B/b frozen"
assert stored[("C", "")] == "C root frozen", (
"the unaffected rows keep their stored text"
)
for key in (("B", ""), ("B", "b"), ("C", "")):
assert _updated_at(db, *key) == stamps[key], (
"the unaffected rows are never re-stamped"
)
def test_changed_docs_manual_affected_row_is_skipped(db: Session, clean_tables) -> None:
"""A manual row INSIDE the affected chain is skipped exactly as in
full mode (phase 97): ``kept_manual``, no lite call, text AND
``updated_at`` byte-identical — the restriction changes the key
set, not the manual-row rule."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
manual_text = "Owner's words about A/a."
row = db.get(FolderSummary, ("A", "a"))
assert row is not None
row.summary = manual_text
row.manually_edited = True
db.commit()
stamp_before = _updated_at(db, "A", "a")
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 2, "failed": 0, "pruned": 0, "kept_manual": 1}
assert llm.calls == 2, "the manual key burns zero lite calls"
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a/b",
]
stored = _rows(db)
assert stored[("A", "a")] == manual_text, "the owner's text is never rewritten"
assert _updated_at(db, "A", "a") == stamp_before, ("never re-stamped")
assert stored[("A", "")] == REPLY and stored[("A", "a/b")] == REPLY
assert stored[("B", "")] == REPLY and stored[("B", "b")] == REPLY
assert stored[("C", "")] == REPLY, "the unaffected rows are left as they were"
def test_changed_docs_failing_affected_folder_fails_soft(db: Session, clean_tables) -> None:
"""Per-folder fail-soft under the restriction: the failing
affected key is attempted and counted, its PREVIOUS row is kept
(an old summary is better than none), the other affected keys
still land, and the unaffected rows are untouched."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
# A DISTINCT previous text on the failing folder, so "kept" is
# visible (it would otherwise be the REPLY either way).
db.execute(
text(
"UPDATE folder_summaries SET summary = 'old A/a/b'"
" WHERE (source, folder_path) = ('A', 'a/b')"
)
)
db.commit()
stamps = _stamps(db, {("B", ""), ("B", "b"), ("C", "")})
llm = _FakeLLM(fail_folders=("A/a/b",))
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3, "the failing key is attempted (the hook's premise)"
stored = _rows(db)
assert stored[("A", "a/b")] == "old A/a/b", "the previous row survives"
assert stored[("A", "")] == REPLY and stored[("A", "a")] == REPLY
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
def test_changed_docs_multi_source_change_set(db: Session, clean_tables) -> None:
"""A change set spanning SOURCES: the affected keys are the UNION
of each document's chain (A's nested chain + B's root and ``b``),
sorted across sources; the unaffected C root row keeps its text
AND ``updated_at``."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
c_stamp = _updated_at(db, "C", "")
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(
db, llm, changed_docs=[("A", "a/b/two.md"), ("B", "b/one.md")]
)
)
assert stats == {"generated": 5, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 5
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a",
"Folder: A/a/b",
"Folder: B",
"Folder: B/b",
]
stored = _rows(db)
assert stored[("C", "")] == REPLY
assert _updated_at(db, "C", "") == c_stamp
def test_changed_docs_gap_in_affected_folder_is_generated(
db: Session, clean_tables
) -> None:
"""An affected folder with NO stored row (a gap in the chain) is a
candidate and gets generated like any other affected key — while a
gap in an UNAFFECTED folder stays the unchanged-walk gap probe's
job (phase 96): the restriction never widens beyond the affected
set."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
db.execute(
text(
"DELETE FROM folder_summaries"
" WHERE (source, folder_path) IN (('A', 'a/b'), ('B', ''))"
)
)
db.commit()
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3
stored = _rows(db)
assert stored[("A", "a/b")] == REPLY, "the gap in the affected chain is filled"
assert ("B", "") not in stored, (
"the gap in the UNAFFECTED source is left for the gap probe"
)
assert stored[("B", "b")] == REPLY
def test_changed_docs_vanished_affected_folder_is_pruned_not_generated(
db: Session, clean_tables
) -> None:
"""The ASSUMPTION pinned: an affected folder that no longer holds
any document is a PRUNE-pass job, not a generation's — the pruned
document's chain (root + its now-vanished folder) is in the change
set, only the surviving candidate (the root) generates, the
vanished folder's row is dropped, and the unaffected rows keep
their ``updated_at``."""
_add_doc(db, "A", "a/b/one.md", "A One")
_add_doc(db, "A", "a/b/two.md", "A Two")
_add_doc(db, "A", "gone/old.md", "Gone")
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
stamps = _stamps(db, {("A", "a"), ("A", "a/b")})
db.execute(text("DELETE FROM documents WHERE path = 'gone/old.md'"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "gone/old.md")])
)
# affected = {("A", ""), ("A", "gone")} — the only surviving
# candidate in that set is the source root.
assert stats == {"generated": 1, "failed": 0, "pruned": 1, "kept_manual": 0}
assert llm.calls == 1
assert [user.splitlines()[0] for _s, user in llm.requests] == ["Folder: A"]
stored = _rows(db)
assert ("A", "gone") not in stored, "the vanished folder's row is pruned"
assert stored[("A", "")] == REPLY, "the root's subtree changed — regenerated"
assert stored[("A", "a")] == REPLY and stored[("A", "a/b")] == REPLY
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
def test_changed_docs_hook_total_is_the_affected_count(
db: Session, clean_tables
) -> None:
"""The progress hook (phase 98) under the restriction fires exactly
``len(affected)`` times with ``total == len(affected)`` — the sync
status's ``summaries_total`` then reports the affected count
through the existing hook — in sorted key order."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
llm = _FakeLLM()
events, record = _record_progress()
stats = asyncio.run(
generate_folder_summaries(
db, llm, changed_docs=[("A", "a/b/one.md")], on_progress=record
)
)
assert events == [
(1, 3, "A", ""),
(2, 3, "A", "a"),
(3, 3, "A", "a/b"),
]
assert llm.calls == 3
assert stats["generated"] == 3
def test_changed_docs_none_is_the_pre_127_full_regeneration(
db: Session, clean_tables
) -> None:
"""The default (no ``changed_docs`` kwarg) is the pre-127 FULL
regeneration over the same three-source catalogue: every
candidate key, one call each, the 4-field stats shape unchanged —
the ``scripts/import_docs.py`` CLI contract."""
_seed_three_sources(db)
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 6, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 6, "no kwarg → every candidate burns one call"
assert set(_rows(db)) == {
("A", ""), ("A", "a"), ("A", "a/b"),
("B", ""), ("B", "b"),
("C", ""),
}
def test_changed_docs_empty_set_generates_nothing_but_still_prunes(
db: Session, clean_tables
) -> None:
"""``changed_docs=[]`` (a change set that IS given but empty)
restricts the generation to zero keys — no lite calls, no
re-stamps, the hook never fires — while the prune pass still runs
in the given mode (the sweep is mode-independent, phase 97)."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
db.add(FolderSummary(source="A", folder_path="gone/old", summary="stale"))
db.commit()
stamps = _stamps(db, {("A", ""), ("A", "a"), ("A", "a/b"), ("B", ""), ("C", "")})
llm = _FakeLLM()
events, record = _record_progress()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[], on_progress=record)
)
assert stats == {"generated": 0, "failed": 0, "pruned": 1, "kept_manual": 0}
assert llm.calls == 0
assert events == [], "zero keys → the hook never fires"
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "no row is re-stamped"
tests/unit/test_importer.py
+236
View File
@@ -15,9 +15,11 @@ from __future__ import annotations
import asyncio
import logging
import os
from datetime import UTC, datetime
from pathlib import Path
import pymupdf as fitz
import pytest
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -28,6 +30,7 @@ from app.models import Chunk, Document
from app.rag.importer import (
EXCLUDED_DIRS,
ImportSummary,
_prune,
_store_summary,
import_sources,
iter_importable_files,
@@ -1194,3 +1197,236 @@ def test_progress_hook_with_limit_keeps_full_total(db, tmp_path: Path) -> None:
assert events == [(root.name, "a.md", 1, 3), (root.name, "b.md", 2, 3)]
finally:
_cleanup_source(db, root.name)
# ---------- phase 127 (task 01): ImportSummary.changed_paths ----------
class _VisionLLM(FakeEmbedder):
"""The phase-122 mock VISION client (the test_pdf pattern): the
multimodal describe call (``user`` content = a part list) gets a fixed
description; text (``lite``) calls keep the ``FakeEmbedder``
behaviour."""
DESCRIPTION = "A test image: a solid swatch of color."
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
return self.DESCRIPTION
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def _png_bytes(width: int, height: int, rgb: tuple[float, float, float]) -> bytes:
"""One solid-color PNG built with PyMuPDF (the phase-126 no-binary-
fixtures convention)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
for y in range(height):
for x in range(width):
pix.set_pixel(x, y, rgb)
return pix.tobytes("png")
def _build_one_page_pdf(path: Path, text: str) -> None:
"""One temp single-page PDF (the phase-126 convention — no binary
fixtures)."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), text)
doc.save(path)
doc.close()
def test_import_summary_changed_paths_defaults_empty() -> None:
"""Completion criterion: default construction gives an empty change
set — no existing construction site migrates."""
assert ImportSummary().changed_paths == []
def test_changed_paths_records_added_updated_and_pruned_in_order(
db, tmp_path: Path
) -> None:
"""The contract (phase 127, task 01): one ``(source, path)`` entry per
ADDED, UPDATED, or PRUNED document — added/updated in walk order, the
pruned pair last (``_prune`` runs after the walk) — and
``len(changed_paths) == added + updated + pruned``."""
root = tmp_path / "changed"
root.mkdir()
for name in ("a.md", "b.md", "c.md"):
(root / name).write_text(f"# {name}\n\nv1\n")
llm = FakeEmbedder()
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 3 and first.pruned == 0
assert first.changed_paths == [
(source, "a.md"),
(source, "b.md"),
(source, "c.md"),
]
# Run 2: update a.md, add d.md + e.md, delete c.md (→ pruned).
(root / "a.md").write_text("# a\n\nv2 changed\n")
(root / "d.md").write_text("# d\n\nnew d\n")
(root / "e.md").write_text("# e\n\nnew e\n")
(root / "c.md").unlink()
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert (second.added, second.updated, second.unchanged, second.pruned) == (
2, 1, 1, 1,
)
assert second.changed_paths == [
(source, "a.md"), # updated — first in walk order
(source, "d.md"), # added
(source, "e.md"), # added
(source, "c.md"), # pruned — the _prune pass runs after the walk
]
assert len(second.changed_paths) == (
second.added + second.updated + second.pruned
)
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_unchanged_rerun(db, tmp_path: Path) -> None:
"""An all-unchanged re-run (prune included) records nothing."""
root = tmp_path / "nochange"
root.mkdir()
(root / "a.md").write_text("# a\n\nstable\n")
(root / "b.md").write_text("# b\n\nstable\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert second.unchanged == 2
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_backfill_rerun(db, tmp_path: Path) -> None:
"""A backfill (phase 118: unchanged doc, NULL summary) is NOT a content
change: the add still records, the backfill re-run stays empty."""
root = tmp_path / "backfill"
root.mkdir()
(root / "a.yaml").write_text("alpha x\n")
try:
first = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
assert first.added == 1 and first.summary_errors == 1
assert first.changed_paths == [(root.name, "a.yaml")] # the ADD records
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "a.yaml")
)
assert doc is not None and doc.summary is None
second = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
assert second.unchanged == 1 and second.summary_backfilled == 1
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_date_only_rerun(db, tmp_path: Path) -> None:
"""A date-only refresh (phase 106, D4: content untouched, mtime moved)
is NOT a content change: the re-run records nothing."""
root = tmp_path / "dateonly"
root.mkdir()
path = root / "a.md"
path.write_text("# a\n\nstable\n")
os.utime(path, (0, datetime(2020, 1, 1, tzinfo=UTC).timestamp()))
llm = FakeEmbedder()
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1 and first.dates_updated == 0
assert first.changed_paths == [(root.name, "a.md")]
# The content is untouched — only the mtime moved (the D4
# date-refresh on the unchanged path).
os.utime(path, (0, datetime(2021, 6, 15, tzinfo=UTC).timestamp()))
second = asyncio.run(import_sources([root], llm, session=db))
assert second.unchanged == 1 and second.dates_updated == 1
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_records_a_changed_image(db, tmp_path: Path) -> None:
"""The image increment site (phase 122 path): a NEW image and a
byte-CHANGED image each record their ``(source, rel)``."""
root = tmp_path / "imgsrc"
root.mkdir()
(root / "pic.png").write_bytes(_png_bytes(1, 1, (1.0, 0.0, 0.0)))
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
)
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1 and first.images_failed == 0
assert first.changed_paths == [(source, "pic.png")]
# The BYTES changed (a new color) — the image digest is over the
# bytes, so this is an UPDATE.
(root / "pic.png").write_bytes(_png_bytes(1, 1, (0.0, 1.0, 1.0)))
second = asyncio.run(import_sources([root], llm, session=db))
assert second.updated == 1
assert second.changed_paths == [(source, "pic.png")]
finally:
_cleanup_source(db, root.name)
def test_changed_paths_records_a_changed_pdf(db, tmp_path: Path) -> None:
"""The PDF increment site (phase 126 path): a NEW PDF and a text-
CHANGED PDF each record their ``(source, rel)``."""
root = tmp_path / "pdfsrc"
root.mkdir()
pdf_path = root / "doc.pdf"
_build_one_page_pdf(pdf_path, "alpha page one")
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
import_extensions="md,txt,pdf",
)
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1
assert first.changed_paths == [(source, "doc.pdf")]
# The extracted TEXT changed — the PDF digest is over it, so this
# is an UPDATE.
_build_one_page_pdf(pdf_path, "bravo page one, revised")
second = asyncio.run(import_sources([root], llm, session=db))
assert second.updated == 1
assert second.changed_paths == [(source, "doc.pdf")]
finally:
_cleanup_source(db, root.name)
def test_prune_records_changed_paths_only_when_summary_given(db, tmp_path: Path) -> None:
"""The ``_prune`` recording seam: with NO summary (the pre-phase
count-only contract) nothing is recorded; with one, each ACTUALLY
deleted doc appends its ``(source, path)`` (skipped docs never do)."""
root = tmp_path / "prunit"
root.mkdir()
(root / "gone.md").write_text("# gone\n\nbye\n")
(root / "gone2.md").write_text("# gone2\n\nbye\n")
llm = FakeEmbedder()
try:
source = root.name
asyncio.run(import_sources([root], llm, session=db))
(root / "gone.md").unlink()
(root / "gone2.md").unlink()
# Default call (summary=None): count only, no recording.
pruned = _prune(db, {source}, {(source, "gone2.md")})
assert pruned == 1
assert db.scalar(
select(Document).where(Document.source == source, Document.path == "gone.md")
) is None
# With a summary: the deleted doc records its pair.
counters = ImportSummary()
pruned = _prune(db, {source}, set(), summary=counters)
assert pruned == 1
assert counters.changed_paths == [(source, "gone2.md")]
finally:
_cleanup_source(db, root.name)
tests/unit/test_sync_button.py
+5 -2
View File
@@ -40,7 +40,7 @@ from __future__ import annotations
import asyncio
import re
import threading
from collections.abc import Callable, Iterator
from collections.abc import Callable, Iterator, Sequence
from datetime import datetime
from pathlib import Path
@@ -819,12 +819,15 @@ def _patch_sync_seams(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
# Phase 98 (task 01): the runner wires the progress hook into
# the generation branches — the fake accepts it (the state-
# machine tests' canned run never fires it: zero summary
# counters, like a no-gap skip).
# counters, like a no-gap skip). Phase 127 (task 03): the
# changed-KB branch passes the change set as changed_docs= —
# the fake accepts it (these canned runs record nothing).
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)