diff --git a/.agents/phases/todo/127_sync_incremental_folder_summaries/00_phase.md b/.agents/phases/complete/127_sync_incremental_folder_summaries/00_phase.md similarity index 100% rename from .agents/phases/todo/127_sync_incremental_folder_summaries/00_phase.md rename to .agents/phases/complete/127_sync_incremental_folder_summaries/00_phase.md diff --git a/.agents/phases/todo/127_sync_incremental_folder_summaries/01_importer_changed_paths.md b/.agents/phases/complete/127_sync_incremental_folder_summaries/01_importer_changed_paths.md similarity index 100% rename from .agents/phases/todo/127_sync_incremental_folder_summaries/01_importer_changed_paths.md rename to .agents/phases/complete/127_sync_incremental_folder_summaries/01_importer_changed_paths.md diff --git a/.agents/phases/todo/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md b/.agents/phases/complete/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md similarity index 100% rename from .agents/phases/todo/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md rename to .agents/phases/complete/127_sync_incremental_folder_summaries/02_folder_summaries_affected_mode.md diff --git a/.agents/phases/todo/127_sync_incremental_folder_summaries/03_sync_wiring.md b/.agents/phases/complete/127_sync_incremental_folder_summaries/03_sync_wiring.md similarity index 100% rename from .agents/phases/todo/127_sync_incremental_folder_summaries/03_sync_wiring.md rename to .agents/phases/complete/127_sync_incremental_folder_summaries/03_sync_wiring.md diff --git a/.agents/phases/todo/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md b/.agents/phases/complete/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md similarity index 100% rename from .agents/phases/todo/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md rename to .agents/phases/complete/127_sync_incremental_folder_summaries/04_e2e_incremental_sync.md diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.md new file mode 100644 index 0000000..dc006ec --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.md @@ -0,0 +1,10 @@ +All verification complete. Final report: + +**Phase 127 — final verification pass: all green** (all 4 tasks were already in `complete/`; no code changes were needed this pass) + +- **Verified implementation:** `ImportSummary.changed_paths` recorded at all 5 added/updated sites + `_prune` (before delete, guards intact); `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys = candidates ∩ affected, prune pass both modes, hook total = affected count); sync's changed-KB branch passes `changed_docs=summary.changed_paths` with the `changed=N` log line; CLI (`scripts/import_docs.py`) passes no change set — full-regen contract preserved +- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, 0 errors +- **Completion criteria:** ① one-file change rewrites only the folder chain (E2E pins sibling rows byte-identical, text AND `updated_at`) ✅ ② coverage >90% ✅ ③ E2E isolated green ✅ ④ no completed-phase behavior change (full suite + first-sync full-regen E2E) ✅ ⑤ atomic commit — delegated to the harness per this pass's rules (I make no git changes) +- **Docs:** README.md — fixed 7 pre-existing inconsistencies: missing Theme row in the pages table; stale `ls` (now drill-down tree, phase 94) and `read` (now capped at `BOR_READ_MAX_CHARS`, phase 95) tool descriptions; missing phase-124 answer images (new section + `show_image` tool row + `BOR_ANSWER_IMAGES` env row); "Document summaries (non-markdown)" → all docs incl. markdown (phase 118); theme palette 8 → 17 colors (phases 92/93); `BOR_LLM_SUMMARY_MODEL` now names folder summaries; CLI section notes overview + folder-summary regen. AGENTS.md — verified consistent, no changes needed +- **Deviations:** none. Untracked pre-existing `NOTES.md` left untouched (owner's notes, will ride in the harness commit) +- **Next pending phase:** `128_sync_label_mobile` diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.validate new file mode 100644 index 0000000..dcee24e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a1.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.md new file mode 100644 index 0000000..38c157e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.md @@ -0,0 +1,10 @@ +All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact. + +**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass) + +- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract +- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors** +- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me) +- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed +- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit) +- **Next pending phase:** `128_sync_label_mobile` diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.validate new file mode 100644 index 0000000..dcee24e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__00_phase.a2.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.md new file mode 100644 index 0000000..b0f3c3b --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.md @@ -0,0 +1,16 @@ +Everything is green. Final state verified: + +- **`app/rag/importer.py`**: `ImportSummary.changed_paths` field (task-specified docstring, `field(default_factory=list)`); `changed_paths.append` beside **every** `added`/`updated` increment — all 9 sites across the 5 verb blocks (text, PDF, PDF-embedded image, image-context enrichment, standalone image) — and in `_prune` (new `summary: ImportSummary | None = None` param, appended before `session.delete`, only for docs actually deleted past both guards); only the `import_sources` call site updated; `log()` untouched. +- **`tests/unit/test_importer.py`**: 8 new tests — mixed 2-added/1-updated/1-pruned run with exact order + `len == added+updated+pruned`, unchanged re-run empty, backfill re-run empty, date-only re-run empty, changed image (added+updated sites), changed PDF (both sites), `_prune` with/without summary, and default-construction `== []`. + +**Results:** +- `uv run pytest --cov=app --cov-report=term-missing` → 3005 passed, exit 0, TOTAL 99% (gate >90% ✅; `app/rag/importer.py` 98%, remaining misses pre-existing) +- `uv run ruff check .` → All checks passed +- `uv run pyright` → 0 errors (fixed two `# pyright: ignore[reportCallIssue]` placements) +- Grep audit: 9 increment sites ↔ 9 matching appends ✅ + +**Docs:** no doc impact (internal record-tracking only — no commands, config, env vars, or API surface changed; README/AGENTS.md untouched). + +**Decisions:** appended right after each verb if/else (single line, same encounter order as the counter); enrichment path records `(source, image_doc.path)` since its docstring already states it fires the KB-change gate; PNG/PDF fixtures generated in-test with PyMuPDF (house no-binary-fixtures convention). + +**Next pending task:** `02_folder_summaries_affected_mode.md` diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.validate new file mode 100644 index 0000000..73420ca --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__01_importer_changed_paths.a1.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 43%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 64%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 79%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 98%] +..................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 123 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5122 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.md new file mode 100644 index 0000000..dca7ba7 --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.md @@ -0,0 +1,12 @@ +Everything is green. The unit run shows all dots through [100%] with exit 0. + +**Task 02 complete — report** + +- **Implemented** in `app/rag/folder_summaries.py`: + - `affected_folder_keys(changed)` pure helper (next to `folder_of`/`group_by_folder`) — per changed doc: source-root key + `folder_of` prefix walk; fragment paths fold in naturally; empty input → empty set + - `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg: `None` = byte-identical full regen (CLI stays on it, verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys)`; manual skip, fail-soft, `on_progress` (total = affected count), and the prune pass are untouched code; docstrings (module + function + progress) updated +- **Tests** (`tests/unit/test_folder_summaries.py`, +380 lines): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts (call log + fresh rows), unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs +- **Results:** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed**, exit 0; `app/` coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit -q` → exit 0, all green; `uv run ruff check .` → passed; `uv run pyright` → 0 errors +- **Docs:** no doc impact (README sync description remains accurate; no new env/API/commands) +- **Decisions:** intersection with `candidates` (per task ASSUMPTION) — vanished affected folders are the prune pass's job; no caller combines `changed_docs` + `only_missing` (documented in the docstring) +- **Next pending task:** `03_sync_wiring.md` (sync's changed-KB branch passes `summary.changed_paths`) diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.validate new file mode 100644 index 0000000..b1df8c1 --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a1.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.md new file mode 100644 index 0000000..25a606d --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.md @@ -0,0 +1,12 @@ +All gates are green. The previous rejection was an infrastructural one (the child run ended without emitting its final report) — no test failures existed; the working tree already contained the completed task. I re-verified everything end-to-end this run: full suite with coverage, the target unit files, and lint + types. + +**Task 02 complete — report** + +- **Implemented** in `app/rag/folder_summaries.py` (unchanged from prior run, verified intact): + - `affected_folder_keys(changed)` pure helper — per changed doc: source-root key `(source, "")` + `folder_of` prefix walk (same concept as `group_by_folder` pass 1); fragment paths (`x.pdf#page=…`) fold into parent chain; empty input → empty set + - `generate_folder_summaries(..., changed_docs=None, ...)` keyword-only kwarg — `None` = byte-identical full regen (CLI verified unmodified at `scripts/import_docs.py:465`); given = `sorted(candidates ∩ affected_folder_keys(changed_docs))`; manual skip / fail-soft / `on_progress` (total = affected count) / prune pass untouched; module + function docstrings updated +- **Tests** (`tests/unit/test_folder_summaries.py`, 14 new): helper (root/nested/multi-source/fragment/empty) + restricted mode over a 3-source catalogue — only-affected upserts, unaffected rows byte-identical text **AND** `updated_at`, manual affected row skipped, fail-soft keeps previous row, gap-in-affected generated / gap-unaffected left to probe, vanished affected folder pruned-not-generated, hook total = affected count, `changed_docs=None` full-regen stats shape, `changed_docs=[]` no-gen but prune still runs +- **Results (this run):** `uv run pytest --cov=app --cov-report=term-missing` → **3019 passed, exit 0**; `app/` TOTAL coverage **99%** (>90%), `folder_summaries.py` **100%**; `uv run pytest tests/unit/test_folder_summaries.py tests/unit/test_importer.py -q` → exit 0; `uv run ruff check .` → all passed; `uv run pyright` → 0 errors, 0 warnings +- **Docs:** no doc impact (README sync section stays accurate; no new env/API/commands) +- **Decisions:** intersection with `candidates` per task ASSUMPTION — vanished affected folders are the prune pass's job; `changed_docs` + `only_missing` documented as never combined by any caller +- **Next pending task:** `03_sync_wiring.md` — the sync's changed-KB branch passes `summary.changed_paths`; docstrings + logging diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.validate new file mode 100644 index 0000000..b1df8c1 --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__02_folder_summaries_affected_mode.a2.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 62%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.md new file mode 100644 index 0000000..4a8a93a --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.md @@ -0,0 +1,20 @@ +Everything is verified green. The previous run completed all of task 03's work but its validation was interrupted by a KeyboardInterrupt; I re-verified every gate from scratch and confirmed the implementation matches the task spec exactly. + +**Task 03 — Sync wiring: complete (verified)** + +- `app/api/sync.py`: changed-KB branch calls `generate_folder_summaries(fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook)`; else-arm (gap probe, `only_missing=True`) byte-identical; log line now `sync: folder_summaries changed=%d stats=%s` +- Docstrings updated to the incremental contract: module docstring step-5 + phase-machine paragraph (incl. stale "at least 2 docs" → "at least 1 doc" fix) and `SyncStatus` `summaries_done/total` note (total = affected count on changed KB, missing count on gap-fill) +- `detail` dict (12 keys) and `GET /api/sync/status` shape unchanged — asserted in tests +- Fakes extended to accept/record `changed_docs=`: `tests/integration/test_sync_api.py` (`FakeFolderSummaries`, `_GatedFolderSummaries`, `_HookThenFail`) + `tests/unit/test_sync_button.py` +- 3 new tests in `tests/integration/test_sync_api.py`: change-set wiring + `changed=N` log + detail shape; unchanged-walk gap probe passes `only_missing=True` with `changed_docs=None`; mid-span status `summaries_total == len(affected_folder_keys(...))` — all 3 pass individually (3 passed, 22 deselected) + +**Results:** +- `uv run pytest tests/unit -q` → 2326 passed +- `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, app/ **99%** (sync.py 100%, folder_summaries.py 100%, importer.py 98%) — gate >90% ✅ +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings + +**Docs:** no doc impact (no setup/commands/config/env/API changes; README never described the regeneration mode, AGENTS.md untouched). + +**Note:** prune-only syncs still take the gap-probe branch (pre-existing `added+updated>0` gate untouched, per task scope). `NOTES.md` (owner's untracked note) left in place. + +**Next pending task:** `04_e2e_incremental_sync.md` (dedicated Playwright E2E in isolation) diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.validate new file mode 100644 index 0000000..dcee24e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__03_sync_wiring.a1.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.md new file mode 100644 index 0000000..5802719 --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.md @@ -0,0 +1,18 @@ +All green. Final report: + +**Implemented** +- New `tests/e2e/test_incremental_folder_summaries.py` — dedicated mock-only Playwright suite (phase 127 task 04), house pattern from `test_sync_summary_visibility.py`: module app on port 8147 (env-overridable), host temp-dir `kind=local` source `incsum/` (4 folders × 2 docs + 1 root doc = 9 files, 5 candidates), autouse truncate + settle-wait cleanup, stale-terminal freshness gate in `_run_sync`. +- 3 tests: (1) first full sync → exactly 5 canned rows, terminal keeps 5/5; (2) one `alpha/` file changed → only `alpha` + source root re-stamped (strictly newer `updated_at`), `bravo`/`charlie`/`delta` byte-identical (text AND `updated_at`), terminal keeps `summaries_done == summaries_total == 2`; (3) `bravo/` vanishes → its row pruned by the prune pass, `charlie` + root re-stamped, `alpha`/`delta` byte-identical, terminal 2/2. + +**Gates (all 100% green)** +- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed, in isolation (run twice; DB up) +- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%) +- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors + +**Docs** +- README.md: added step 4 to the "Sync from the UI" pipeline (folder summaries: incremental on changed KB, gap-fill on unchanged) — it predated folder summaries entirely (stale since phase 94). AGENTS.md: no impact. + +**Decisions** +- "bravo's last doc" read as *the docs whose deletion makes the folder vanish* (0 docs → not a candidate); both `bravo/` files deleted, since a 1-doc folder is still a candidate and would not be pruned. All other assertions match the task verbatim. + +**Next pending task:** none — task 04 is the last task of phase 127. diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.validate new file mode 100644 index 0000000..dcee24e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a1.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.err b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.md b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.md new file mode 100644 index 0000000..9704c82 --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.md @@ -0,0 +1,20 @@ +All checks are green and consistent. The documentation is accurate: + +- **README.md** — the step-4 addition from the prior run is intact and matches the implemented behavior (incremental on changed KB via the changed set's folder chains, gap-fill on unchanged walks). The "Updating the documents" CLI section makes no regeneration-scope claims (CLI keeps full regen — consistent). +- **AGENTS.md** — no references to the sync pipeline internals; operating rules, test gates, and conventions all still accurate. No changes needed. + +All gates re-verified from scratch this run (the prior a2 attempt was an infrastructural interruption — it died mid-run without a report, leaving no port or DB residue): + +**What was verified/fixed this run** +- Confirmed working tree intact from the successful a1 run: `tests/e2e/test_incremental_folder_summaries.py` (3 tests: first-full-sync 5 canned rows @ terminal 5/5; one `alpha/` file → only `alpha`+root re-stamped, siblings byte-identical text AND `updated_at`, terminal 2/2; `bravo/` vanishes → row pruned, `charlie`+root re-stamped, terminal 2/2) plus tasks 01–03 code/tests. +- No code changes required — re-ran every gate fresh; found no defects, no leftovers (ports clean, DB truncated). +- Doc consistency check: README step-4 accurate; AGENTS.md untouched. + +**Results (this run)** +- `uv run pytest --cov=app --cov-report=term-missing` → 3022 passed; `app/` coverage 99% (>90%) +- `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → 3 passed in isolation (run twice) +- `uv run ruff check .` → all passed; `uv run pyright` → 0 errors + +**Notable decision** (unchanged from a1): "bravo's last doc" = the docs whose deletion makes the folder vanish (both deleted; a 1-doc folder is still a candidate and wouldn't prune). + +**Next pending task:** none — task 04 is the last task of phase 127. diff --git a/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.validate b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.validate new file mode 100644 index 0000000..dcee24e --- /dev/null +++ b/.agents/reports/127_sync_incremental_folder_summaries/127_sync_incremental_folder_summaries__04_e2e_incremental_sync.a2.validate @@ -0,0 +1,113 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 244 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5133 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/README.md b/README.md index f1f7ad8..3517ae2 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,7 @@ client-side router. Direct URLs deep-link to the matching view. | **Tuning** | `/tuning.html` | Admin only | | **Saved chats** | `/history.html` | Admin only | | **Access tokens** | `/tokens.html` | Admin only | +| **Theme** | `/theme.html` | Admin only | | **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins | | **Login** | `/login.html` | Everyone | | **Shared chat** | `/shared/` | Everyone (anonymous) | @@ -207,6 +208,10 @@ reflect the new files. - With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif, bmp`) are indexed as documents too — see [Image support](#image-support). +- A run that changed the KB also regenerates the **KB overview** and the + **folder summaries** (the drill-down catalog's stored per-folder + descriptions) with the `lite` model — best-effort: a summary failure + never fails the import. ### Git-based sources @@ -262,6 +267,12 @@ whole git-source refresh in one click: 2. **re-import with prune** — files deleted upstream leave the index. 3. **regenerate the KB overview** (the `` outline every chat turn injects) — only when the import changed the knowledge base. +4. **refresh the folder summaries** (the drill-down catalog's stored + per-folder descriptions) — a changed knowledge base rewrites ONLY the + folders the changed files touch (each file's folder chain up to its + source root — phase 127: one changed file re-stamps a handful of + rows, not the whole table); an unchanged re-sync instead gap-fills + any missing rows (`only_missing`). - States: the button shows **Syncing…** with a spinner while polling `GET /api/sync/status` every 2 s. No client-side timeout — a clone + @@ -288,13 +299,16 @@ To hide it, set `BOR_STREAM_THINKING=0`. Retrieval only puts the top documents in context. When an answer depends on a file a note *references* ("the exact JSON shape is in `example-record-file.json`"), the model can extend its own context with -three server-side tools — on **grounded** (high-relevance) turns only: +server-side tools — on **grounded** (high-relevance) turns only: | Tool | Description | |------|-------------| -| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source | -| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string | +| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` | +| **`read(path)`** | Appends one indexed document's text to context by its combined `source/path`; very large documents are truncated at `BOR_READ_MAX_CHARS` with a notice that points at `grep` to find the rest | | **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document | +| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer | + +(See [Answer images](#answer-images-phase-124) for the `show_image` tool.) Each call is executed against Postgres only (no extra LLM round trip) and streamed as an SSE `tool` frame. In the chat, each call shows a transient @@ -361,12 +375,14 @@ docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees. --- -## Document summaries (non-markdown) +## Document summaries Raw yaml/json/py/txt embeds badly — flags and keys are not language, so retrieval can miss exactly the documents that are all configuration. At -import time, every **non-markdown** A9 document is summarized by the aipi -`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`): +import time, **every** new or changed document gets a `lite`-model summary — +**markdown included**: the summary is a natural-language retrieval target +and the seed text for the onboarding suggestion blocks (phase 118 retired +the old "markdown is already natural language" exclusion): * The summary is stored on `documents.summary` **and** indexed as one extra embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a @@ -452,6 +468,30 @@ image: A question image is **never indexed as a document** — it is turn-local storage, not a source. +### Answer images (phase 124) + +A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when +on, the grounded chat model may show a standalone image document as part +of its answer: + +- **The model decides.** The agent is offered a fourth tool, + `show_image(path)`, and calls it only when the image itself contains + the answer (a diagram, screenshot, or photo of the thing asked about) — + never for mere topical relevance. At most **one** image per answer + (enforced server-side); the text answer is always given in full — the + image supplements, never replaces. +- **Rendered as an answer figure.** A shown image appears as a + first-class figure below the answer text and above the source chips — + on the live turn, after a page reload, and on the shared chat page; a + failed load collapses the figure (never a broken-image icon). +- **Image chips on zero-read turns.** A retrieved image document in the + suggested tier is auto-cited: its source chip (with figure) appears + even when the model never `read`s it — the model's seed block already + carries the image's full description, so a real model has no reason to + read it (this applies with `BOR_IMAGES=1`, any `BOR_ANSWER_IMAGES`). +- **Off = byte-identical.** No fourth tool offered, no figure, no + `answer_image` key in any SSE frame or saved record. + --- ## Caching & deploys @@ -515,7 +555,7 @@ to `.env` and adjust. | `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key | | `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` | | `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model | -| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview | +| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries | | `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) | | `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off | | `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries | @@ -578,7 +618,7 @@ to `.env` and adjust. | `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) | | `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) | -### Image support (phases 122/123) +### Image support (phases 122/123/124) | Env | Default | Meaning | |-----|---------|---------| @@ -587,6 +627,7 @@ to `.env` and adjust. | `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) | | `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) | | `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) | +| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) | ### Docs push @@ -615,10 +656,11 @@ Every identity string is an env var: `BOR_APP_NAME` (display name), applied by `assets/brand.js` at boot. The colors are set from the admin **Theme tab** (`/theme.html`, -admin-only — phase 91): the 8 identity colors plus the three strings -above are edited with text fields and color pickers, persisted in the -`ui_settings` table, and injected into every served page as an inline -`