feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Build and Push Containers / build-and-push-app (push) Successful in 2m10s
Build and Push Containers / build-and-push-db (push) Successful in 13s

Relax the phase-94 folder-summary scope rule from ≥ 2 documents to
≥ 1: a folder (or source root) is a candidate while ANY document
lives under it, so single-file folders and single-file source roots
get their own lite-written description. A row is now pruned only
when its folder loses its last document (vanishes from the
catalogue).

The constant is the single source of truth, so the flip propagates
to the generator's candidate set, the prune pass, the
missing_folder_summaries gap probe (the next sync self-heals the
new gaps), and the KB-tree summary_pending markers (1-doc folders /
sources now read "Summary pending" until their row lands).

Docstrings/comments across app/, scripts/import_docs.py, and the
E2E fixtures updated to the ≥ 1 wording. Unit + integration tests
updated to the new semantics (the pruned-below-minimum scenario is
now a folder losing its LAST doc; single-doc folders are pinned as
candidates/pending). Full suite: 2314 passed, app coverage 99%;
ruff + pyright clean; folder-summary E2E stories pass in isolation
(ls_tree_drilldown, sync_summary_visibility, kb_tree,
kb_tree_nav, document_dates, oneshot_llm_retry).
This commit is contained in:
2026-09-14 08:57:57 -04:00
parent 3a81793565
commit 35d65d2f25
13 changed files with 297 additions and 212 deletions
+10 -8
View File
@@ -576,7 +576,7 @@ def build_kb_tree(
* **Pending** — ``summary_pending`` on the SOURCE and every FOLDER * **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
node (phase 98, decision D3 — ONE concept): true iff the node's node (phase 98, decision D3 — ONE concept): true iff the node's
recursive ``documents`` count ≥ recursive ``documents`` count ≥
:data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (2) AND it :data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (1) AND it
has NO stored ``folder_summaries`` row (AI or manual — any row; has NO stored ``folder_summaries`` row (AI or manual — any row;
the builder sees stored rows only). That is EXACTLY the builder sees stored rows only). That is EXACTLY
:func:`app.rag.folder_summaries.missing_folder_summaries`'s :func:`app.rag.folder_summaries.missing_folder_summaries`'s
@@ -584,9 +584,10 @@ def build_kb_tree(
keys on the next sync — the marker is honest: "waiting to keys on the next sync — the marker is honest: "waiting to
generate", and the integration cross-check pins the tree's generate", and the integration cross-check pins the tree's
pending set to that function so the marker can never drift from pending set to that function so the marker can never drift from
the gap-fill). A < 2-document folder is NEVER pending (it never the gap-fill). A 0-document folder cannot exist (a folder is a
gets a summary — its one file line IS its description), and a catalogue prefix only) — every existing folder with no row is
registered 0-document source never is. pending, single-file folders included — and a registered
0-document source never is.
ONE concept end to end: the builder reuses ONE concept end to end: the builder reuses
:func:`app.rag.folder_summaries.folder_of` and the phase-94 :func:`app.rag.folder_summaries.folder_of` and the phase-94
@@ -674,10 +675,11 @@ def list_kb_tree(
exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped
catalogue query ``GET /api/docs`` runs (the document ``id`` catalogue query ``GET /api/docs`` runs (the document ``id``
excluded — the tree has no document ids) + ALL stored excluded — the tree has no document ids) + ALL stored
``folder_summaries`` rows (a bounded select — one row per ≥ 2-doc ``folder_summaries`` rows (a bounded select — one row per
folder; rows for sources the tree does not list are never existing folder at the ≥ 1-doc rule; rows for sources the tree
referenced by the builder) — through the pure does not list are never referenced by the builder) — through the
:func:`build_kb_tree`. ``GET /api/docs`` itself is untouched. pure :func:`build_kb_tree`. ``GET /api/docs`` itself is
untouched.
""" """
names = list_source_names(db) names = list_source_names(db)
rows = db.execute( rows = db.execute(
+2 -1
View File
@@ -353,7 +353,8 @@ async def _run_sync() -> None:
# changed KB (added + updated > 0) is a full regeneration # changed KB (added + updated > 0) is a full regeneration
# (today's behavior, byte-identical); an unchanged re-sync # (today's behavior, byte-identical); an unchanged re-sync
# takes the GAP probe instead of the old table-empty one — a # takes the GAP probe instead of the old table-empty one — a
# candidate folder (at least 2 docs) with no stored row: a gap # candidate folder (at least 1 doc — every existing folder)
# with no stored row: a gap
# fills ONLY the missing rows (only_missing=True — the # fills ONLY the missing rows (only_missing=True — the
# subsumed table-empty first-sync trigger included, where # subsumed table-empty first-sync trigger included, where
# every candidate is missing), a complete table burns zero # every candidate is missing), a complete table burns zero
+13 -11
View File
@@ -65,8 +65,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
(``id = 1``); every column NULL = "use the (``id = 1``); every column NULL = "use the
default" (env value for the strings, the built-in default" (env value for the strings, the built-in
palette for the colors — task 01). palette for the colors — task 01).
* ``folder_summaries`` — one row per folder with ≥ 2 documents: * ``folder_summaries`` — one row per folder (recursive count
the sync-time ``lite`` summary the drill-down ≥ 1 — every existing folder): the sync-time
``lite`` summary the drill-down
``ls`` shows next to each folder (phase 94; ``ls`` shows next to each folder (phase 94;
``folder_path = ""`` = the source root). ``folder_path = ""`` = the source root).
""" """
@@ -501,14 +502,15 @@ class FolderSummary(Base):
``documents.path`` (String(1000)) and is the source-relative ``documents.path`` (String(1000)) and is the source-relative
folder prefix. ``folder_path = ""`` is the SOURCE ROOT — the folder prefix. ``folder_path = ""`` is the SOURCE ROOT — the
top-level source summary (the whole source's recursive subtree). top-level source summary (the whole source's recursive subtree).
* Rows exist only for folders with ≥ 2 documents (recursive count * Rows exist for folders with ≥ 1 document (recursive count —
— the same set the ``ls`` count rule counts): a the same set the ``ls`` count rule counts): a folder is a
single-document folder is fully described by its one file line, candidate while any document lives under it, single-file folders
so no ``lite`` burn. After a changed sync, rows whose folder and single-file source roots included. After a changed sync,
dropped below 2 documents are pruned (a pruned/renamed folder's rows whose folder dropped below 1 document (vanished from the
summary would otherwise go stale); rows for folders that still catalogue) are pruned (a pruned/renamed folder's summary would
have ≥ 2 documents persist (an unchanged folder's summary is otherwise go stale); rows for folders that still hold at least
still true). Both rules are generator policy (``app.rag. one document persist (an unchanged folder's summary is still
true). Both rules are generator policy (``app.rag.
folder_summaries``), not schema constraints. folder_summaries``), not schema constraints.
* ``summary`` — the 1–3 sentence plain-text description the aipi * ``summary`` — the 1–3 sentence plain-text description the aipi
``lite`` model wrote at sync time (``FOLDER_SUMMARY_MODE``, ``lite`` model wrote at sync time (``FOLDER_SUMMARY_MODE``,
@@ -521,7 +523,7 @@ class FolderSummary(Base):
folder_summaries``) SKIPS a manual row on regeneration (no folder_summaries``) SKIPS a manual row on regeneration (no
``lite`` burn on owner text — counted ``kept_manual``) and never ``lite`` burn on owner text — counted ``kept_manual``) and never
prunes it (owner content persists until cleared — even for a prunes it (owner content persists until cleared — even for a
folder below the 2-document minimum); an owner correction is folder below the 1-document minimum); an owner correction is
never silently rewritten (the phase-97 ``00_phase.md`` decision). never silently rewritten (the phase-97 ``00_phase.md`` decision).
Clearing the description deletes the row — the next KB-changing Clearing the description deletes the row — the next KB-changing
sync regenerates an AI description (the reset path). sync regenerates an AI description (the reset path).
+35 -22
View File
@@ -20,13 +20,16 @@ summary describes.
Storage: ``folder_summaries`` (migration 0017) — PK Storage: ``folder_summaries`` (migration 0017) — PK
``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT ``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT
(the top-level source summary). Rows exist only for folders with (the top-level source summary). Rows exist for every folder whose
≥ 2 documents (the :data:`MIN_DOCS_PER_FOLDER` rule — a recursive subtree holds ≥ 1 document (the
single-document folder is fully described by its one file line, so no :data:`MIN_DOCS_PER_FOLDER` rule — a folder is a catalogue prefix
``lite`` burn); after a changed sync, rows whose folder dropped below only while some document lives under it, so EVERY existing folder is
2 documents are pruned (a pruned/renamed folder's summary would a candidate, single-file folders and single-file source roots
otherwise go stale), while rows for folders that still have included); after a changed sync, rows whose folder dropped below 1
≥ 2 documents persist (an unchanged folder's summary is still true). document (vanished from the catalogue) are pruned (a pruned/renamed
folder's summary would otherwise go stale), while rows for folders
that still hold at least one document persist (an unchanged folder's
summary is still true).
Manual rows (phase 97, task 01): ``manually_edited`` (migration 0018) Manual rows (phase 97, task 01): ``manually_edited`` (migration 0018)
marks the descriptions the OWNER edited — ``PATCH /api/folders/ marks the descriptions the OWNER edited — ``PATCH /api/folders/
@@ -104,10 +107,13 @@ SYSTEM_PROMPT = f"{FOLDER_SUMMARY_MODE}: {FOLDER_SUMMARY_INSTRUCTION}"
FOLDER_HEADER_PREFIX = "Folder: " FOLDER_HEADER_PREFIX = "Folder: "
#: A folder is summarized only while its recursive subtree holds at #: A folder is summarized only while its recursive subtree holds at
#: least this many documents — a single-document folder is fully #: least this many documents — 1: EVERY folder the catalogue knows
#: described by its one file line, so no ``lite`` burn (phase 94 #: (a folder exists only while some document lives under it, so the
#: ``00_phase.md`` scope rule; the prune rule applies the same count). #: filter admits every group :func:`group_by_folder` returns,
MIN_DOCS_PER_FOLDER = 2 #: single-file folders included) gets a row (the phase-94 ≥ 2 scope
#: rule relaxed by owner decision; the prune rule applies the same
#: count — a row is pruned when its folder loses its last document).
MIN_DOCS_PER_FOLDER = 1
#: One document row for the grouping/prompting: #: One document row for the grouping/prompting:
#: ``(source, path, title, summary)`` — the ``app.rag.overview`` #: ``(source, path, title, summary)`` — the ``app.rag.overview``
@@ -164,8 +170,10 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
catalogue query). Returns ``{(source, folder_path): [rows]}`` — catalogue query). Returns ``{(source, folder_path): [rows]}`` —
group lists keep the input (catalogue) order, so downstream prompt group lists keep the input (catalogue) order, so downstream prompt
building is deterministic. Groups of ANY size (≥ 1) are returned; building is deterministic. Groups of ANY size (≥ 1) are returned;
the ≥ 2 :data:`MIN_DOCS_PER_FOLDER` rule is applied by at the :data:`MIN_DOCS_PER_FOLDER` = 1 rule every group is
:func:`generate_folder_summaries`, not here. already a candidate, so :func:`generate_folder_summaries`'s
filter is an inert safety net (it would bite only if the minimum
were ever raised above 1), not a behavioral gate.
""" """
# Pass 1: the distinct TRUE folder prefixes of the catalogue — a # Pass 1: the distinct TRUE folder prefixes of the catalogue — a
@@ -336,7 +344,9 @@ detector and the fill can never disagree about what the catalogue
def _candidates(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]: def _candidates(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
"""The generator's candidate map (recursive subtree ≥ 2 docs). """The generator's candidate map (recursive subtree ≥
:data:`MIN_DOCS_PER_FOLDER` docs — at the current minimum of 1,
every :func:`group_by_folder` group).
The :func:`group_by_folder` groups filtered by The :func:`group_by_folder` groups filtered by
:data:`MIN_DOCS_PER_FOLDER` — the EXACT set a full regeneration :data:`MIN_DOCS_PER_FOLDER` — the EXACT set a full regeneration
@@ -379,10 +389,11 @@ gap persisted until the next KB change. This function names the gap:
Returns the missing keys sorted by ``(source, folder_path)``; Returns the missing keys sorted by ``(source, folder_path)``;
``[]`` when there is no gap — including an empty catalogue over an ``[]`` when there is no gap — including an empty catalogue over an
empty table (no candidates, no gap). A single-document folder is empty table (no candidates, no gap). A stored row for a folder
never listed (it is not a candidate), and a stored row for a folder that dropped below the minimum (vanished — 0 documents) is NOT
that dropped below the minimum is NOT missing (it is stale — the missing (it is stale — the prune pass owns it); a single-document
prune pass owns it). folder without a row IS listed (it is a candidate at the ≥ 1
rule).
""" """
return sorted( return sorted(
key for key in _candidates(_catalog_rows(db)) if key not in _stored_keys(db) key for key in _candidates(_catalog_rows(db)) if key not in _stored_keys(db)
@@ -408,8 +419,9 @@ async def generate_folder_summaries(
2. Group the document catalogue by :func:`group_by_folder` (the 2. Group the document catalogue by :func:`group_by_folder` (the
recursive-subtree concept) and keep the candidate folders — recursive-subtree concept) and keep the candidate folders —
the ones whose recursive subtree holds the ones whose recursive subtree holds
:data:`MIN_DOCS_PER_FOLDER` (≥ 2) documents. Single-document :data:`MIN_DOCS_PER_FOLDER` (1) documents: at the current
folders get no row (their one file line IS their summary). minimum every existing folder (single-file ones included, and
the single-file source root) is a candidate.
3. For each candidate (deterministic ``(source, folder_path)`` 3. For each candidate (deterministic ``(source, folder_path)``
order) call :func:`summarize_folder` and UPSERT — fail-soft PER order) call :func:`summarize_folder` and UPSERT — fail-soft PER
FOLDER: one folder's :class:`LLMError` is logged and counted, FOLDER: one folder's :class:`LLMError` is logged and counted,
@@ -430,8 +442,9 @@ async def generate_folder_summaries(
staleness is the changed-KB regeneration's job) and no ``lite`` staleness is the changed-KB regeneration's job) and no ``lite``
call is burned for a folder that already has a summary — the call is burned for a folder that already has a summary — the
unchanged-sync self-heal fill (phase 96, task 02). unchanged-sync self-heal fill (phase 96, task 02).
4. DELETE rows whose folder no longer has ≥ 2 documents — a 4. DELETE rows whose folder no longer has ≥ 1 document (vanished
pruned/renamed folder's summary goes stale and is dropped — from the catalogue) — a pruned/renamed folder's summary goes
stale and is dropped —
EXCEPT a manual row: owner content persists until the owner EXCEPT a manual row: owner content persists until the owner
clears it, even for a folder that dropped below the minimum clears it, even for a folder that dropped below the minimum
(phase 97, task 01; the clear deletes the row, so the next (phase 97, task 01; the clear deletes the row, so the next
+8 -6
View File
@@ -288,12 +288,13 @@ class KbTreeFolder(BaseModel):
resolves it with ``from __future__ import annotations``). resolves it with ``from __future__ import annotations``).
``summary_pending`` (phase 98, D3 — ONE concept): true iff this ``summary_pending`` (phase 98, D3 — ONE concept): true iff this
folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (2) AND it has folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (1) AND it has
NO stored ``folder_summaries`` row (AI or manual — any row) — NO stored ``folder_summaries`` row (AI or manual — any row) —
exactly the candidate the sync-time gap-fill regenerates (the exactly the candidate the sync-time gap-fill regenerates (the
:func:`app.rag.folder_summaries.missing_folder_summaries` set). :func:`app.rag.folder_summaries.missing_folder_summaries` set).
A < 2-document folder is never pending (it never gets a summary — A 0-document folder cannot exist (a folder is a catalogue prefix
its one file line IS its description). only), so every existing folder with no stored row is pending —
single-file folders included.
""" """
kind: Literal["folder"] = "folder" kind: Literal["folder"] = "folder"
@@ -325,12 +326,13 @@ class KbTreeSource(BaseModel):
``summary_pending`` (phase 98, D3 — ONE concept): true iff the ``summary_pending`` (phase 98, D3 — ONE concept): true iff the
source's recursive ``documents`` count ≥ ``MIN_DOCS_PER_FOLDER`` source's recursive ``documents`` count ≥ ``MIN_DOCS_PER_FOLDER``
(2) AND no stored ``(source, "")`` row (AI or manual — any row) — (1) AND no stored ``(source, "")`` row (AI or manual — any row) —
exactly the source-root candidate the sync-time gap-fill exactly the source-root candidate the sync-time gap-fill
regenerates (the regenerates (the
:func:`app.rag.folder_summaries.missing_folder_summaries` set). :func:`app.rag.folder_summaries.missing_folder_summaries` set).
A registered 0-document source is never pending (there is nothing A registered 0-document source is never pending (there is nothing
to summarize). to summarize); a single-document source IS pending while its root
row is absent.
""" """
name: str name: str
@@ -353,7 +355,7 @@ class KbTree(BaseModel):
Every SOURCE and FOLDER node carries ``summary_pending`` (phase Every SOURCE and FOLDER node carries ``summary_pending`` (phase
98, D3): true iff its recursive document count ≥ 98, D3): true iff its recursive document count ≥
``MIN_DOCS_PER_FOLDER`` (2) AND it has no stored ``folder_summaries`` ``MIN_DOCS_PER_FOLDER`` (1) AND it has no stored ``folder_summaries``
row — exactly ``missing_folder_summaries``'s candidate set (the row — exactly ``missing_folder_summaries``'s candidate set (the
marker never drifts from the gap-fill); FILE nodes carry no flag marker never drifts from the gap-fill); FILE nodes carry no flag
(the file table has no description column). (the file table has no description column).
+7 -4
View File
@@ -68,7 +68,8 @@ The stored folder summaries (phase 94 — the drill-down ``ls``'s
per-level descriptions, the ``folder_summaries`` table) regenerate in per-level descriptions, the ``folder_summaries`` table) regenerate in
the same run under the same gate: a changed KB (added + updated > 0 — the same run under the same gate: a changed KB (added + updated > 0 —
full regeneration), or, after an unchanged walk, a GAP — a candidate full regeneration), or, after an unchanged walk, a GAP — a candidate
folder (≥ 2 docs) with no stored row (phase 96: this subsumes the old folder (≥ 1 doc — every existing folder) with no stored row
(phase 96: this subsumes the old
table-empty trigger exactly — an empty table leaves EVERY candidate table-empty trigger exactly — an empty table leaves EVERY candidate
missing, as after the first full run after migration 0017 or a missing, as after the first full run after migration 0017 or a
``--limit`` first walk that skipped them — and catches the single row ``--limit`` first walk that skipped them — and catches the single row
@@ -282,7 +283,8 @@ def _overview_row_exists() -> bool:
def _folder_summaries_gap() -> list[tuple[str, str]]: def _folder_summaries_gap() -> list[tuple[str, str]]:
"""The folder-summary gaps (phase 96, task 03): the candidate """The folder-summary gaps (phase 96, task 03): the candidate
folders (≥ 2 docs) with no stored row, sorted. folders (≥ 1 doc — every existing folder) with no stored row,
sorted.
The unchanged-walk self-heal trigger, replacing the phase-94 The unchanged-walk self-heal trigger, replacing the phase-94
table-emptiness probe (which the gap subsumes exactly: an empty table-emptiness probe (which the gap subsumes exactly: an empty
@@ -358,8 +360,9 @@ def main(argv: list[str] | None = None) -> int:
The folder summaries (phase 94, task 02; phase 96, task 03) The folder summaries (phase 94, task 02; phase 96, task 03)
follow the same gate — a changed KB (full regeneration), or, follow the same gate — a changed KB (full regeneration), or,
after an unchanged walk, a GAP: a candidate folder (≥ 2 docs) after an unchanged walk, a GAP: a candidate folder (≥ 1 doc —
with no stored row (the subsumed table-empty trigger — an every existing folder) with no stored row (the subsumed
table-empty trigger — an
empty table leaves every candidate missing — plus a row an empty table leaves every candidate missing — plus a row an
exhausted one-shot retry lost) — with per-folder fail-soft exhausted one-shot retry lost) — with per-folder fail-soft
inside the generator (a ``lite`` failure keeps the failed inside the generator (a ``lite`` failure keeps the failed
+10 -5
View File
@@ -26,7 +26,9 @@ real-Sync pattern; no git anywhere):
Total: 7 documents; alpha counts 5 (root + 2 + 2), beta counts 2. Every Total: 7 documents; alpha counts 5 (root + 2 + 2), beta counts 2. Every
stored description is deterministic: the mock's EXISTING stored description is deterministic: the mock's EXISTING
``FOLDER_SUMMARY_MODE`` branch (phase 94 — no mock changes needed) ``FOLDER_SUMMARY_MODE`` branch (phase 94 — no mock changes needed)
stores, per ≥ 2-doc folder, the canned one-liner naming the folder, stores, per existing folder (the ≥ 1 rule — this fixture holds no
single-doc folders, so the pinned rows are unchanged), the canned
one-liner naming the folder,
``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb`` ``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb``
module fixture pins those exact rows (all with module fixture pins those exact rows (all with
``manually_edited = false``) after the sync, and the tests assert on ``manually_edited = false``) after the sync, and the tests assert on
@@ -154,8 +156,9 @@ TOTAL_DOCS = ALPHA_COUNT + BETA_COUNT
#: The sync-time folder descriptions the mock's canned #: The sync-time folder descriptions the mock's canned
#: ``FOLDER_SUMMARY_MODE`` branch stores (the phase-94 byte-stable #: ``FOLDER_SUMMARY_MODE`` branch stores (the phase-94 byte-stable
#: template — the one-liner names the folder), in ``(source, #: template — the one-liner names the folder), in ``(source,
#: folder_path)`` order: one row per ≥ 2-doc folder (the #: folder_path)`` order: one row per existing folder (the ≥ 1-doc
#: recursive-subtree rule) — the ``""`` rows are the source roots. #: recursive-subtree rule — this fixture holds no single-doc folders)
#: — the ``""`` rows are the source roots.
SUMMARY_FOR = "Fixture folder summary for {}." SUMMARY_FOR = "Fixture folder summary for {}."
ALPHA_ROOT_SUM = SUMMARY_FOR.format(ALPHA) ALPHA_ROOT_SUM = SUMMARY_FOR.format(ALPHA)
ONE_SUM = SUMMARY_FOR.format(f"{ALPHA}/one") ONE_SUM = SUMMARY_FOR.format(f"{ALPHA}/one")
@@ -382,7 +385,8 @@ def synced_kb(app_server: str, kb_tree_dirs: tuple[Path, Path]) -> None:
walk → chunk → embed → overview → folder summaries → version walk → chunk → embed → overview → folder summaries → version
bump), and pins the stored folder descriptions: the mock's canned bump), and pins the stored folder descriptions: the mock's canned
``FOLDER_SUMMARY_MODE`` branch (phase 94) makes the sync store one ``FOLDER_SUMMARY_MODE`` branch (phase 94) makes the sync store one
deterministic row per ≥ 2-doc folder — the tests assert on that deterministic row per existing folder (the ≥ 1-doc rule — this
fixture holds no single-doc folders) — the tests assert on that
exact text (and on the phase-97 ``manually_edited`` flag: every exact text (and on the phase-97 ``manually_edited`` flag: every
stored row starts out AI-written). stored row starts out AI-written).
""" """
@@ -409,7 +413,8 @@ def synced_kb(app_server: str, kb_tree_dirs: tuple[Path, Path]) -> None:
assert detail["pruned"] == 0, detail assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail assert detail["overview"] is True, detail
# The change-gated folder descriptions (phase 94) landed: one row # The change-gated folder descriptions (phase 94) landed: one row
# per ≥ 2-doc folder, the mock's byte-stable text — and every row # per existing folder (the ≥ 1-doc rule — this fixture holds no
# single-doc folders), the mock's byte-stable text — and every row
# AI-written (the ``manually_edited`` flag starts false). # AI-written (the ``manually_edited`` flag starts false).
with SessionLocal() as db: with SessionLocal() as db:
rows = db.execute( rows = db.execute(
+9 -5
View File
@@ -42,7 +42,8 @@ of the mock's cosine distribution, and the ``<tools>`` section is present
(the flow's precondition). (the flow's precondition).
The mock's canned ``FOLDER_SUMMARY_MODE`` branch (phase 94 task 01) The mock's canned ``FOLDER_SUMMARY_MODE`` branch (phase 94 task 01)
stores, per ≥ 2-doc folder, the deterministic one-liner stores, per existing folder (the ≥ 1-doc rule — this fixture holds
no single-doc folders), the deterministic one-liner
``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb`` ``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb``
module fixture pins those exact rows in ``folder_summaries`` after the module fixture pins those exact rows in ``folder_summaries`` after the
sync, and the drill answers assert on them in the ``ls`` output. sync, and the drill answers assert on them in the ``ls`` output.
@@ -159,8 +160,9 @@ assert "\n" not in TWO_A_CONTENT[:80] # the quote must stay one line
#: The sync-time folder summaries the mock's canned ``FOLDER_SUMMARY_MODE`` #: The sync-time folder summaries the mock's canned ``FOLDER_SUMMARY_MODE``
#: branch stores (task 01's byte-stable template), in #: branch stores (task 01's byte-stable template), in
#: ``(source, folder_path)`` order: one row per ≥ 2-doc folder (the #: ``(source, folder_path)`` order: one row per existing folder (the
#: recursive-subtree rule) — the ``""`` rows are the source roots. #: ≥ 1-doc recursive-subtree rule — this fixture holds no single-doc
#: folders) — the ``""`` rows are the source roots.
EXPECTED_SUMMARIES: list[tuple[str, str, str]] = [ EXPECTED_SUMMARIES: list[tuple[str, str, str]] = [
(ALPHA, "", f"Fixture folder summary for {ALPHA}."), (ALPHA, "", f"Fixture folder summary for {ALPHA}."),
(ALPHA, "one", f"Fixture folder summary for {ALPHA}/one."), (ALPHA, "one", f"Fixture folder summary for {ALPHA}/one."),
@@ -426,7 +428,8 @@ def synced_kb(app_server: str, drill_dirs: tuple[Path, Path]) -> None:
walk → chunk → embed → overview → folder summaries → version bump), walk → chunk → embed → overview → folder summaries → version bump),
and pins the stored folder summaries: the mock's canned and pins the stored folder summaries: the mock's canned
``FOLDER_SUMMARY_MODE`` branch (task 01) makes the sync store one ``FOLDER_SUMMARY_MODE`` branch (task 01) makes the sync store one
deterministic row per ≥ 2-doc folder — the drill turns' answers deterministic row per existing folder (the ≥ 1-doc rule — this
fixture holds no single-doc folders) — the drill turns' answers
assert on that exact text. assert on that exact text.
""" """
alpha, beta = drill_dirs alpha, beta = drill_dirs
@@ -452,7 +455,8 @@ def synced_kb(app_server: str, drill_dirs: tuple[Path, Path]) -> None:
assert detail["pruned"] == 0, detail assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail assert detail["overview"] is True, detail
# The change-gated folder summaries (phase 94 task 02) landed: one # The change-gated folder summaries (phase 94 task 02) landed: one
# row per ≥ 2-doc folder, the mock's byte-stable text (the drill # row per existing folder (the ≥ 1-doc rule — this fixture holds
# no single-doc folders), the mock's byte-stable text (the drill
# answers quote exactly these lines). # answers quote exactly these lines).
with SessionLocal() as db: with SessionLocal() as db:
rows = db.execute( rows = db.execute(
+4 -2
View File
@@ -46,7 +46,9 @@ the same host) registered as a ``kind=local`` source (the
pattern; no git anywhere): ``syncsum/`` with TWO ≥ 2-doc folders — pattern; no git anywhere): ``syncsum/`` with TWO ≥ 2-doc folders —
``alpha/`` (2 docs) and ``bravo/`` (2 docs), no root-level files. The ``alpha/`` (2 docs) and ``bravo/`` (2 docs), no root-level files. The
generator's candidate set is therefore exactly 3: the source root generator's candidate set is therefore exactly 3: the source root
(``folder_path ""``) + the two folders (the recursive-subtree ≥ 2 rule). (``folder_path ""``) + the two folders (the recursive-subtree ≥ 1
rule — this fixture holds no single-doc folders, so the rule's
relaxation is inert for it).
Autouse cleanup (the phase-96 pattern): before each test wait for no Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on EITHER app, then truncate the shared registry + KB running sync on EITHER app, then truncate the shared registry + KB
@@ -160,7 +162,7 @@ FOLDER_B = "bravo"
FOLDERS = (FOLDER_A, FOLDER_B) FOLDERS = (FOLDER_A, FOLDER_B)
N_FILES = 4 # two folders × 2 docs N_FILES = 4 # two folders × 2 docs
N_CANDIDATES = 3 # the source root + the two folders (the ≥ 2-doc rule) N_CANDIDATES = 3 # the source root + the two folders (the ≥ 1-doc rule)
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture #: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX`` #: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
+22 -16
View File
@@ -310,9 +310,9 @@ def test_docs_tree_indexed_only_source_after_registered(admin_client, db) -> Non
assert (alpha["documents"], alpha["children"], alpha["summary"]) == (0, [], None) assert (alpha["documents"], alpha["children"], alpha["summary"]) == (0, [], None)
assert alpha["summary_pending"] is False # 0 documents — never pending assert alpha["summary_pending"] is False # 0 documents — never pending
assert midx["documents"] == 1 assert midx["documents"] == 1
assert midx["summary_pending"] is False # 1 document — below the minimum assert midx["summary_pending"] is True # 1 document — the ≥ 1 minimum, no row
assert zeta["documents"] == 1 assert zeta["documents"] == 1
assert zeta["summary_pending"] is False # 1 document — below the minimum assert zeta["summary_pending"] is True # 1 document — the ≥ 1 minimum, no row
assert [c["path"] for c in midx["children"]] == ["m1.md"] assert [c["path"] for c in midx["children"]] == ["m1.md"]
_truncate_tree_tables(db) _truncate_tree_tables(db)
@@ -321,16 +321,16 @@ def test_docs_tree_indexed_only_source_after_registered(admin_client, db) -> Non
def test_docs_tree_summary_pending_on_source_and_folder_nodes(admin_client, db) -> None: def test_docs_tree_summary_pending_on_source_and_folder_nodes(admin_client, db) -> None:
"""The endpoint returns ``summary_pending`` on SOURCE + FOLDER """The endpoint returns ``summary_pending`` on SOURCE + FOLDER
nodes (phase 98, task 03 — D3): true iff the recursive count is nodes (phase 98, task 03 — D3): true iff the recursive count is
≥ 2 AND no stored row; false WITH a stored row (any — the endpoint ≥ 1 AND no stored row — the NESTED 1-doc folder included (the ≥ 1
cannot tell AI from manual); false for a < 2-document folder rule); false WITH a stored row (any — the endpoint cannot tell AI
(never pending) — including one NESTED. File nodes carry no flag.""" from manual). File nodes carry no flag."""
_truncate_tree_tables(db) _truncate_tree_tables(db)
base = datetime.now(UTC) base = datetime.now(UTC)
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base)) db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
# k8s → 3 documents (talos + cluster + charts), NO stored row → pending # k8s → 3 documents (talos + cluster + charts), NO stored row → pending
_seed_doc(db, "Homelab", "k8s/talos.md", "Talos", 1, base) _seed_doc(db, "Homelab", "k8s/talos.md", "Talos", 1, base)
_seed_doc(db, "Homelab", "k8s/cluster.md", "Cluster", 1, base) _seed_doc(db, "Homelab", "k8s/cluster.md", "Cluster", 1, base)
# k8s/helm → 1 document — below the 2-doc minimum, never pending # k8s/helm → 1 document — a candidate at the ≥ 1 rule (no row)
_seed_doc(db, "Homelab", "k8s/helm/charts.md", "Charts", 1, base) _seed_doc(db, "Homelab", "k8s/helm/charts.md", "Charts", 1, base)
# wiki → 2 documents, WITH a stored row → not pending # wiki → 2 documents, WITH a stored row → not pending
_seed_doc(db, "Homelab", "wiki/one.md", "One", 1, base) _seed_doc(db, "Homelab", "wiki/one.md", "One", 1, base)
@@ -360,7 +360,7 @@ def test_docs_tree_summary_pending_on_source_and_folder_nodes(admin_client, db)
assert k8s["summary_pending"] is True # 3 docs, no stored row assert k8s["summary_pending"] is True # 3 docs, no stored row
helm = k8s["children"][0] helm = k8s["children"][0]
assert (helm["kind"], helm["path"]) == ("folder", "k8s/helm") assert (helm["kind"], helm["path"]) == ("folder", "k8s/helm")
assert helm["summary_pending"] is False # 1 doc — never pending assert helm["summary_pending"] is True # 1 doc — pending at the ≥ 1 rule
assert wiki["summary"] == "Wiki pages." assert wiki["summary"] == "Wiki pages."
assert wiki["summary_pending"] is False # 2 docs, but a stored row covers it assert wiki["summary_pending"] is False # 2 docs, but a stored row covers it
# File nodes carry no pending flag at all (the file table has no # File nodes carry no pending flag at all (the file table has no
@@ -397,8 +397,9 @@ def test_docs_tree_pending_set_equals_missing_folder_summaries(admin_client, db)
added_at=base + timedelta(hours=2), added_at=base + timedelta(hours=2),
) )
) )
# Alpha: a/b holds 2 docs, c holds 1 (NEVER a candidate), the root # Alpha: a/b holds 2 docs, c holds 1 (a candidate at the ≥ 1
# holds 4 — candidates (Alpha, ""), (Alpha, "a"), (Alpha, "a/b"). # rule), the root holds 4 — candidates (Alpha, ""), (Alpha, "a"),
# (Alpha, "a/b"), (Alpha, "c").
_seed_doc(db, "Alpha", "a/b/c1.md", "C1", 1, base) _seed_doc(db, "Alpha", "a/b/c1.md", "C1", 1, base)
_seed_doc(db, "Alpha", "a/b/c2.md", "C2", 1, base) _seed_doc(db, "Alpha", "a/b/c2.md", "C2", 1, base)
_seed_doc(db, "Alpha", "c/solo.md", "Solo", 1, base) _seed_doc(db, "Alpha", "c/solo.md", "Solo", 1, base)
@@ -416,6 +417,7 @@ def test_docs_tree_pending_set_equals_missing_folder_summaries(admin_client, db)
("Alpha", ""): "Alpha root.", ("Alpha", ""): "Alpha root.",
("Alpha", "a"): "Alpha a.", ("Alpha", "a"): "Alpha a.",
("Alpha", "a/b"): "Alpha a b.", ("Alpha", "a/b"): "Alpha a b.",
("Alpha", "c"): "Alpha c.",
("Beta", ""): "Beta root.", ("Beta", ""): "Beta root.",
("Beta", "x"): "Beta x.", ("Beta", "x"): "Beta x.",
}.items() }.items()
@@ -441,8 +443,10 @@ def test_docs_tree_pending_set_equals_missing_folder_summaries(admin_client, db)
# And the explicit expectation (the test is readable without the # And the explicit expectation (the test is readable without the
# helper): exactly the two deleted keys, root riding "". # helper): exactly the two deleted keys, root riding "".
assert pending == {("Alpha", "a"), ("Beta", "")} assert pending == {("Alpha", "a"), ("Beta", "")}
# Never flagged: the < 2-doc folder (Alpha/c), the 0-document # Never flagged: the 0-document registered source (Gamma) and every
# registered source (Gamma), and every node that still holds a row. # node that still holds a row — incl. the 1-doc Alpha/c, whose
# stored row covers it (the ≥ 1 rule makes it a candidate, but a
# candidate WITH a row is not missing).
assert ("Alpha", "c") not in pending assert ("Alpha", "c") not in pending
assert not any(name == "Gamma" for name, _ in pending) assert not any(name == "Gamma" for name, _ in pending)
assert ("Alpha", "a/b") not in pending assert ("Alpha", "a/b") not in pending
@@ -464,8 +468,8 @@ OLDER_STAMP = datetime(2020, 1, 1, tzinfo=UTC)
def _seed_folder_pair(db, source: str, folder: str, base: datetime) -> None: def _seed_folder_pair(db, source: str, folder: str, base: datetime) -> None:
"""Two documents under ``source/folder`` — the ≥ 2-document """Two documents under ``source/folder`` — a summarizable folder
minimum a folder must hold to be summarizable.""" (≥ 1 doc at the current minimum; two keep the pair realistic)."""
_seed_doc(db, source, f"{folder}/one.md", f"{folder} one", 1, base) _seed_doc(db, source, f"{folder}/one.md", f"{folder} one", 1, base)
_seed_doc(db, source, f"{folder}/two.md", f"{folder} two", 2, base) _seed_doc(db, source, f"{folder}/two.md", f"{folder} two", 2, base)
@@ -543,15 +547,17 @@ def test_folder_summary_create_where_no_row_exists(
admin_client: TestClient, db admin_client: TestClient, db
) -> None: ) -> None:
"""A manual description can be CREATED where no row exists — a """A manual description can be CREATED where no row exists — a
< 2-document folder the generator never wrote (or its fail-soft folder the generator has not written yet (its fail-soft miss, or
miss): insert, not update, with ``manually_edited = true``.""" a pre-sync state): insert, not update, with
``manually_edited = true``."""
_truncate_tree_tables(db) _truncate_tree_tables(db)
base = datetime.now(UTC) base = datetime.now(UTC)
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base)) db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
_seed_doc(db, "Homelab", "solo/only.md", "Only", 1, base) # 1-doc folder _seed_doc(db, "Homelab", "solo/only.md", "Only", 1, base) # 1-doc folder
db.commit() db.commit()
db.expire_all() db.expire_all()
assert _get_folder_row(db, "Homelab", "solo") is None # no AI row for < 2 docs # No AI row yet (the generator has not run in this test).
assert _get_folder_row(db, "Homelab", "solo") is None
r = admin_client.patch( r = admin_client.patch(
"/api/folders/summary", "/api/folders/summary",
+57 -46
View File
@@ -11,9 +11,9 @@ convention — the generator only flushes, the sync path commits).
Script path (``scripts.import_docs.main`` end to end, fake LLM, real Script path (``scripts.import_docs.main`` end to end, fake LLM, real
DB, explicit ``--source``): DB, explicit ``--source``):
- a KB-changing import → one row per ≥ 2-doc subtree (the source root - a KB-changing import → one row per existing subtree (the source
+ the 2-doc folder; the 1-doc folder gets none), committed in the root, the 2-doc folder a, AND the 1-doc folder b — the ≥ 1 rule),
run's transaction, the summary line ending committed in the run's transaction, the summary line ending
``folder_summaries=<generated>/<failed>/<pruned>`` (unchanged by ``folder_summaries=<generated>/<failed>/<pruned>`` (unchanged by
phase 96 — no gap-fill suffix on a full regeneration); phase 96 — no gap-fill suffix on a full regeneration);
- an unchanged re-import with a COMPLETE table → zero ``lite`` calls, - an unchanged re-import with a COMPLETE table → zero ``lite`` calls,
@@ -27,10 +27,10 @@ DB, explicit ``--source``):
failed folder summary self-heals on the next sync); failed folder summary self-heals on the next sync);
- a KB change on a second run → still a FULL regeneration (call count - a KB change on a second run → still a FULL regeneration (call count
== candidate count, every row re-stamped, no gap-fill suffix); == candidate count, every row re-stamped, no gap-fill suffix);
- a subtree dropping below 2 docs after a changed re-walk → its row - a subtree losing its LAST doc after a changed re-walk → its row
pruned; pruned;
- one folder's ``lite`` failure → its previous row kept, the other - one folder's ``lite`` failure → its previous row kept, the other
lands, exit code 0, stats ``1/1/0``; land, exit code 0, stats ``2/1/0``;
- a ``--limit`` debug run → no generation, no rows, - a ``--limit`` debug run → no generation, no rows,
``folder_summaries=skipped``; ``folder_summaries=skipped``;
- a fresh (empty) table after a ``--limit`` first walk → an unchanged - a fresh (empty) table after a ``--limit`` first walk → an unchanged
@@ -189,7 +189,8 @@ def _fresh_sync_state() -> Iterator[None]:
@pytest.fixture() @pytest.fixture()
def src(tmp_path: Path) -> Path: def src(tmp_path: Path) -> Path:
"""MyDocs: a/ (2 docs) + b/ (1 doc) → subtree counts root 3, a 2, b 1. """MyDocs: a/ (2 docs) + b/ (1 doc) → subtree counts root 3, a 2, b 1
(b/ is a candidate too — the ≥ 1 rule).
md → no document-summary chat calls, so the ``lite`` traffic is md → no document-summary chat calls, so the ``lite`` traffic is
exactly the overview + the folder summaries.""" exactly the overview + the folder summaries."""
@@ -235,9 +236,9 @@ def test_changed_import_generates_folder_rows(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str], capsys: pytest.CaptureFixture[str],
) -> None: ) -> None:
"""A KB-changing import upserts one row per ≥ 2-doc subtree in the """A KB-changing import upserts one row per existing subtree in
run's own transaction — committed and visible afterwards — with the the run's own transaction — committed and visible afterwards —
stats on the summary line (PLAN §9).""" with the stats on the summary line (PLAN §9)."""
llm = FakeEmbedder() llm = FakeEmbedder()
records: list[logging.LogRecord] = [] records: list[logging.LogRecord] = []
@@ -257,25 +258,26 @@ def test_changed_import_generates_folder_rows(
assert rc == 0 assert rc == 0
assert "added=3" in out assert "added=3" in out
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=2/0/0" "overview=updated sources_version=1 folder_summaries=3/0/0"
) )
# Two FOLDER_SUMMARY_MODE calls — the source root (3 docs) and the # Three FOLDER_SUMMARY_MODE calls — the source root (3 docs), the
# 2-doc folder a; the 1-doc folder b gets no row (no lite burn). # 2-doc folder a, and the 1-doc folder b (the ≥ 1 rule: every
# existing folder is a candidate).
calls = _folder_calls(llm) calls = _folder_calls(llm)
assert len(calls) == 2 assert len(calls) == 3
headers = [c[1]["content"].splitlines()[0] for c in calls] headers = [c[1]["content"].splitlines()[0] for c in calls]
assert headers == ["Folder: MyDocs", "Folder: MyDocs/a"] assert headers == ["Folder: MyDocs", "Folder: MyDocs/a", "Folder: MyDocs/b"]
for c in calls: for c in calls:
assert "FOLDER_SUMMARY_MODE" in c[0]["content"] assert "FOLDER_SUMMARY_MODE" in c[0]["content"]
# The rows land committed (the run's own session committed them) — # The rows land committed (the run's own session committed them) —
# one per ≥ 2-doc subtree, never empty, stamped. # one per existing subtree, never empty, stamped.
rows = _rows(db) rows = _rows(db)
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")} assert set(rows) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
assert all(rows.values()) assert all(rows.values())
assert _updated_at(db, "MyDocs", "a") is not None assert _updated_at(db, "MyDocs", "a") is not None
# The stats log line (PLAN §9 ample logging). # The stats log line (PLAN §9 ample logging).
assert any( assert any(
"folder_summaries: generated=2 failed=0 pruned=0 kept_manual=0" "folder_summaries: generated=3 failed=0 pruned=0 kept_manual=0"
in r.getMessage() in r.getMessage()
for r in records for r in records
) )
@@ -293,10 +295,10 @@ def test_unchanged_reimport_burns_zero_folder_calls(
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys) rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0 assert rc == 0
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=2/0/0" "overview=updated sources_version=1 folder_summaries=3/0/0"
) )
rows = _rows(db) rows = _rows(db)
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")} assert set(rows) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
llm2 = FakeEmbedder() llm2 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys) rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
@@ -326,7 +328,7 @@ def test_unchanged_reimport_with_gap_fills_only_the_missing_row(
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys) rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0 assert rc == 0
rows_before = _rows(db) rows_before = _rows(db)
assert set(rows_before) == {("MyDocs", ""), ("MyDocs", "a")} assert set(rows_before) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
root_stamp_before = _updated_at(db, "MyDocs", "") root_stamp_before = _updated_at(db, "MyDocs", "")
assert root_stamp_before is not None assert root_stamp_before is not None
@@ -387,13 +389,15 @@ def test_changed_reimport_is_a_full_regeneration(
assert "updated=1" in out assert "updated=1" in out
# Full-regeneration token — the stats without the gap-fill suffix. # Full-regeneration token — the stats without the gap-fill suffix.
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=2/0/0" "overview=updated sources_version=2 folder_summaries=3/0/0"
) )
# Call count == candidate count — BOTH folders, not a targeted fill. # Call count == candidate count — ALL existing folders, not a
# targeted fill.
calls = _folder_calls(llm2) calls = _folder_calls(llm2)
assert [c[1]["content"].splitlines()[0] for c in calls] == [ assert [c[1]["content"].splitlines()[0] for c in calls] == [
"Folder: MyDocs", "Folder: MyDocs",
"Folder: MyDocs/a", "Folder: MyDocs/a",
"Folder: MyDocs/b",
] ]
# All rows re-stamped (the full regeneration re-writes every # All rows re-stamped (the full regeneration re-writes every
# candidate, even the unchanging one). # candidate, even the unchanging one).
@@ -404,35 +408,37 @@ def test_changed_reimport_is_a_full_regeneration(
assert _rows(db) == rows_before # deterministic fake → same texts assert _rows(db) == rows_before # deterministic fake → same texts
def test_subtree_dropping_below_two_docs_is_pruned( def test_subtree_losing_its_last_doc_is_pruned(
db: Session, db: Session,
src: Path, src: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str], capsys: pytest.CaptureFixture[str],
) -> None: ) -> None:
"""A subtree that drops below the ≥ 2 rule on a changed re-walk """A subtree that loses its LAST doc (0 recursive docs — below the
loses its (stale) row; the remaining qualifiers regenerate.""" ≥ 1 minimum) on a changed re-walk loses its (stale) row; the
remaining qualifiers (≥ 1 doc — the 1-doc b/ included) regenerate."""
llm1 = FakeEmbedder() llm1 = FakeEmbedder()
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys) rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0 assert rc == 0
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")} assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
# a/ loses one of its two docs (below the ≥ 2 rule) while b's doc # a/ loses BOTH its docs (the folder vanishes — 0 docs) while b's
# changes → the re-walk is a KB change, so generation runs and the # doc changes → the re-walk is a KB change, so generation runs and
# stale a/ row is pruned; b (1 doc) still gets no row. # the stale a/ row is pruned; b (1 doc) keeps qualifying.
(src / "a" / "one.md").unlink()
(src / "a" / "two.md").unlink() (src / "a" / "two.md").unlink()
(src / "b" / "three.md").write_text("# B Three\nChanged content.\n", encoding="utf-8") (src / "b" / "three.md").write_text("# B Three\nChanged content.\n", encoding="utf-8")
llm2 = FakeEmbedder() llm2 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src), "--prune"], capsys) rc, out = _run_main(monkeypatch, llm2, ["--source", str(src), "--prune"], capsys)
assert rc == 0 assert rc == 0
assert "pruned=1" in out assert "pruned=2" in out # the two a/ documents left the index
assert "updated=1" in out assert "updated=1" in out
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=1/0/1" "overview=updated sources_version=2 folder_summaries=2/0/1"
) )
# Only the source root (the 2 remaining docs) still qualifies. # The source root (1 remaining doc) and b/ (1 doc) still qualify.
assert set(_rows(db)) == {("MyDocs", "")} assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "b")}
assert _updated_at(db, "MyDocs", "a") is None # the row is gone assert _updated_at(db, "MyDocs", "a") is None # the row is gone
@@ -443,8 +449,8 @@ def test_folder_lite_failure_keeps_previous_row_and_stays_green(
capsys: pytest.CaptureFixture[str], capsys: pytest.CaptureFixture[str],
) -> None: ) -> None:
"""One folder's ``lite`` failure: its previous row is kept, the """One folder's ``lite`` failure: its previous row is kept, the
other folder lands, and the run's exit code stays 0 — the stats other folders land, and the run's exit code stays 0 — the stats
carry the failure (``1/1/0``).""" carry the failure (``2/1/0``)."""
llm1 = FakeEmbedder() llm1 = FakeEmbedder()
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys) rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0 assert rc == 0
@@ -462,7 +468,7 @@ def test_folder_lite_failure_keeps_previous_row_and_stays_green(
assert rc == 0 # a failed folder must not fail the import assert rc == 0 # a failed folder must not fail the import
assert "updated=1" in out assert "updated=1" in out
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=1/1/0" "overview=updated sources_version=2 folder_summaries=2/1/0"
) )
rows_after = _rows(db) rows_after = _rows(db)
assert rows_after["MyDocs", "a"] == rows_before["MyDocs", "a"] # kept assert rows_after["MyDocs", "a"] == rows_before["MyDocs", "a"] # kept
@@ -484,14 +490,15 @@ def test_changed_import_never_overwrites_a_manual_row(
— ``kept_manual`` is a stat, not a token), the manual row's text, — ``kept_manual`` is a stat, not a token), the manual row's text,
stamp, and flag are untouched, and the other folders regenerate stamp, and flag are untouched, and the other folders regenerate
(the ``kept_manual`` stat lands on the generator's log line).""" (the ``kept_manual`` stat lands on the generator's log line)."""
# First sync: full generation — root + a/ (b/ holds 1 doc: none). # First sync: full generation — root + a/ + b/ (the ≥ 1 rule: b/
# holds 1 doc — still a candidate).
llm1 = FakeEmbedder() llm1 = FakeEmbedder()
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys) rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
assert rc == 0 assert rc == 0
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=1 folder_summaries=2/0/0" "overview=updated sources_version=1 folder_summaries=3/0/0"
) )
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")} assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
# The owner edits the source-root description (task 03's PATCH is # The owner edits the source-root description (task 03's PATCH is
# the writer; task 01 pins the generator's behavior, so the row is # the writer; task 01 pins the generator's behavior, so the row is
@@ -533,11 +540,15 @@ def test_changed_import_never_overwrites_a_manual_row(
assert "added=1" in out assert "added=1" in out
# The token STAYS 3 fields — kept_manual is a stat, not a token. # The token STAYS 3 fields — kept_manual is a stat, not a token.
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=2 folder_summaries=1/0/0" "overview=updated sources_version=2 folder_summaries=2/0/0"
) )
# Zero folder calls for the owner's folder — only a/ (now 3 docs). # Zero folder calls for the owner's folder — a/ (now 3 docs) and
# b/ regenerate.
calls = _folder_calls(llm2) calls = _folder_calls(llm2)
assert [c[1]["content"].splitlines()[0] for c in calls] == ["Folder: MyDocs/a"] assert [c[1]["content"].splitlines()[0] for c in calls] == [
"Folder: MyDocs/a",
"Folder: MyDocs/b",
]
# The owner's text, stamp, and flag are untouched ... # The owner's text, stamp, and flag are untouched ...
rows_after = _rows(db) rows_after = _rows(db)
assert rows_after[("MyDocs", "")] == manual_text assert rows_after[("MyDocs", "")] == manual_text
@@ -555,7 +566,7 @@ def test_changed_import_never_overwrites_a_manual_row(
assert rows_after[("MyDocs", "a")] is not None assert rows_after[("MyDocs", "a")] is not None
# The 4-field stats line carries the skip (PLAN §9 ample logging). # The 4-field stats line carries the skip (PLAN §9 ample logging).
assert any( assert any(
"folder_summaries: generated=1 failed=0 pruned=0 kept_manual=1" "folder_summaries: generated=2 failed=0 pruned=0 kept_manual=1"
in r.getMessage() in r.getMessage()
for r in records for r in records
) )
@@ -610,10 +621,10 @@ def test_empty_table_generates_on_unchanged_walk(
# `` (gap-fill)`` (the generated set is the full candidate set). # `` (gap-fill)`` (the generated set is the full candidate set).
assert out.rstrip().endswith( assert out.rstrip().endswith(
"overview=updated sources_version=skipped " "overview=updated sources_version=skipped "
"folder_summaries=2/0/0 (gap-fill)" "folder_summaries=3/0/0 (gap-fill)"
) )
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")} assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a"), ("MyDocs", "b")}
assert len(_folder_calls(llm2)) == 2 assert len(_folder_calls(llm2)) == 3
assert current_sources_version(db) == 0 # the unchanged walk never bumps assert current_sources_version(db) == 0 # the unchanged walk never bumps
+112 -79
View File
@@ -161,9 +161,11 @@ def test_group_by_folder_nested_multi_source_recursive_subtree() -> None:
def test_group_by_folder_single_doc_folder_is_a_group_too() -> None: def test_group_by_folder_single_doc_folder_is_a_group_too() -> None:
"""Grouping is pure subtree membership (≥ 1 docs): the ≥ 2 rule is """Grouping is pure subtree membership: at the
the GENERATOR's (the recursive count below the minimum yields no :data:`MIN_DOCS_PER_FOLDER` = 1 rule every group (≥ 1 docs) is
row — pinned by the generator tests, not the grouping).""" already a generator candidate, so grouping and candidacy agree
(the generator's filter is an inert safety net — pinned by the
generator tests, not the grouping)."""
rows = [("S", "a/only.md", "O", None)] rows = [("S", "a/only.md", "O", None)]
groups = group_by_folder(rows) groups = group_by_folder(rows)
assert len(groups[("S", "a")]) == 1 # present, but below the minimum assert len(groups[("S", "a")]) == 1 # present, but below the minimum
@@ -417,8 +419,9 @@ def _rows(db: Session) -> dict[tuple[str, str], str]:
def _seed_catalogue(db: Session) -> None: def _seed_catalogue(db: Session) -> None:
"""The shared catalogue: FSU has four docs in three candidate """The shared catalogue: FSU has four docs in three candidate
folders (root 4, a 3, a/b 2 — all ≥ the minimum); FSU-solo has one folders (root 4, a 3, a/b 2); FSU-solo has one doc (its root
doc (its root folder is below the minimum — no row, no call).""" folder is a candidate too — the ≥ 1 rule: a single-file source
root gets a row)."""
_add_doc(db, "FSU", "a/b/one.md", "One", "One lead.\nSource: FSU/a/b/one.md") _add_doc(db, "FSU", "a/b/one.md", "One", "One lead.\nSource: FSU/a/b/one.md")
_add_doc(db, "FSU", "a/b/two.md", "Two") _add_doc(db, "FSU", "a/b/two.md", "Two")
_add_doc(db, "FSU", "a/three.md", "Three") _add_doc(db, "FSU", "a/three.md", "Three")
@@ -436,22 +439,23 @@ def clean_tables(db: Session):
def test_generate_happy_path_upserts_every_candidate_folder( def test_generate_happy_path_upserts_every_candidate_folder(
db: Session, clean_tables, caplog: pytest.LogCaptureFixture db: Session, clean_tables, caplog: pytest.LogCaptureFixture
) -> None: ) -> None:
"""Every folder with ≥ 2 recursive docs gets a row (the source root """Every folder with ≥ 1 recursive doc gets a row (the source root
row included — ``folder_path = ''``); single-doc folders get none; row included — ``folder_path = ''`` — and the single-doc FSU-solo
rows are stamped fresh; the stats dict and the log line are right; root: the ≥ 1 rule); rows are stamped fresh; the stats dict and
folders are processed in deterministic (source, folder_path) order.""" the log line are right; folders are processed in deterministic
(source, folder_path) order."""
_seed_catalogue(db) _seed_catalogue(db)
llm = _FakeLLM() llm = _FakeLLM()
with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"): with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm)) stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0} assert stats == {"generated": 4, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3, "one lite call per candidate folder (the solo folder: none)" assert llm.calls == 4, "one lite call per candidate folder (the solo root: too)"
stored = _rows(db) stored = _rows(db)
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")} assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b"), ("FSU-solo", "")}
assert all(summary == REPLY for summary in stored.values()) assert all(summary == REPLY for summary in stored.values())
assert ("FSU-solo", "") not in stored, ( assert stored[("FSU-solo", "")] == REPLY, (
"a single-doc folder is fully described by its one file line — no row" "the single-doc source root is a candidate at the ≥ 1 rule"
) )
row = db.get(FolderSummary, ("FSU", "a/b")) row = db.get(FolderSummary, ("FSU", "a/b"))
@@ -467,6 +471,7 @@ def test_generate_happy_path_upserts_every_candidate_folder(
"Folder: FSU", "Folder: FSU",
"Folder: FSU/a", "Folder: FSU/a",
"Folder: FSU/a/b", "Folder: FSU/a/b",
"Folder: FSU-solo",
] ]
# The recursive-subtree input: the a/ prompt carries a/b's docs too. # The recursive-subtree input: the a/ prompt carries a/b's docs too.
a_prompt = llm.requests[1][1] a_prompt = llm.requests[1][1]
@@ -475,7 +480,7 @@ def test_generate_happy_path_upserts_every_candidate_folder(
assert "root.md — Root" not in a_prompt assert "root.md — Root" not in a_prompt
assert ( assert (
"folder_summaries: generated=3 failed=0 pruned=0 kept_manual=0" "folder_summaries: generated=4 failed=0 pruned=0 kept_manual=0"
in caplog.text in caplog.text
), "the stats line must be greppable (PLAN §9 ample logging)" ), "the stats line must be greppable (PLAN §9 ample logging)"
@@ -492,16 +497,18 @@ def test_generate_per_folder_fail_soft_keeps_previous_and_lands_others(
llm = _FakeLLM(fail_folders=("FSU/a/b",)) llm = _FakeLLM(fail_folders=("FSU/a/b",))
with caplog.at_level(logging.ERROR, logger="app.rag.folder_summaries"): with caplog.at_level(logging.ERROR, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm)) stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0} assert stats == {"generated": 3, "failed": 1, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3 # the failing folder was attempted too assert llm.calls == 4 # the failing folder was attempted too
stored = _rows(db) stored = _rows(db)
assert stored[("FSU", "a/b")] == "old summary", ( assert stored[("FSU", "a/b")] == "old summary", (
"the previous row survives the per-folder failure" "the previous row survives the per-folder failure"
) )
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a")] == REPLY, ( assert (
"the other folders still land" stored[("FSU", "")] == REPLY
) and stored[("FSU", "a")] == REPLY
and stored[("FSU-solo", "")] == REPLY
), "the other folders still land"
assert "folder summary failed for FSU/a/b" in caplog.text assert "folder summary failed for FSU/a/b" in caplog.text
assert "simulated lite-model failure for FSU/a/b" in caplog.text assert "simulated lite-model failure for FSU/a/b" in caplog.text
@@ -519,24 +526,25 @@ def test_generate_per_folder_fail_soft_without_previous_row_creates_nothing(
def test_generate_prunes_stale_rows_and_keeps_live_ones(db: Session, clean_tables) -> None: def test_generate_prunes_stale_rows_and_keeps_live_ones(db: Session, clean_tables) -> None:
"""Rows for folders that dropped below 2 recursive docs are deleted """Rows for folders that VANISHED (0 recursive docs) are deleted
(pruned/renamed — the summary would go stale); rows for folders (pruned/renamed — the summary would go stale); rows for folders
that still qualify persist (an unchanged folder's summary is still that still hold ≥ 1 doc persist (an unchanged folder's summary is
true — regenerated in place).""" still true — regenerated in place), the 1-doc FSU-solo root
included (it is a candidate at the ≥ 1 rule)."""
_seed_catalogue(db) _seed_catalogue(db)
# A stale row for a folder no longer in the catalogue (3→1 docs / # A stale row for a folder no longer in the catalogue (renamed
# renamed away) + a live row with old content. # away — 0 docs) + live rows with old content.
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale")) db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
db.add(FolderSummary(source="FSU", folder_path="a", summary="old a summary")) db.add(FolderSummary(source="FSU", folder_path="a", summary="old a summary"))
db.add(FolderSummary(source="FSU-solo", folder_path="", summary="solo stale")) db.add(FolderSummary(source="FSU-solo", folder_path="", summary="solo stale"))
db.commit() db.commit()
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM())) stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
assert stats["pruned"] == 2 # gone/old + the FSU-solo root (1 doc) assert stats["pruned"] == 1 # gone/old only (its folder vanished)
stored = _rows(db) stored = _rows(db)
assert ("FSU", "gone/old") not in stored, "the stale folder row must be pruned" assert ("FSU", "gone/old") not in stored, "the stale folder row must be pruned"
assert ("FSU-solo", "") not in stored, ( assert stored[("FSU-solo", "")] == REPLY, (
"a folder that dropped below 2 docs loses its row" "the 1-doc root qualifies — regenerated, not pruned"
) )
assert ("FSU", "a") in stored, "the still-qualifying folder keeps its row" assert ("FSU", "a") in stored, "the still-qualifying folder keeps its row"
assert stored[("FSU", "a")] == REPLY # regenerated, not stale assert stored[("FSU", "a")] == REPLY # regenerated, not stale
@@ -614,7 +622,7 @@ def test_generate_only_flushes_caller_commits(db: Session, clean_tables) -> None
) )
assert n == 3, "the caller's commit makes the flushed rows durable" assert n == 3, "the caller's commit makes the flushed rows durable"
assert MIN_DOCS_PER_FOLDER == 2 # the ≥ 2 scope rule, pinned by name assert MIN_DOCS_PER_FOLDER == 1 # the ≥ 1 scope rule, pinned by name
# ---------- manually_edited (phase 97, task 01) ---------- # ---------- manually_edited (phase 97, task 01) ----------
@@ -644,17 +652,22 @@ def test_manual_row_survives_regeneration(
with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"): with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm)) stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 2, "failed": 0, "pruned": 0, "kept_manual": 1} assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 1}
assert llm.calls == 2, "the manual folder burns zero lite calls" assert llm.calls == 3, "the manual folder burns zero lite calls"
assert [user.splitlines()[0] for _s, user in llm.requests] == [ assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: FSU", "Folder: FSU",
"Folder: FSU/a/b", "Folder: FSU/a/b",
"Folder: FSU-solo",
], "no prompt is ever built for the owner's folder" ], "no prompt is ever built for the owner's folder"
stored = _rows(db) stored = _rows(db)
assert stored[("FSU", "a")] == manual_text, "the owner's text survives" assert stored[("FSU", "a")] == manual_text, "the owner's text survives"
assert _updated_at(db, "FSU", "a") == stamp_before, ("never re-stamped") assert _updated_at(db, "FSU", "a") == stamp_before, ("never re-stamped")
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY, ( assert (
stored[("FSU", "")] == REPLY
and stored[("FSU", "a/b")] == REPLY
and stored[("FSU-solo", "")] == REPLY
), (
"the non-manual candidates still regenerate (the flag is the difference)" "the non-manual candidates still regenerate (the flag is the difference)"
) )
row = db.get(FolderSummary, ("FSU", "a")) row = db.get(FolderSummary, ("FSU", "a"))
@@ -662,19 +675,21 @@ def test_manual_row_survives_regeneration(
"the generator never clears the flag" "the generator never clears the flag"
) )
assert ( assert (
"folder_summaries: generated=2 failed=0 pruned=0 kept_manual=1" "folder_summaries: generated=3 failed=0 pruned=0 kept_manual=1"
in caplog.text in caplog.text
), "the 4-field stats line must be greppable (PLAN §9 ample logging)" ), "the 4-field stats line must be greppable (PLAN §9 ample logging)"
def test_manual_row_survives_the_prune(db: Session, clean_tables) -> None: def test_manual_row_survives_the_prune(db: Session, clean_tables) -> None:
"""A manual row is NEVER pruned (phase 97, task 01): two folders """A manual row is NEVER pruned and NEVER overwritten (phase 97,
drop below 2 documents — the MANUAL one keeps its row (owner task 01): a/ and b/ each drop to 1 recursive doc — still
content persists until cleared — the clear deletes it, so the next candidates at the ≥ 1 rule — so the MANUAL a/ row is SKIPPED
KB-changing sync regenerates an AI description) while the (owner text kept, no re-stamp, no lite burn) while the NON-manual
NON-manual twin loses its now-stale row; the flag is the only b/ twin regenerates in place; the vanished folders' rows show the
difference. A vanished folder's manual row is kept too, and its prune rule — the manual gone/manual row persists (owner content
non-manual twin is pruned.""" until cleared — the clear deletes it, so the next KB-changing sync
regenerates an AI description) and the non-manual gone/ai twin is
pruned; the flag is the only difference in each pair."""
_add_doc(db, "FSU", "a/one.md", "One") _add_doc(db, "FSU", "a/one.md", "One")
_add_doc(db, "FSU", "a/two.md", "Two") _add_doc(db, "FSU", "a/two.md", "Two")
_add_doc(db, "FSU", "b/one.md", "B One") _add_doc(db, "FSU", "b/one.md", "B One")
@@ -696,7 +711,8 @@ def test_manual_row_survives_the_prune(db: Session, clean_tables) -> None:
db.add(FolderSummary(source="FSU", folder_path="gone/ai", summary="stale ai")) db.add(FolderSummary(source="FSU", folder_path="gone/ai", summary="stale ai"))
db.commit() db.commit()
# a/ and b/ each drop below the minimum (2 -> 1 recursive doc). # a/ and b/ each drop to 1 recursive doc — still candidates at
# the ≥ 1 rule.
db.execute( db.execute(
text( text(
"DELETE FROM documents WHERE source = 'FSU'" "DELETE FROM documents WHERE source = 'FSU'"
@@ -708,14 +724,14 @@ def test_manual_row_survives_the_prune(db: Session, clean_tables) -> None:
llm = _FakeLLM() llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm)) stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 1, "failed": 0, "pruned": 2, "kept_manual": 0} assert stats == {"generated": 2, "failed": 0, "pruned": 1, "kept_manual": 1}
assert llm.calls == 1, "only the surviving candidate (the root) regenerates" assert llm.calls == 2, "root + the non-manual b/ regenerate; manual a/ burns nothing"
stored = _rows(db) stored = _rows(db)
assert stored[("FSU", "a")] == manual_text, ( assert stored[("FSU", "a")] == manual_text, (
"the manual row survives its folder dropping below the minimum" "the manual row is skipped — never overwritten, even at 1 doc"
) )
assert ("FSU", "b") not in stored, ( assert stored[("FSU", "b")] == REPLY, (
"the non-manual twin loses its stale row (the flag is the difference)" "the non-manual twin regenerates — 1 doc still qualifies (the ≥ 1 rule)"
) )
assert stored[("FSU", "gone/manual")] == "owner kept", ( assert stored[("FSU", "gone/manual")] == "owner kept", (
"a vanished folder's manual row is kept — owner content until cleared" "a vanished folder's manual row is kept — owner content until cleared"
@@ -731,15 +747,15 @@ def test_missing_fresh_table_is_exactly_the_candidate_set(
db: Session, clean_tables db: Session, clean_tables
) -> None: ) -> None:
"""No stored rows → every candidate folder is a gap, sorted by """No stored rows → every candidate folder is a gap, sorted by
``(source, folder_path)``; the single-doc FSU-solo root is not a ``(source, folder_path)`` — the single-doc FSU-solo root
candidate and can never be a gap.""" included (a candidate at the ≥ 1 rule)."""
_seed_catalogue(db) _seed_catalogue(db)
assert missing_folder_summaries(db) == [ assert missing_folder_summaries(db) == [
("FSU", ""), ("FSU", ""),
("FSU", "a"), ("FSU", "a"),
("FSU", "a/b"), ("FSU", "a/b"),
("FSU-solo", ""),
] ]
assert ("FSU-solo", "") not in missing_folder_summaries(db)
def test_missing_fully_populated_table_is_empty(db: Session, clean_tables) -> None: def test_missing_fully_populated_table_is_empty(db: Session, clean_tables) -> None:
@@ -770,16 +786,18 @@ def test_missing_empty_kb_empty_table_is_no_gap(db: Session, clean_tables) -> No
assert missing_folder_summaries(db) == [] assert missing_folder_summaries(db) == []
def test_missing_single_doc_folder_is_never_listed(db: Session, clean_tables) -> None: def test_missing_single_doc_folder_is_listed(db: Session, clean_tables) -> None:
"""A below-minimum folder without a row is NOT a gap — it is not a """The ≥ 1 rule: a single-doc folder WITHOUT a row IS a gap (it is
candidate (its one file line IS its summary).""" a candidate — the one file line no longer exempts it), alongside
its source root (1 doc)."""
_add_doc(db, "FSU", "solo/one.md", "One") _add_doc(db, "FSU", "solo/one.md", "One")
assert missing_folder_summaries(db) == [] assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "solo")]
def test_missing_stale_row_is_not_a_gap(db: Session, clean_tables) -> None: def test_missing_stale_row_is_not_a_gap(db: Session, clean_tables) -> None:
"""A stored row for a folder that dropped below 2 docs is stale, """A stored row for a folder that dropped below the minimum
not missing — the prune pass owns it, the gap detector ignores it.""" (vanished — 0 docs) is stale, not missing — the prune pass owns
it, the gap detector ignores it."""
_seed_catalogue(db) _seed_catalogue(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM())) asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit() db.commit()
@@ -865,12 +883,16 @@ def test_only_missing_still_prunes_stale_rows(db: Session, clean_tables) -> None
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale")) db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
db.commit() db.commit()
a_stamp = _updated_at(db, "FSU", "a") a_stamp = _updated_at(db, "FSU", "a")
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a/b")] assert missing_folder_summaries(db) == [
("FSU", ""),
("FSU", "a/b"),
("FSU-solo", ""),
]
llm = _FakeLLM() llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True)) stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
assert stats == {"generated": 2, "failed": 0, "pruned": 1, "kept_manual": 0} assert stats == {"generated": 3, "failed": 0, "pruned": 1, "kept_manual": 0}
assert llm.calls == 2 assert llm.calls == 3
stored = _rows(db) stored = _rows(db)
assert ("FSU", "gone/old") not in stored, ( assert ("FSU", "gone/old") not in stored, (
@@ -878,7 +900,11 @@ def test_only_missing_still_prunes_stale_rows(db: Session, clean_tables) -> None
) )
assert stored[("FSU", "a")] == "keep me" assert stored[("FSU", "a")] == "keep me"
assert _updated_at(db, "FSU", "a") == a_stamp assert _updated_at(db, "FSU", "a") == a_stamp
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY assert (
stored[("FSU", "")] == REPLY
and stored[("FSU", "a/b")] == REPLY
and stored[("FSU-solo", "")] == REPLY
)
def test_only_missing_fail_soft_keeps_prior_and_lands_others( def test_only_missing_fail_soft_keeps_prior_and_lands_others(
@@ -892,11 +918,12 @@ def test_only_missing_fail_soft_keeps_prior_and_lands_others(
db.commit() db.commit()
llm = _FakeLLM(fail_folders=("FSU/a/b",)) llm = _FakeLLM(fail_folders=("FSU/a/b",))
stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True)) stats = asyncio.run(generate_folder_summaries(db, llm, only_missing=True))
assert stats == {"generated": 1, "failed": 1, "pruned": 0, "kept_manual": 0} assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0}
assert llm.calls == 2 # both missing folders were attempted assert llm.calls == 3 # all three missing folders were attempted
stored = _rows(db) stored = _rows(db)
assert stored[("FSU", "")] == REPLY, "the other missing folder still lands" assert stored[("FSU", "")] == REPLY, "the other missing folders still land"
assert stored[("FSU-solo", "")] == REPLY, "the single-doc root still lands"
assert ("FSU", "a/b") not in stored, "the failed folder stays absent" assert ("FSU", "a/b") not in stored, "the failed folder stays absent"
assert stored[("FSU", "a")] == "keep me", "the present row is untouched" assert stored[("FSU", "a")] == "keep me", "the present row is untouched"
@@ -941,18 +968,19 @@ def test_on_progress_fires_once_per_candidate_in_sorted_key_order(
"""Phase 98 (task 01): the hook fires once per candidate, in the """Phase 98 (task 01): the hook fires once per candidate, in the
same sorted ``(source, folder_path)`` order the folders are same sorted ``(source, folder_path)`` order the folders are
attempted — done climbs 1..total, total = the candidate count attempted — done climbs 1..total, total = the candidate count
(the single-doc FSU-solo root is not a candidate — no event).""" (the single-doc FSU-solo root IS a candidate at the ≥ 1 rule)."""
_seed_catalogue(db) _seed_catalogue(db)
llm = _FakeLLM() llm = _FakeLLM()
events, record = _record_progress() events, record = _record_progress()
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record)) stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
assert events == [ assert events == [
(1, 3, "FSU", ""), (1, 4, "FSU", ""),
(2, 3, "FSU", "a"), (2, 4, "FSU", "a"),
(3, 3, "FSU", "a/b"), (3, 4, "FSU", "a/b"),
(4, 4, "FSU-solo", ""),
] ]
assert llm.calls == 3 # one event per attempt, in the same order assert llm.calls == 4 # one event per attempt, in the same order
assert stats["generated"] == 3 assert stats["generated"] == 4
def test_on_progress_manual_skip_still_advances(db: Session, clean_tables) -> None: def test_on_progress_manual_skip_still_advances(db: Session, clean_tables) -> None:
@@ -971,12 +999,13 @@ def test_on_progress_manual_skip_still_advances(db: Session, clean_tables) -> No
events, record = _record_progress() events, record = _record_progress()
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record)) stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
assert events == [ assert events == [
(1, 3, "FSU", ""), (1, 4, "FSU", ""),
(2, 3, "FSU", "a"), # the instant manual skip still advances (2, 4, "FSU", "a"), # the instant manual skip still advances
(3, 3, "FSU", "a/b"), (3, 4, "FSU", "a/b"),
(4, 4, "FSU-solo", ""),
] ]
assert llm.calls == 2, "the skip itself burns no call" assert llm.calls == 3, "the skip itself burns no call"
assert stats["kept_manual"] == 1 and stats["generated"] == 2 assert stats["kept_manual"] == 1 and stats["generated"] == 3
def test_on_progress_failed_key_still_advances(db: Session, clean_tables) -> None: def test_on_progress_failed_key_still_advances(db: Session, clean_tables) -> None:
@@ -989,11 +1018,12 @@ def test_on_progress_failed_key_still_advances(db: Session, clean_tables) -> Non
events, record = _record_progress() events, record = _record_progress()
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record)) stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
assert events == [ assert events == [
(1, 3, "FSU", ""), (1, 4, "FSU", ""),
(2, 3, "FSU", "a"), # the failed attempt still advances (2, 4, "FSU", "a"), # the failed attempt still advances
(3, 3, "FSU", "a/b"), (3, 4, "FSU", "a/b"),
(4, 4, "FSU-solo", ""),
] ]
assert stats["failed"] == 1 and stats["generated"] == 2 assert stats["failed"] == 1 and stats["generated"] == 3
def test_on_progress_only_missing_reports_the_missing_count( def test_on_progress_only_missing_reports_the_missing_count(
@@ -1049,7 +1079,7 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
generate_folder_summaries(db, hooked, on_progress=record) generate_folder_summaries(db, hooked, on_progress=record)
) )
requests_hooked = hooked.requests.copy() requests_hooked = hooked.requests.copy()
assert len(events) == 3, "the hooked run fired (the contrast is real)" assert len(events) == 4, "the hooked run fired (the contrast is real)"
db.rollback() # the generator only flushes — drop the uncommitted rows db.rollback() # the generator only flushes — drop the uncommitted rows
plain = _FakeLLM() plain = _FakeLLM()
@@ -1058,5 +1088,8 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
assert plain.requests == requests_hooked, "the fake LLM's call log is unchanged" assert plain.requests == requests_hooked, "the fake LLM's call log is unchanged"
assert stats_plain == stats_hooked assert stats_plain == stats_hooked
assert _rows(db) == { assert _rows(db) == {
("FSU", ""): REPLY, ("FSU", "a"): REPLY, ("FSU", "a/b"): REPLY ("FSU", ""): REPLY,
("FSU", "a"): REPLY,
("FSU", "a/b"): REPLY,
("FSU-solo", ""): REPLY,
} }
+8 -7
View File
@@ -314,7 +314,7 @@ def test_cross_check_nested_level_matches_group_folder_listing() -> None:
# -------------------------------------------------------------------- # --------------------------------------------------------------------
# summary_pending (phase 98, task 03) — the D3 rule (ONE concept): # summary_pending (phase 98, task 03) — the D3 rule (ONE concept):
# a SOURCE or FOLDER node is pending iff its recursive document count # a SOURCE or FOLDER node is pending iff its recursive document count
# ≥ MIN_DOCS_PER_FOLDER (2) AND it has NO stored folder_summaries row # ≥ MIN_DOCS_PER_FOLDER (1) AND it has NO stored folder_summaries row
# (AI or manual — any row). That is exactly # (AI or manual — any row). That is exactly
# ``app.rag.folder_summaries.missing_folder_summaries``'s candidate # ``app.rag.folder_summaries.missing_folder_summaries``'s candidate
# set — the marker never drifts from the gap-fill (the integration # set — the marker never drifts from the gap-fill (the integration
@@ -359,10 +359,11 @@ def test_folder_with_stored_row_is_not_pending() -> None:
assert source.summary_pending is True assert source.summary_pending is True
def test_single_document_folder_never_pending() -> None: def test_single_document_folder_is_pending() -> None:
"""A < 2-document folder is NEVER pending (it never gets a summary """The ≥ 1 rule: a 1-document folder with NO stored row IS pending
— its one file line IS its description), even with no stored row (its one file line no longer exempts it — the next sync
— while its ≥ 2-doc source root (no root row) still is.""" summarizes it), as is its source root (2 docs, no root row).
A registered 0-document source is the only never-pending case."""
rows = [ rows = [
("S", "solo/only.md", "Only", 1, T0, C0), # 1-doc folder ("S", "solo/only.md", "Only", 1, T0, C0), # 1-doc folder
("S", "top.md", "Top", 1, T0, C1), # source total = 2 ("S", "top.md", "Top", 1, T0, C1), # source total = 2
@@ -370,7 +371,7 @@ def test_single_document_folder_never_pending() -> None:
(source,) = build_kb_tree(["S"], rows, {}) (source,) = build_kb_tree(["S"], rows, {})
(solo,) = _folder_nodes(source) (solo,) = _folder_nodes(source)
assert solo.documents == 1 assert solo.documents == 1
assert solo.summary_pending is False assert solo.summary_pending is True
assert source.summary_pending is True assert source.summary_pending is True
@@ -407,7 +408,7 @@ def test_name_collision_pending_follows_recursive_count() -> None:
def test_source_root_pending_and_zero_document_source_never() -> None: def test_source_root_pending_and_zero_document_source_never() -> None:
"""The source root: a source with ≥ 2 docs and NO ``(source, "")`` """The source root: a source with ≥ 1 docs and NO ``(source, "")``
row → the SOURCE node is pending; the stored root row clears it. row → the SOURCE node is pending; the stored root row clears it.
A registered 0-document source is NEVER pending (0 < the minimum — A registered 0-document source is NEVER pending (0 < the minimum —
there is nothing to summarize), with or without a manual row.""" there is nothing to summarize), with or without a manual row."""