feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Build and Push Containers / build-and-push-app (push) Successful in 2m10s
Build and Push Containers / build-and-push-db (push) Successful in 13s

Relax the phase-94 folder-summary scope rule from ≥ 2 documents to
≥ 1: a folder (or source root) is a candidate while ANY document
lives under it, so single-file folders and single-file source roots
get their own lite-written description. A row is now pruned only
when its folder loses its last document (vanishes from the
catalogue).

The constant is the single source of truth, so the flip propagates
to the generator's candidate set, the prune pass, the
missing_folder_summaries gap probe (the next sync self-heals the
new gaps), and the KB-tree summary_pending markers (1-doc folders /
sources now read "Summary pending" until their row lands).

Docstrings/comments across app/, scripts/import_docs.py, and the
E2E fixtures updated to the ≥ 1 wording. Unit + integration tests
updated to the new semantics (the pruned-below-minimum scenario is
now a folder losing its LAST doc; single-doc folders are pinned as
candidates/pending). Full suite: 2314 passed, app coverage 99%;
ruff + pyright clean; folder-summary E2E stories pass in isolation
(ls_tree_drilldown, sync_summary_visibility, kb_tree,
kb_tree_nav, document_dates, oneshot_llm_retry).
This commit is contained in:
2026-09-14 08:57:57 -04:00
parent 3a81793565
commit 35d65d2f25
13 changed files with 297 additions and 212 deletions
+35 -22
View File
@@ -20,13 +20,16 @@ summary describes.
Storage: ``folder_summaries`` (migration 0017) — PK
``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT
(the top-level source summary). Rows exist only for folders with
≥ 2 documents (the :data:`MIN_DOCS_PER_FOLDER` rule — a
single-document folder is fully described by its one file line, so no
``lite`` burn); after a changed sync, rows whose folder dropped below
2 documents are pruned (a pruned/renamed folder's summary would
otherwise go stale), while rows for folders that still have
≥ 2 documents persist (an unchanged folder's summary is still true).
(the top-level source summary). Rows exist for every folder whose
recursive subtree holds ≥ 1 document (the
:data:`MIN_DOCS_PER_FOLDER` rule — a folder is a catalogue prefix
only while some document lives under it, so EVERY existing folder is
a candidate, single-file folders and single-file source roots
included); after a changed sync, rows whose folder dropped below 1
document (vanished from the catalogue) are pruned (a pruned/renamed
folder's summary would otherwise go stale), while rows for folders
that still hold at least one document persist (an unchanged folder's
summary is still true).
Manual rows (phase 97, task 01): ``manually_edited`` (migration 0018)
marks the descriptions the OWNER edited — ``PATCH /api/folders/
@@ -104,10 +107,13 @@ SYSTEM_PROMPT = f"{FOLDER_SUMMARY_MODE}: {FOLDER_SUMMARY_INSTRUCTION}"
FOLDER_HEADER_PREFIX = "Folder: "
#: A folder is summarized only while its recursive subtree holds at
#: least this many documents — a single-document folder is fully
#: described by its one file line, so no ``lite`` burn (phase 94
#: ``00_phase.md`` scope rule; the prune rule applies the same count).
MIN_DOCS_PER_FOLDER = 2
#: least this many documents — 1: EVERY folder the catalogue knows
#: (a folder exists only while some document lives under it, so the
#: filter admits every group :func:`group_by_folder` returns,
#: single-file folders included) gets a row (the phase-94 ≥ 2 scope
#: rule relaxed by owner decision; the prune rule applies the same
#: count — a row is pruned when its folder loses its last document).
MIN_DOCS_PER_FOLDER = 1
#: One document row for the grouping/prompting:
#: ``(source, path, title, summary)`` — the ``app.rag.overview``
@@ -164,8 +170,10 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
catalogue query). Returns ``{(source, folder_path): [rows]}`` —
group lists keep the input (catalogue) order, so downstream prompt
building is deterministic. Groups of ANY size (≥ 1) are returned;
the ≥ 2 :data:`MIN_DOCS_PER_FOLDER` rule is applied by
:func:`generate_folder_summaries`, not here.
at the :data:`MIN_DOCS_PER_FOLDER` = 1 rule every group is
already a candidate, so :func:`generate_folder_summaries`'s
filter is an inert safety net (it would bite only if the minimum
were ever raised above 1), not a behavioral gate.
"""
# Pass 1: the distinct TRUE folder prefixes of the catalogue — a
@@ -336,7 +344,9 @@ detector and the fill can never disagree about what the catalogue
def _candidates(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
"""The generator's candidate map (recursive subtree ≥ 2 docs).
"""The generator's candidate map (recursive subtree ≥
:data:`MIN_DOCS_PER_FOLDER` docs — at the current minimum of 1,
every :func:`group_by_folder` group).
The :func:`group_by_folder` groups filtered by
:data:`MIN_DOCS_PER_FOLDER` — the EXACT set a full regeneration
@@ -379,10 +389,11 @@ gap persisted until the next KB change. This function names the gap:
Returns the missing keys sorted by ``(source, folder_path)``;
``[]`` when there is no gap — including an empty catalogue over an
empty table (no candidates, no gap). A single-document folder is
never listed (it is not a candidate), and a stored row for a folder
that dropped below the minimum is NOT missing (it is stale — the
prune pass owns it).
empty table (no candidates, no gap). A stored row for a folder
that dropped below the minimum (vanished — 0 documents) is NOT
missing (it is stale — the prune pass owns it); a single-document
folder without a row IS listed (it is a candidate at the ≥ 1
rule).
"""
return sorted(
key for key in _candidates(_catalog_rows(db)) if key not in _stored_keys(db)
@@ -408,8 +419,9 @@ async def generate_folder_summaries(
2. Group the document catalogue by :func:`group_by_folder` (the
recursive-subtree concept) and keep the candidate folders —
the ones whose recursive subtree holds
:data:`MIN_DOCS_PER_FOLDER` (≥ 2) documents. Single-document
folders get no row (their one file line IS their summary).
:data:`MIN_DOCS_PER_FOLDER` (1) documents: at the current
minimum every existing folder (single-file ones included, and
the single-file source root) is a candidate.
3. For each candidate (deterministic ``(source, folder_path)``
order) call :func:`summarize_folder` and UPSERT — fail-soft PER
FOLDER: one folder's :class:`LLMError` is logged and counted,
@@ -430,8 +442,9 @@ async def generate_folder_summaries(
staleness is the changed-KB regeneration's job) and no ``lite``
call is burned for a folder that already has a summary — the
unchanged-sync self-heal fill (phase 96, task 02).
4. DELETE rows whose folder no longer has ≥ 2 documents — a
pruned/renamed folder's summary goes stale and is dropped —
4. DELETE rows whose folder no longer has ≥ 1 document (vanished
from the catalogue) — a pruned/renamed folder's summary goes
stale and is dropped —
EXCEPT a manual row: owner content persists until the owner
clears it, even for a folder that dropped below the minimum
(phase 97, task 01; the clear deletes the row, so the next