feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Relax the phase-94 folder-summary scope rule from ≥ 2 documents to ≥ 1: a folder (or source root) is a candidate while ANY document lives under it, so single-file folders and single-file source roots get their own lite-written description. A row is now pruned only when its folder loses its last document (vanishes from the catalogue). The constant is the single source of truth, so the flip propagates to the generator's candidate set, the prune pass, the missing_folder_summaries gap probe (the next sync self-heals the new gaps), and the KB-tree summary_pending markers (1-doc folders / sources now read "Summary pending" until their row lands). Docstrings/comments across app/, scripts/import_docs.py, and the E2E fixtures updated to the ≥ 1 wording. Unit + integration tests updated to the new semantics (the pruned-below-minimum scenario is now a folder losing its LAST doc; single-doc folders are pinned as candidates/pending). Full suite: 2314 passed, app coverage 99%; ruff + pyright clean; folder-summary E2E stories pass in isolation (ls_tree_drilldown, sync_summary_visibility, kb_tree, kb_tree_nav, document_dates, oneshot_llm_retry).
This commit is contained in:
+10
-8
@@ -576,7 +576,7 @@ def build_kb_tree(
|
||||
* **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
|
||||
node (phase 98, decision D3 — ONE concept): true iff the node's
|
||||
recursive ``documents`` count ≥
|
||||
:data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (2) AND it
|
||||
:data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (1) AND it
|
||||
has NO stored ``folder_summaries`` row (AI or manual — any row;
|
||||
the builder sees stored rows only). That is EXACTLY
|
||||
:func:`app.rag.folder_summaries.missing_folder_summaries`'s
|
||||
@@ -584,9 +584,10 @@ def build_kb_tree(
|
||||
keys on the next sync — the marker is honest: "waiting to
|
||||
generate", and the integration cross-check pins the tree's
|
||||
pending set to that function so the marker can never drift from
|
||||
the gap-fill). A < 2-document folder is NEVER pending (it never
|
||||
gets a summary — its one file line IS its description), and a
|
||||
registered 0-document source never is.
|
||||
the gap-fill). A 0-document folder cannot exist (a folder is a
|
||||
catalogue prefix only) — every existing folder with no row is
|
||||
pending, single-file folders included — and a registered
|
||||
0-document source never is.
|
||||
|
||||
ONE concept end to end: the builder reuses
|
||||
:func:`app.rag.folder_summaries.folder_of` and the phase-94
|
||||
@@ -674,10 +675,11 @@ def list_kb_tree(
|
||||
exactly as ``app/api/chat.py`` does) + the SAME outerjoin/grouped
|
||||
catalogue query ``GET /api/docs`` runs (the document ``id``
|
||||
excluded — the tree has no document ids) + ALL stored
|
||||
``folder_summaries`` rows (a bounded select — one row per ≥ 2-doc
|
||||
folder; rows for sources the tree does not list are never
|
||||
referenced by the builder) — through the pure
|
||||
:func:`build_kb_tree`. ``GET /api/docs`` itself is untouched.
|
||||
``folder_summaries`` rows (a bounded select — one row per
|
||||
existing folder at the ≥ 1-doc rule; rows for sources the tree
|
||||
does not list are never referenced by the builder) — through the
|
||||
pure :func:`build_kb_tree`. ``GET /api/docs`` itself is
|
||||
untouched.
|
||||
"""
|
||||
names = list_source_names(db)
|
||||
rows = db.execute(
|
||||
|
||||
+2
-1
@@ -353,7 +353,8 @@ async def _run_sync() -> None:
|
||||
# changed KB (added + updated > 0) is a full regeneration
|
||||
# (today's behavior, byte-identical); an unchanged re-sync
|
||||
# takes the GAP probe instead of the old table-empty one — a
|
||||
# candidate folder (at least 2 docs) with no stored row: a gap
|
||||
# candidate folder (at least 1 doc — every existing folder)
|
||||
# with no stored row: a gap
|
||||
# fills ONLY the missing rows (only_missing=True — the
|
||||
# subsumed table-empty first-sync trigger included, where
|
||||
# every candidate is missing), a complete table burns zero
|
||||
|
||||
+13
-11
@@ -65,8 +65,9 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
|
||||
(``id = 1``); every column NULL = "use the
|
||||
default" (env value for the strings, the built-in
|
||||
palette for the colors — task 01).
|
||||
* ``folder_summaries`` — one row per folder with ≥ 2 documents:
|
||||
the sync-time ``lite`` summary the drill-down
|
||||
* ``folder_summaries`` — one row per folder (recursive count
|
||||
≥ 1 — every existing folder): the sync-time
|
||||
``lite`` summary the drill-down
|
||||
``ls`` shows next to each folder (phase 94;
|
||||
``folder_path = ""`` = the source root).
|
||||
"""
|
||||
@@ -501,14 +502,15 @@ class FolderSummary(Base):
|
||||
``documents.path`` (String(1000)) and is the source-relative
|
||||
folder prefix. ``folder_path = ""`` is the SOURCE ROOT — the
|
||||
top-level source summary (the whole source's recursive subtree).
|
||||
* Rows exist only for folders with ≥ 2 documents (recursive count
|
||||
— the same set the ``ls`` count rule counts): a
|
||||
single-document folder is fully described by its one file line,
|
||||
so no ``lite`` burn. After a changed sync, rows whose folder
|
||||
dropped below 2 documents are pruned (a pruned/renamed folder's
|
||||
summary would otherwise go stale); rows for folders that still
|
||||
have ≥ 2 documents persist (an unchanged folder's summary is
|
||||
still true). Both rules are generator policy (``app.rag.
|
||||
* Rows exist for folders with ≥ 1 document (recursive count —
|
||||
the same set the ``ls`` count rule counts): a folder is a
|
||||
candidate while any document lives under it, single-file folders
|
||||
and single-file source roots included. After a changed sync,
|
||||
rows whose folder dropped below 1 document (vanished from the
|
||||
catalogue) are pruned (a pruned/renamed folder's summary would
|
||||
otherwise go stale); rows for folders that still hold at least
|
||||
one document persist (an unchanged folder's summary is still
|
||||
true). Both rules are generator policy (``app.rag.
|
||||
folder_summaries``), not schema constraints.
|
||||
* ``summary`` — the 1–3 sentence plain-text description the aipi
|
||||
``lite`` model wrote at sync time (``FOLDER_SUMMARY_MODE``,
|
||||
@@ -521,7 +523,7 @@ class FolderSummary(Base):
|
||||
folder_summaries``) SKIPS a manual row on regeneration (no
|
||||
``lite`` burn on owner text — counted ``kept_manual``) and never
|
||||
prunes it (owner content persists until cleared — even for a
|
||||
folder below the 2-document minimum); an owner correction is
|
||||
folder below the 1-document minimum); an owner correction is
|
||||
never silently rewritten (the phase-97 ``00_phase.md`` decision).
|
||||
Clearing the description deletes the row — the next KB-changing
|
||||
sync regenerates an AI description (the reset path).
|
||||
|
||||
+35
-22
@@ -20,13 +20,16 @@ summary describes.
|
||||
|
||||
Storage: ``folder_summaries`` (migration 0017) — PK
|
||||
``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT
|
||||
(the top-level source summary). Rows exist only for folders with
|
||||
≥ 2 documents (the :data:`MIN_DOCS_PER_FOLDER` rule — a
|
||||
single-document folder is fully described by its one file line, so no
|
||||
``lite`` burn); after a changed sync, rows whose folder dropped below
|
||||
2 documents are pruned (a pruned/renamed folder's summary would
|
||||
otherwise go stale), while rows for folders that still have
|
||||
≥ 2 documents persist (an unchanged folder's summary is still true).
|
||||
(the top-level source summary). Rows exist for every folder whose
|
||||
recursive subtree holds ≥ 1 document (the
|
||||
:data:`MIN_DOCS_PER_FOLDER` rule — a folder is a catalogue prefix
|
||||
only while some document lives under it, so EVERY existing folder is
|
||||
a candidate, single-file folders and single-file source roots
|
||||
included); after a changed sync, rows whose folder dropped below 1
|
||||
document (vanished from the catalogue) are pruned (a pruned/renamed
|
||||
folder's summary would otherwise go stale), while rows for folders
|
||||
that still hold at least one document persist (an unchanged folder's
|
||||
summary is still true).
|
||||
|
||||
Manual rows (phase 97, task 01): ``manually_edited`` (migration 0018)
|
||||
marks the descriptions the OWNER edited — ``PATCH /api/folders/
|
||||
@@ -104,10 +107,13 @@ SYSTEM_PROMPT = f"{FOLDER_SUMMARY_MODE}: {FOLDER_SUMMARY_INSTRUCTION}"
|
||||
FOLDER_HEADER_PREFIX = "Folder: "
|
||||
|
||||
#: A folder is summarized only while its recursive subtree holds at
|
||||
#: least this many documents — a single-document folder is fully
|
||||
#: described by its one file line, so no ``lite`` burn (phase 94
|
||||
#: ``00_phase.md`` scope rule; the prune rule applies the same count).
|
||||
MIN_DOCS_PER_FOLDER = 2
|
||||
#: least this many documents — 1: EVERY folder the catalogue knows
|
||||
#: (a folder exists only while some document lives under it, so the
|
||||
#: filter admits every group :func:`group_by_folder` returns,
|
||||
#: single-file folders included) gets a row (the phase-94 ≥ 2 scope
|
||||
#: rule relaxed by owner decision; the prune rule applies the same
|
||||
#: count — a row is pruned when its folder loses its last document).
|
||||
MIN_DOCS_PER_FOLDER = 1
|
||||
|
||||
#: One document row for the grouping/prompting:
|
||||
#: ``(source, path, title, summary)`` — the ``app.rag.overview``
|
||||
@@ -164,8 +170,10 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
|
||||
catalogue query). Returns ``{(source, folder_path): [rows]}`` —
|
||||
group lists keep the input (catalogue) order, so downstream prompt
|
||||
building is deterministic. Groups of ANY size (≥ 1) are returned;
|
||||
the ≥ 2 :data:`MIN_DOCS_PER_FOLDER` rule is applied by
|
||||
:func:`generate_folder_summaries`, not here.
|
||||
at the :data:`MIN_DOCS_PER_FOLDER` = 1 rule every group is
|
||||
already a candidate, so :func:`generate_folder_summaries`'s
|
||||
filter is an inert safety net (it would bite only if the minimum
|
||||
were ever raised above 1), not a behavioral gate.
|
||||
|
||||
"""
|
||||
# Pass 1: the distinct TRUE folder prefixes of the catalogue — a
|
||||
@@ -336,7 +344,9 @@ detector and the fill can never disagree about what the catalogue
|
||||
|
||||
|
||||
def _candidates(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
|
||||
"""The generator's candidate map (recursive subtree ≥ 2 docs).
|
||||
"""The generator's candidate map (recursive subtree ≥
|
||||
:data:`MIN_DOCS_PER_FOLDER` docs — at the current minimum of 1,
|
||||
every :func:`group_by_folder` group).
|
||||
|
||||
The :func:`group_by_folder` groups filtered by
|
||||
:data:`MIN_DOCS_PER_FOLDER` — the EXACT set a full regeneration
|
||||
@@ -379,10 +389,11 @@ gap persisted until the next KB change. This function names the gap:
|
||||
|
||||
Returns the missing keys sorted by ``(source, folder_path)``;
|
||||
``[]`` when there is no gap — including an empty catalogue over an
|
||||
empty table (no candidates, no gap). A single-document folder is
|
||||
never listed (it is not a candidate), and a stored row for a folder
|
||||
that dropped below the minimum is NOT missing (it is stale — the
|
||||
prune pass owns it).
|
||||
empty table (no candidates, no gap). A stored row for a folder
|
||||
that dropped below the minimum (vanished — 0 documents) is NOT
|
||||
missing (it is stale — the prune pass owns it); a single-document
|
||||
folder without a row IS listed (it is a candidate at the ≥ 1
|
||||
rule).
|
||||
"""
|
||||
return sorted(
|
||||
key for key in _candidates(_catalog_rows(db)) if key not in _stored_keys(db)
|
||||
@@ -408,8 +419,9 @@ async def generate_folder_summaries(
|
||||
2. Group the document catalogue by :func:`group_by_folder` (the
|
||||
recursive-subtree concept) and keep the candidate folders —
|
||||
the ones whose recursive subtree holds
|
||||
:data:`MIN_DOCS_PER_FOLDER` (≥ 2) documents. Single-document
|
||||
folders get no row (their one file line IS their summary).
|
||||
:data:`MIN_DOCS_PER_FOLDER` (1) documents: at the current
|
||||
minimum every existing folder (single-file ones included, and
|
||||
the single-file source root) is a candidate.
|
||||
3. For each candidate (deterministic ``(source, folder_path)``
|
||||
order) call :func:`summarize_folder` and UPSERT — fail-soft PER
|
||||
FOLDER: one folder's :class:`LLMError` is logged and counted,
|
||||
@@ -430,8 +442,9 @@ async def generate_folder_summaries(
|
||||
staleness is the changed-KB regeneration's job) and no ``lite``
|
||||
call is burned for a folder that already has a summary — the
|
||||
unchanged-sync self-heal fill (phase 96, task 02).
|
||||
4. DELETE rows whose folder no longer has ≥ 2 documents — a
|
||||
pruned/renamed folder's summary goes stale and is dropped —
|
||||
4. DELETE rows whose folder no longer has ≥ 1 document (vanished
|
||||
from the catalogue) — a pruned/renamed folder's summary goes
|
||||
stale and is dropped —
|
||||
EXCEPT a manual row: owner content persists until the owner
|
||||
clears it, even for a folder that dropped below the minimum
|
||||
(phase 97, task 01; the clear deletes the row, so the next
|
||||
|
||||
+8
-6
@@ -288,12 +288,13 @@ class KbTreeFolder(BaseModel):
|
||||
resolves it with ``from __future__ import annotations``).
|
||||
|
||||
``summary_pending`` (phase 98, D3 — ONE concept): true iff this
|
||||
folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (2) AND it has
|
||||
folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (1) AND it has
|
||||
NO stored ``folder_summaries`` row (AI or manual — any row) —
|
||||
exactly the candidate the sync-time gap-fill regenerates (the
|
||||
:func:`app.rag.folder_summaries.missing_folder_summaries` set).
|
||||
A < 2-document folder is never pending (it never gets a summary —
|
||||
its one file line IS its description).
|
||||
A 0-document folder cannot exist (a folder is a catalogue prefix
|
||||
only), so every existing folder with no stored row is pending —
|
||||
single-file folders included.
|
||||
"""
|
||||
|
||||
kind: Literal["folder"] = "folder"
|
||||
@@ -325,12 +326,13 @@ class KbTreeSource(BaseModel):
|
||||
|
||||
``summary_pending`` (phase 98, D3 — ONE concept): true iff the
|
||||
source's recursive ``documents`` count ≥ ``MIN_DOCS_PER_FOLDER``
|
||||
(2) AND no stored ``(source, "")`` row (AI or manual — any row) —
|
||||
(1) AND no stored ``(source, "")`` row (AI or manual — any row) —
|
||||
exactly the source-root candidate the sync-time gap-fill
|
||||
regenerates (the
|
||||
:func:`app.rag.folder_summaries.missing_folder_summaries` set).
|
||||
A registered 0-document source is never pending (there is nothing
|
||||
to summarize).
|
||||
to summarize); a single-document source IS pending while its root
|
||||
row is absent.
|
||||
"""
|
||||
|
||||
name: str
|
||||
@@ -353,7 +355,7 @@ class KbTree(BaseModel):
|
||||
|
||||
Every SOURCE and FOLDER node carries ``summary_pending`` (phase
|
||||
98, D3): true iff its recursive document count ≥
|
||||
``MIN_DOCS_PER_FOLDER`` (2) AND it has no stored ``folder_summaries``
|
||||
``MIN_DOCS_PER_FOLDER`` (1) AND it has no stored ``folder_summaries``
|
||||
row — exactly ``missing_folder_summaries``'s candidate set (the
|
||||
marker never drifts from the gap-fill); FILE nodes carry no flag
|
||||
(the file table has no description column).
|
||||
|
||||
Reference in New Issue
Block a user