phase: 97_kb_tree_catalog
All completion criteria verified — everything is green, no defects found. Final report: ## Phase 97 final verification pass — ALL GREEN **Verified (no code changes needed):** - `GET /api/docs/tree` (admin), `build_kb_tree` pure builder, `PATCH /api/folders/summary`, migration 0018 (`manually_edited`, head confirmed), generator skip/keep + `kept_manual` stat, RAG tree UI + edit affordance in `sources.js`/`index.html`/`styles.css` - `tests/e2e/test_kb_tree.py`: 8 passed — top level, drill source/folder, edit round-trip, clear, manual-desc-survives-sync, reload fallback, anonymous gate - Integration: tree shape/order/403/empty/indexed-only + PATCH update/create/root/clear/404/403/no-LLM + stat-walk equivalence (in `test_docs_api.py`); 3-field `folder_summaries=` import token preserved **Gates (exact commands):** - `uv run pytest --cov=app --cov-report=term-missing` → **2053 passed**, TOTAL coverage **99%** (>90% ✓) - `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors** - `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` → **8 passed** in isolation - 30 story/RAG-view E2E suites run **one per process**: all passed, incl. `test_ls_tree_drilldown` (agent `ls` byte-identical ✓), `test_import_documents`, `test_edit_summaries`, `test_admin_auth`, `test_kb_overview` **Completion criteria:** tree view ✓ · edit round-trip + clear ✓ · manual persists/clear resets ✓ · `ls` unchanged ✓ · pytest/coverage/lint ✓ · E2E isolation ✓ · commit — left to harness per protocol (working tree untouched, `git add/commit` not run) **Deviations:** none. **Next pending phase:** none — `todo/` contains only 97 (96 already committed).
This commit is contained in:
+56
-26
@@ -28,6 +28,16 @@ single-document folder is fully described by its one file line, so no
|
||||
otherwise go stale), while rows for folders that still have
|
||||
≥ 2 documents persist (an unchanged folder's summary is still true).
|
||||
|
||||
Manual rows (phase 97, task 01): ``manually_edited`` (migration 0018)
|
||||
marks the descriptions the OWNER edited — ``PATCH /api/folders/
|
||||
summary`` (phase 97, task 03) is the ONLY writer. The generator's two
|
||||
rules for a manual row: it is SKIPPED on regeneration (no ``lite``
|
||||
burn on owner text — counted ``kept_manual`` in the stats) and it is
|
||||
NEVER pruned (owner content persists until the owner clears it — even
|
||||
for a folder that dropped below the minimum). Clearing the
|
||||
description DELETES the row, so the next KB-changing sync regenerates
|
||||
an AI description for that folder (the reset path).
|
||||
|
||||
Chat turns never generate folder summaries — the agent's ``ls`` output
|
||||
(phase 94, task 03) only reads the stored rows. Generation is the
|
||||
caller's job at sync time (phase 94, task 02), and :func:`generate_
|
||||
@@ -399,20 +409,29 @@ async def generate_folder_summaries(
|
||||
FOLDER: one folder's :class:`LLMError` is logged and counted,
|
||||
its previous row (if any) is kept, and the remaining folders
|
||||
still land (a ``lite`` outage must never fail the sync — the KB
|
||||
is the product, the summaries are auxiliary). With
|
||||
is the product, the summaries are auxiliary). An EXISTING row
|
||||
with ``manually_edited`` is SKIPPED instead of regenerated —
|
||||
no ``lite`` call for owner text (no burn on the owner's words),
|
||||
the row stays byte-identical (text AND ``updated_at``), and the
|
||||
skip counts ``kept_manual`` (phase 97, task 01: an owner
|
||||
correction is never silently rewritten). With
|
||||
``only_missing=True`` the iteration is restricted to the
|
||||
candidates that have NO stored row (the same gap
|
||||
:func:`missing_folder_summaries` reports, derived from the SAME
|
||||
catalogue pass — one bounded stored-key select, no second
|
||||
catalogue query): existing rows stay byte-identical (summary
|
||||
text AND ``updated_at`` — never re-stamped, even a stale-looking
|
||||
one; staleness is the changed-KB regeneration's job) and no
|
||||
``lite`` call is burned for a folder that already has a summary
|
||||
— the unchanged-sync self-heal fill (phase 96, task 02).
|
||||
stored-rows pass — one select, no second catalogue query):
|
||||
existing rows stay byte-identical (summary text AND
|
||||
``updated_at`` — never re-stamped, even a stale-looking one;
|
||||
staleness is the changed-KB regeneration's job) and no ``lite``
|
||||
call is burned for a folder that already has a summary — the
|
||||
unchanged-sync self-heal fill (phase 96, task 02).
|
||||
4. DELETE rows whose folder no longer has ≥ 2 documents — a
|
||||
pruned/renamed folder's summary goes stale and is dropped.
|
||||
Rows for folders that still qualify persist (regenerated in
|
||||
step 3 — an unchanged folder's summary is still true). The
|
||||
pruned/renamed folder's summary goes stale and is dropped —
|
||||
EXCEPT a manual row: owner content persists until the owner
|
||||
clears it, even for a folder that dropped below the minimum
|
||||
(phase 97, task 01; the clear deletes the row, so the next
|
||||
KB-changing sync regenerates an AI description — the reset
|
||||
path). Rows for folders that still qualify persist (regenerated
|
||||
in step 3 — an unchanged folder's summary is still true). The
|
||||
prune pass runs in BOTH modes: under ``only_missing`` on an
|
||||
unchanged catalogue it is a no-op (the invariant kept), and it
|
||||
still drops rows whose folder fell below the minimum.
|
||||
@@ -421,22 +440,37 @@ async def generate_folder_summaries(
|
||||
``bump_sources_version`` convention: the sync path owns the
|
||||
transaction, so a failed sync rolls the summaries back with it).
|
||||
|
||||
Returns the small stats dict ``{"generated", "failed", "pruned"}``
|
||||
for the caller's summary-line logging (PLAN §9 ample logging) —
|
||||
the caller logs the mode, not the generator.
|
||||
Returns the small stats dict ``{"generated", "failed", "pruned",
|
||||
"kept_manual"}`` for the caller's summary-line logging (PLAN §9
|
||||
ample logging) — the caller logs the mode, not the generator. The
|
||||
``import_docs`` summary-line token stays its 3 fields
|
||||
(``<generated>/<failed>/<pruned>`` — ``kept_manual`` is a stat, not
|
||||
a token; phase 97, task 01).
|
||||
"""
|
||||
stats = {"generated": 0, "failed": 0, "pruned": 0}
|
||||
stats = {"generated": 0, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
if skip:
|
||||
return stats
|
||||
|
||||
candidates = _candidates(_catalog_rows(db))
|
||||
# The existing rows, fetched ONCE (phase 97, task 01): both the
|
||||
# ``only_missing`` filter and the prune pass key off this single
|
||||
# {(source, folder_path): row} dict — one fetch, one concept.
|
||||
existing = {
|
||||
(row.source, row.folder_path): row
|
||||
for row in db.execute(select(FolderSummary)).scalars()
|
||||
}
|
||||
keys = sorted(candidates)
|
||||
if only_missing:
|
||||
stored = _stored_keys(db)
|
||||
keys = [key for key in keys if key not in stored]
|
||||
keys = [key for key in keys if key not in existing]
|
||||
|
||||
for key in keys:
|
||||
source, folder_path = key
|
||||
stored = existing.get(key)
|
||||
if stored is not None and stored.manually_edited:
|
||||
# Owner-edited description (phase 97, task 01): NEVER
|
||||
# overwrite it — no lite burn on owner text.
|
||||
stats["kept_manual"] += 1
|
||||
continue
|
||||
docs = candidates[key]
|
||||
try:
|
||||
summary = await summarize_folder(source, folder_path, docs, llm)
|
||||
@@ -449,21 +483,17 @@ async def generate_folder_summaries(
|
||||
_upsert(db, source, folder_path, summary)
|
||||
stats["generated"] += 1
|
||||
|
||||
existing = db.execute(
|
||||
select(FolderSummary.source, FolderSummary.folder_path)
|
||||
).all()
|
||||
for source, folder_path in existing:
|
||||
if (source, folder_path) not in candidates:
|
||||
row = db.get(FolderSummary, (source, folder_path))
|
||||
if row is not None:
|
||||
db.delete(row)
|
||||
stats["pruned"] += 1
|
||||
for key, row in existing.items():
|
||||
if key not in candidates and not row.manually_edited:
|
||||
db.delete(row)
|
||||
stats["pruned"] += 1
|
||||
|
||||
db.flush()
|
||||
logger.info(
|
||||
"folder_summaries: generated=%d failed=%d pruned=%d",
|
||||
"folder_summaries: generated=%d failed=%d pruned=%d kept_manual=%d",
|
||||
stats["generated"],
|
||||
stats["failed"],
|
||||
stats["pruned"],
|
||||
stats["kept_manual"],
|
||||
)
|
||||
return stats
|
||||
|
||||
Reference in New Issue
Block a user