phase: 127_sync_incremental_folder_summaries
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Build and Push Containers / build-and-push-db (push) Successful in 15s

All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
2026-09-27 21:12:25 -04:00
parent f50d4c8bc3
commit 6e654df387
38 changed files with 2741 additions and 46 deletions
+54 -12
View File
@@ -65,6 +65,7 @@ client-side router. Direct URLs deep-link to the matching view.
| **Tuning** | `/tuning.html` | Admin only |
| **Saved chats** | `/history.html` | Admin only |
| **Access tokens** | `/tokens.html` | Admin only |
| **Theme** | `/theme.html` | Admin only |
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
| **Login** | `/login.html` | Everyone |
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
@@ -207,6 +208,10 @@ reflect the new files.
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
bmp`) are indexed as documents too — see
[Image support](#image-support).
- A run that changed the KB also regenerates the **KB overview** and the
**folder summaries** (the drill-down catalog's stored per-folder
descriptions) with the `lite` model — best-effort: a summary failure
never fails the import.
### Git-based sources
@@ -262,6 +267,12 @@ whole git-source refresh in one click:
2. **re-import with prune** — files deleted upstream leave the index.
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
chat turn injects) — only when the import changed the knowledge base.
4. **refresh the folder summaries** (the drill-down catalog's stored
per-folder descriptions) — a changed knowledge base rewrites ONLY the
folders the changed files touch (each file's folder chain up to its
source root — phase 127: one changed file re-stamps a handful of
rows, not the whole table); an unchanged re-sync instead gap-fills
any missing rows (`only_missing`).
- States: the button shows **Syncing…** with a spinner while polling
`GET /api/sync/status` every 2 s. No client-side timeout — a clone +
@@ -288,13 +299,16 @@ To hide it, set `BOR_STREAM_THINKING=0`.
Retrieval only puts the top documents in context. When an answer depends on
a file a note *references* ("the exact JSON shape is in
`example-record-file.json`"), the model can extend its own context with
three server-side tools — on **grounded** (high-relevance) turns only:
server-side tools — on **grounded** (high-relevance) turns only:
| Tool | Description |
|------|-------------|
| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source |
| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string |
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
| **`read(path)`** | Appends one indexed document's text to context by its combined `source/path`; very large documents are truncated at `BOR_READ_MAX_CHARS` with a notice that points at `grep` to find the rest |
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
Each call is executed against Postgres only (no extra LLM round trip) and
streamed as an SSE `tool` frame. In the chat, each call shows a transient
@@ -361,12 +375,14 @@ docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
---
## Document summaries (non-markdown)
## Document summaries
Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
retrieval can miss exactly the documents that are all configuration. At
import time, every **non-markdown** A9 document is summarized by the aipi
`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`):
import time, **every** new or changed document gets a `lite`-model summary —
**markdown included**: the summary is a natural-language retrieval target
and the seed text for the onboarding suggestion blocks (phase 118 retired
the old "markdown is already natural language" exclusion):
* The summary is stored on `documents.summary` **and** indexed as one extra
embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
@@ -452,6 +468,30 @@ image:
A question image is **never indexed as a document** — it is turn-local
storage, not a source.
### Answer images (phase 124)
A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
on, the grounded chat model may show a standalone image document as part
of its answer:
- **The model decides.** The agent is offered a fourth tool,
`show_image(path)`, and calls it only when the image itself contains
the answer (a diagram, screenshot, or photo of the thing asked about) —
never for mere topical relevance. At most **one** image per answer
(enforced server-side); the text answer is always given in full — the
image supplements, never replaces.
- **Rendered as an answer figure.** A shown image appears as a
first-class figure below the answer text and above the source chips —
on the live turn, after a page reload, and on the shared chat page; a
failed load collapses the figure (never a broken-image icon).
- **Image chips on zero-read turns.** A retrieved image document in the
suggested tier is auto-cited: its source chip (with figure) appears
even when the model never `read`s it — the model's seed block already
carries the image's full description, so a real model has no reason to
read it (this applies with `BOR_IMAGES=1`, any `BOR_ANSWER_IMAGES`).
- **Off = byte-identical.** No fourth tool offered, no figure, no
`answer_image` key in any SSE frame or saved record.
---
## Caching & deploys
@@ -515,7 +555,7 @@ to `.env` and adjust.
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
@@ -578,7 +618,7 @@ to `.env` and adjust.
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
### Image support (phases 122/123)
### Image support (phases 122/123/124)
| Env | Default | Meaning |
|-----|---------|---------|
@@ -587,6 +627,7 @@ to `.env` and adjust.
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |
### Docs push
@@ -615,10 +656,11 @@ Every identity string is an env var: `BOR_APP_NAME` (display name),
applied by `assets/brand.js` at boot.
The colors are set from the admin **Theme tab** (`/theme.html`,
admin-only — phase 91): the 8 identity colors plus the three strings
above are edited with text fields and color pickers, persisted in the
`ui_settings` table, and injected into every served page as an inline
`<style>` BEFORE first paint (no red flash, no pop-in). The old
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
the 8 semantic state colors — success/error/accent families) and the
three strings above are edited with text fields and color pickers,
persisted in the `ui_settings` table, and injected into every served page
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The old
CSS-file theming is retired — a leftover `BOR_THEME` line in a
deployment's `.env` is simply ignored. Leave everything unset and the
app renders the built-in dark-tech palette byte-identically.