phase: 127_sync_incremental_folder_summaries
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact. **Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass) - **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors** - **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me) - **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed - **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit) - **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
@@ -65,6 +65,7 @@ client-side router. Direct URLs deep-link to the matching view.
|
||||
| **Tuning** | `/tuning.html` | Admin only |
|
||||
| **Saved chats** | `/history.html` | Admin only |
|
||||
| **Access tokens** | `/tokens.html` | Admin only |
|
||||
| **Theme** | `/theme.html` | Admin only |
|
||||
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
|
||||
| **Login** | `/login.html` | Everyone |
|
||||
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
|
||||
@@ -207,6 +208,10 @@ reflect the new files.
|
||||
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
|
||||
bmp`) are indexed as documents too — see
|
||||
[Image support](#image-support).
|
||||
- A run that changed the KB also regenerates the **KB overview** and the
|
||||
**folder summaries** (the drill-down catalog's stored per-folder
|
||||
descriptions) with the `lite` model — best-effort: a summary failure
|
||||
never fails the import.
|
||||
|
||||
### Git-based sources
|
||||
|
||||
@@ -262,6 +267,12 @@ whole git-source refresh in one click:
|
||||
2. **re-import with prune** — files deleted upstream leave the index.
|
||||
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
|
||||
chat turn injects) — only when the import changed the knowledge base.
|
||||
4. **refresh the folder summaries** (the drill-down catalog's stored
|
||||
per-folder descriptions) — a changed knowledge base rewrites ONLY the
|
||||
folders the changed files touch (each file's folder chain up to its
|
||||
source root — phase 127: one changed file re-stamps a handful of
|
||||
rows, not the whole table); an unchanged re-sync instead gap-fills
|
||||
any missing rows (`only_missing`).
|
||||
|
||||
- States: the button shows **Syncing…** with a spinner while polling
|
||||
`GET /api/sync/status` every 2 s. No client-side timeout — a clone +
|
||||
@@ -288,13 +299,16 @@ To hide it, set `BOR_STREAM_THINKING=0`.
|
||||
Retrieval only puts the top documents in context. When an answer depends on
|
||||
a file a note *references* ("the exact JSON shape is in
|
||||
`example-record-file.json`"), the model can extend its own context with
|
||||
three server-side tools — on **grounded** (high-relevance) turns only:
|
||||
server-side tools — on **grounded** (high-relevance) turns only:
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source |
|
||||
| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string |
|
||||
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
|
||||
| **`read(path)`** | Appends one indexed document's text to context by its combined `source/path`; very large documents are truncated at `BOR_READ_MAX_CHARS` with a notice that points at `grep` to find the rest |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
|
||||
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
|
||||
|
||||
Each call is executed against Postgres only (no extra LLM round trip) and
|
||||
streamed as an SSE `tool` frame. In the chat, each call shows a transient
|
||||
@@ -361,12 +375,14 @@ docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.
|
||||
|
||||
---
|
||||
|
||||
## Document summaries (non-markdown)
|
||||
## Document summaries
|
||||
|
||||
Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
|
||||
retrieval can miss exactly the documents that are all configuration. At
|
||||
import time, every **non-markdown** A9 document is summarized by the aipi
|
||||
`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`):
|
||||
import time, **every** new or changed document gets a `lite`-model summary —
|
||||
**markdown included**: the summary is a natural-language retrieval target
|
||||
and the seed text for the onboarding suggestion blocks (phase 118 retired
|
||||
the old "markdown is already natural language" exclusion):
|
||||
|
||||
* The summary is stored on `documents.summary` **and** indexed as one extra
|
||||
embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
|
||||
@@ -452,6 +468,30 @@ image:
|
||||
A question image is **never indexed as a document** — it is turn-local
|
||||
storage, not a source.
|
||||
|
||||
### Answer images (phase 124)
|
||||
|
||||
A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
|
||||
on, the grounded chat model may show a standalone image document as part
|
||||
of its answer:
|
||||
|
||||
- **The model decides.** The agent is offered a fourth tool,
|
||||
`show_image(path)`, and calls it only when the image itself contains
|
||||
the answer (a diagram, screenshot, or photo of the thing asked about) —
|
||||
never for mere topical relevance. At most **one** image per answer
|
||||
(enforced server-side); the text answer is always given in full — the
|
||||
image supplements, never replaces.
|
||||
- **Rendered as an answer figure.** A shown image appears as a
|
||||
first-class figure below the answer text and above the source chips —
|
||||
on the live turn, after a page reload, and on the shared chat page; a
|
||||
failed load collapses the figure (never a broken-image icon).
|
||||
- **Image chips on zero-read turns.** A retrieved image document in the
|
||||
suggested tier is auto-cited: its source chip (with figure) appears
|
||||
even when the model never `read`s it — the model's seed block already
|
||||
carries the image's full description, so a real model has no reason to
|
||||
read it (this applies with `BOR_IMAGES=1`, any `BOR_ANSWER_IMAGES`).
|
||||
- **Off = byte-identical.** No fourth tool offered, no figure, no
|
||||
`answer_image` key in any SSE frame or saved record.
|
||||
|
||||
---
|
||||
|
||||
## Caching & deploys
|
||||
@@ -515,7 +555,7 @@ to `.env` and adjust.
|
||||
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
|
||||
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
|
||||
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
|
||||
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview |
|
||||
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
|
||||
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
|
||||
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
|
||||
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
|
||||
@@ -578,7 +618,7 @@ to `.env` and adjust.
|
||||
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
||||
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
||||
|
||||
### Image support (phases 122/123)
|
||||
### Image support (phases 122/123/124)
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|-----|---------|---------|
|
||||
@@ -587,6 +627,7 @@ to `.env` and adjust.
|
||||
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
|
||||
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
|
||||
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
|
||||
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |
|
||||
|
||||
### Docs push
|
||||
|
||||
@@ -615,10 +656,11 @@ Every identity string is an env var: `BOR_APP_NAME` (display name),
|
||||
applied by `assets/brand.js` at boot.
|
||||
|
||||
The colors are set from the admin **Theme tab** (`/theme.html`,
|
||||
admin-only — phase 91): the 8 identity colors plus the three strings
|
||||
above are edited with text fields and color pickers, persisted in the
|
||||
`ui_settings` table, and injected into every served page as an inline
|
||||
`<style>` BEFORE first paint (no red flash, no pop-in). The old
|
||||
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
|
||||
the 8 semantic state colors — success/error/accent families) and the
|
||||
three strings above are edited with text fields and color pickers,
|
||||
persisted in the `ui_settings` table, and injected into every served page
|
||||
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The old
|
||||
CSS-file theming is retired — a leftover `BOR_THEME` line in a
|
||||
deployment's `.env` is simply ignored. Leave everything unset and the
|
||||
app renders the built-in dark-tech palette byte-identically.
|
||||
|
||||
Reference in New Issue
Block a user