docs(agent): restore master plan at .agents/PLAN.md with locked decisions through phase 92

Re-lands the master plan (removed from VCS in 281f355 as .agent/PLAN.md)
at .agents/PLAN.md — the path AGENTS.md and the codebase's PLAN §…
references use. Rebuilt from the original text plus every
owner-permitted revision recorded in the completed phases: A1–A17
anchors with the full revision log (auth, SSE thinking/tool/retry
events, agent tools, history, SPA shell), new A18–A20 (docs push,
deploy caching, security headers), current data model (11 tables),
locked persona + <tools> copy, current UI/UX contracts (46rem/92rem
column, theme tab, never-stale feedback, no autoscroll), the full
per-turn log line, and the phase 01–92 roadmap with the open TODO.md
items as next up.

.agents/PLAN.md

@@ -0,0 +1,825 @@
+# Brain of Reese — Master Plan
+
+> **Status:** v1 complete — phases 01–92 shipped (see §12);
+> `.agents/phases/todo/` is empty as of this rebuild (2026-09-10).
+> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in
+> the §2 Anchors table are settled — do not re-litigate them in a phase.
+> A change to a LOCKED anchor requires explicit owner permission and is
+> recorded as a dated revision note under the table — never a silent
+> deviation (AGENTS.md rule 3).
+>
+> **Provenance:** This file restores the master plan that lived at
+> `.agent/PLAN.md` until commit 281f355 (removed from VCS by the owner,
+> 2026-08-31). It is re-landed at `.agents/PLAN.md` — the path `AGENTS.md`
+> and the codebase reference throughout (`PLAN §…` comments) — rebuilt from
+> the original text plus every owner-permitted revision recorded in the
+> completed phases through 92.
+>
+> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
+> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
+> theme (Phase 08); clickable document viewer (Phase 10); thinking display
+> (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll
+> (Phase 18, owner choice 2026-08-23); shared header (Phase 19);
+> whole-document context (Phase 24 — A7's 24k context cap removed,
+> owner permission 2026-08-24).
+>
+> **Revisions (2026-08-25 → 2026-09-09, owner permission):** `lite` summary
+> model (Phase 30); agent document tools `ls`/`read`/`grep`
+> (Phases 37/45/68/70/71/72); single-admin auth + saved chats + shares
+> (Phases 16/50/51/53/79); any-extension import (Phase 56); docs push
+> (Phase 59); SPA shell (Phase 76); LLM retries (Phase 67); login rate
+> limit (Phase 81); security headers (Phase 82); per-source ignore paths
+> (Phase 89); admin Theme tab (Phases 91/92). See the revision log under
+> §2 and the phase records in `.agents/phases/complete/`.
+
+---
+
+## 1. Mission
+
+A **knowledge base chatbot** that embeds Reese's homelab documentation —
+git repos, local directories, and uploaded archives registered on the admin
+Sources page — into a Postgres vector database and lets anyone ask *Reese*
+(the bot) questions about them.
+
+**Product feel:** a chippy, upbeat assistant that is optimistic about the
+user's ability and **radically honest** — if retrieval didn't surface
+anything relevant it says *"I haven't done anything like that"* and offers
+alternatives instead of hallucinating.
+
+### Delivered (v1, phases 01–92)
+- Chat UI — mobile-friendly, WCAG 2.1 AA, no CDN — as a single-page shell
+  with deep-linkable views: Chat `/`, Sources `/sources.html`, Git sources
+  `/git-sources.html`, Tuning `/tuning.html`, History `/history.html`,
+  Tokens `/tokens.html`, Theme `/theme.html`, plus Login, Shared chat, and
+  Document viewer/edit pages.
+- RAG over text knowledge files — the A9 family by default
+  (`md, markdown, txt, yaml, yml, json, py` + the Podman quadlet family +
+  `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension,
+  A9 revised) — from admin-registered **git sources**, **local directory
+  sources**, and **uploaded archives**.
+- Hybrid retrieval (pgvector cosine ∪ Postgres FTS, RRF-fused), an honesty
+  gate with honest deflection, whole-document context (never truncated),
+  `lite`-model summaries of non-markdown docs, a `lite`-generated KB
+  overview, and grounded-turn agent tools (`ls`/`read`/`grep`).
+- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat,
+  with streamed thinking), `embed` (embeddings, **768 dims — verified**),
+  `lite` (one-shot: document summaries, KB overview).
+- Single-admin signed-cookie auth + admin-issued access tokens; saved
+  chats with anonymous share links and KB-generation staleness; steering
+  notes; save-answer-as-doc git push; per-source ignore paths; admin Theme
+  tab; deploy-safe asset caching; security headers; login rate limit.
+- Idempotent delta import (`uv run python -m scripts.import_docs`), the
+  one-click UI **Sync sources** button, and ample server logging +
+  explicit UI feedback (never a stale submit button).
+
+### Still out of scope (deliberately)
+- Binary / non-text content ingestion.
+- Real-time document watching — the import script / Sync button is the
+  refresh loop (commit → re-run).
+- PR tooling for docs push — the owner opens the PR themselves.
+- Multi-admin / per-user accounts — one admin + hand-out tokens is the
+  access model.
+
+---
+
+## 2. Architectural Anchors (LOCKED DECISIONS)
+
+| # | Component | Decision | Rationale | Status |
+|---|-----------|----------|-----------|--------|
+| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
+| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
+| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
+| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
+| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat), **`embed`** (embeddings), **`lite`** (one-shot — A5 extended, phase 30); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED (extended phase 30) |
+| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
+| A7 | Retrieval→context | **Hybrid:** cosine top-100 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM — **never truncated** (A7 revised 2026-08-24; vector window widened 30→100 on 2026-08-22; summary chunks are first-class retrieval targets, phase 30) | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; the 100-wide vector window lets double-hit chunks outrank generic templates (measured on the live 2774-chunk KB, 2026-08-22); whole-document context contract preserved — no `[…truncated…]` cap (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
+| A8 | Honesty gate | **Deflection mode** (LLM must NOT pretend to know; offers 2–3 alternative questions about things it DOES have notes on) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture). Deflected prompt: weak-hit *titles only* (never document content) + the `DEFLECT_MODE` marker + the phase-71 plain-text line "Reply in plain text only — you have no tools in this mode." | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged. Phase 71 (owner-permitted 2026-09-03): the one added line closes the tool-markup door at the prompt — the `DEFLECT_MODE` marker contract is unchanged | LOCKED (revised 2026-08-21) |
+| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** + the Podman quadlet family (`container, network, volume, image, pod, kube, swap, os, endpoint`) + `j2` (default, `BOR_IMPORT_EXTENSIONS` — may name **any** well-formed extension or narrow to a subset, A9 revised 2026-08-31), **hidden (dot) directories/files skipped**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …), plus **per-source ignore paths** (phase 89: raw source-relative prefixes, no globs, `git_sources.ignore_paths`) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content. Phase 47 (2026-08-27): quadlet + j2 join the default — plain-text chunking. Phase 56 (2026-08-31): any well-formed extension allowed (the A9 family is a default, not a ceiling); a blank list or malformed token fails startup loudly, naming the value | LOCKED (revised 2026-08-31) |
+| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED (revised phase 16 — see revision log) |
+| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED (extended phase 76 — see revision log) |
+| A12 | Aux services | **None** (no Valkey, no SeaweedFS, no queue) — everything in-process: session state is a signed cookie (no store), the Sync button runs the import in-process, the login rate limit is in-memory per-process | No server-side session state; a homelab single-box deploy; add later only if a need appears | LOCKED |
+| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** | Standard, reversible, reviewable schema history | LOCKED |
+| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
+| A15 | Chat transport | **SSE streaming** from `POST /api/chat` — event types `thinking`, `delta`, `tool`, `retry`, `done`, `error` (§4) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED (extended phases 17/37/45/67 — see revision log) |
+| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
+| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
+| A18 | Docs push | Save a chat answer as documentation: server-side draft (`doc_drafts`, the long body never lives in a URL — an unguessable `uuid4` token is the edit-screen URL credential), pushed to a generic git remote (`BOR_DOCS_REPO` — URL or local path; no `gh`, no GitHub assumption) on a dedicated branch (`git push --ff-only`, re-cut from `BOR_DOCS_BASE_BRANCH` per push); **no PR tooling** — the owner opens the PR; the feature is inert (hidden + 409) while `BOR_DOCS_REPO` is empty | Phase 59, owner permission 2026-08-30 (decisions D1–D3): answers should flow back into the docs with a review step, without coupling the app to one forge; the empty-repo kill switch mirrors the git-sources env-fallback pattern | LOCKED (added phase 59) |
+| A19 | Deploys & caching | One Starlette middleware: HTML pages ship `Cache-Control: no-cache` (no `etag`/`last-modified`); assets are versioned (`/assets/*?v=<token>`) and ship `Cache-Control: public, max-age=31536000, immutable`; **the token is the deploy** — the short SHA of git `HEAD` (content hash of the `frontend/` tree without a `.git`) | A deploy is a commit — the browser must see it without a hard refresh; no CDN, no new services, no build-step change (phases 33 + 54) | LOCKED (added phase 33, revised phase 54) |
+| A20 | Security headers | Every response carries `Content-Security-Policy: default-src 'self'` (the themed HTML pages additionally allow the inline `<style>` hash of the current theme), `frame-ancestors 'none'`, and `X-Content-Type-Options: nosniff`; login is rate-limited in-memory (10 attempts / 15 min, stdlib-only, per-process — A12) | Owner-approved policy, phase 82 (audit SEC findings); the CSP hash keeps the theme's pre-paint `<style>` legal without `unsafe-inline` anywhere else | LOCKED (added phase 82) |
+
+### Revision log (owner permission noted — the established style)
+
+> **A10 revision (phase 16, owner permission 2026-08-22):** single-admin
+> signed-cookie auth — public: health (and, until the phase-79 split
+> below, chat / documents / suggestions); admin-only: docs catalog +
+> steering. The row above keeps the original v1
+> decision text; the public API surface stays stateless (the signed
+> session cookie is the only session state) — recorded as a revision,
+> not a silent deviation. Fail-loud at boot while `BOR_ADMIN_PASSWORD`
+> or `BOR_SESSION_SECRET` is empty.
+>
+> **A10 UI revision (phase 19, owner permission 2026-08-23):** the
+> "Sources" nav link is hidden from anonymous users on all pages — the
+> soft-gate page and the API split above are unchanged.
+>
+> **A10 extension (phase 50, owner permission 2026-08-29):** conversations
+> the owner explicitly **Saves** persist in `saved_chats` (one row per
+> saved conversation, the `bor.chat.v1` message list as JSONB); `POST
+> /api/chat` itself stays stateless and nothing is stored about a
+> conversation that was not saved.
+>
+> **A10 extension (phase 51, owner permission 2026-08-30):** a saved chat
+> can be shared anonymously — `share_token` (NULL = private, `uuid4` =
+> publicly readable at `/shared/<token>` without any session); unsharing
+> revokes it. Shared chats are the only content that stays anonymous.
+>
+> **A10 extension (phase 53):** saved chats stamp the KB generation
+> (`sources_version`, bumped once per KB-changing sync in `sources_meta`)
+> and are surfaced as **stale** (regenerable) when older than current.
+>
+> **A10 extension (phase 74, owner-locked 2026-09-08):** `POST /api/chat`
+> accepts an optional client-provided `history` (the prior turns,
+> stateless per A10 — nothing is stored): kept while both budgets hold
+> (`BOR_HISTORY_MAX_TURNS`=40 turns, `BOR_HISTORY_MAX_CHARS`=24000 chars
+> counting prior thinking), trimmed newest-first, a turn dropped WHOLE
+> never cut mid-answer; `who="brain"` turns map to `assistant` messages
+> with `reasoning_content` only when thinking is non-empty (the
+> preserve-thinking wire convention).
+>
+> **A10 extension (phase 79, owner permission 2026-09-08):** admin-issued
+> access tokens (`bor_` + 32 hex) — the only stored credential is the
+> SHA-256 digest of the full token (shown once at create); a token user
+> can chat and open cited documents, nothing else (chat, suggestions,
+> documents; the admin surfaces stay 403); revocation is immediate
+> (live-checked on the holder's next request).
+>
+> **A11 extension (phase 76, owner permission 2026-09-08):** the navbar
+> views (Tuning, Sources, Git sources, History, Tokens, Theme) are views
+> of ONE shell document (`frontend/index.html`) — the client-side router
+> (`frontend/assets/router.js`) shows the view matching
+> `location.pathname`, so direct URLs deep-link; the shell routes are
+> registered after the API routers and before the static catch-all, so
+> the A19 caching contract applies untouched.
+>
+> **A15 extension (phase 17, owner permission 2026-08-23):** the contract
+> gains `{"type":"thinking","text":"…"}` — the model's reasoning ahead of
+> the `delta` events (`turbo` emits `delta.reasoning_content` chunks
+> before the first content chunk, verified live 2026-08-23);
+> `BOR_STREAM_THINKING=0` suppresses the frames server-side (the chars are
+> still counted for the §9 line). `delta` and `done` shapes unchanged.
+>
+> **A15 extension (phase 37, owner permission 2026-08-26):** the contract
+> gains `{"type":"tool","name":"…","argument":…|null}` — the model's
+> document tool calls on grounded turns. Deflected turns keep the direct
+> stream — the LOW prompt never carries tools.
+>
+> **A15 revision (phase 45, owner permission 2026-08-27):** the phase-37
+> per-tool budgets (`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS`) are
+> **removed** (owner: "allow the LLM to make as many tool calls as it
+> wants" — `TODO.md` L8); `BOR_AGENT_MAX_ROUNDS` (default 10) caps the
+> tool rounds and `0` disables the tools entirely (the pre-phase-37
+> path — one `tools=None` request, the kill switch). The `tool` and
+> `done` shapes are unchanged; `done.sources` includes the agent-read
+> documents (deduped by `(source, path)`, order preserved).
+>
+> **A15 extension (phase 67, owner-locked 2026-09-01):** the contract
+> gains `{"type":"retry","attempt":n,"max_attempts":N}` — when the aipi
+> endpoint dies **before a request has streamed its first output frame**,
+> the request restarts up to `BOR_LLM_RETRIES` (default 3) times, a flat
+> `BOR_LLM_RETRY_DELAY` (default 5 s) between attempts; every restart is
+> announced with a `retry` frame so the UI shows the transient
+> "Communication interrupted — retrying (n of N)…" status, not an error.
+> Exhaustion and any failure after the first frame keep the terminal
+> `error` frames (byte-identical). The pre-stream question embedding
+> retries on the same budget.
+>
+> **A15 revision (phase 70, owner permission 2026-09-03):** the tool
+> surface is the harness-trained **`ls` / `read` / `grep`** (replacing
+> phase 37's `list_documents` / `read_document`, plus the `grep` locator
+> of phase 68): `ls` lists indexed documents as `source: X | path: Y |
+> title: Z` lines (optional `path` = a **source name**, not a directory);
+> `read` takes the combined `source/path` string **including the source
+> name** and appends the full document, never truncated; `grep` is a
+> case-insensitive fixed-substring locator (up to 20
+> `source/path:line: text` matches, optional one-document scope) — a
+> locator that adds **no** source or context (the model `read`s the
+> winner). `argument` in the `tool` frame is the single string the model
+> passed (`read`'s `path`, `grep`'s `pattern`, `ls`'s `path`) or null.
+>
+> **A12 reaffirmed (phases 32/64/81):** the Sync button and the model
+> liveness probe run the work **in-process** (no queue, no new service);
+> the login rate limit is in-memory per-process stdlib (a restart clears
+> the counter — accepted).
+
+---
+
+## 3. High-Level Architecture
+
+```
+                       ┌────────────────────────────────────────────┐
+                       │              Podman Compose                │
+   Browser             │  ┌──────────────────────────────────────┐  │
+   ┌──────────┐  HTTP  │  │  brain-of-reese/app (FastAPI)        │  │
+   │ index.html│◄──────┼─►│  • SPA shell + views (no CDN, A11)   │  │
+   │ app.js    │  SSE   │  │  • SessionMiddleware (signed cookie)│  │
+   └──────────┘        │  │  • SecurityHeaders + caching middle-│  │
+                       │  │    ware (A19/A20)                   │  │
+                       │  │  • /api/* (chat, docs, git-sources,  │  │
+                       │  │    steering, sync, chats, tokens, …)│  │
+                       │  │  • RAG pipeline (app/rag/)          │  │
+                       │  │    embed→retrieve→gate→agent→turbo  │  │
+                       │  └──────┬──────────────────┬───────────┘  │
+                       │         │ SQL (psycopg)    │ OpenAI-compat│
+                       │  ┌──────▼──────┐   ┌───────▼────────────┐ │
+                       │  │ db:         │   └─────────┬──────────┘ │
+                       │  │ postgres:17 │             │            │
+                       │  │ + pgvector  │             │            │
+                       │  └─────────────┘             │            │
+                       └──────────────────────────────┼────────────┘
+                                                      ▼
+                                       https://aipi.reeseapps.com/v1
+                                       (self-hosted: turbo, embed, lite)
+
+   Source registry (Postgres `git_sources`, admin-managed):
+     git repos  → cloned/pulled into BOR_SOURCES_DIR (git CLI via
+                  scripts/git_sync.py — the ONLY place git is invoked)
+     local dirs → walked directly (no checkout)
+     archives   → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)
+
+   Offline tooling (same repo, same venv):
+     scripts/import_docs.py    → walks the sources, chunks, embeds, upserts
+     scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
+     scripts/llm_probe.py      → verifies models + embedding dim
+     scripts/model_benchmark.py / test_embed_model.py /
+     test_summary_model.py     → model benchmarks (benchmarks/)
+```
+
+### Component breakdown
+| Component | Responsibility | Lives in |
+|-----------|----------------|----------|
+| **App (FastAPI)** | Serves the SPA shell + `/api`; session auth; middlewares (caching A19, security headers A20) | `app/main.py`, `app/core/` |
+| **RAG pipeline** | embed → hybrid retrieve (cosine ∪ FTS, RRF) → honesty gate → prompt assembly (persona + KB overview + tuning + documents + tools) → `turbo` streamed, with the agent loop for grounded turns | `app/rag/` (retriever, prompts, agent, llm, scaffolding, chunker, importer, summarizer, overview, git_sources, archive_upload, source_removal, sources_meta, suggestions) |
+| **Importer** | Source walk (A9 scope, ignore paths), sha256 delta, format-aware chunking, batched embedding, two-phase upsert/prune, `lite` summaries, KB overview regeneration, sources-version bump | `app/rag/importer.py` + `scripts/import_docs.py` + `app/api/sync.py` (UI Sync) |
+| **Auth & access** | Fail-loud admin config; signed cookie (`require_admin` / `require_user`); token auth + admin token API; login rate limit | `app/core/auth.py`, `app/core/tokens.py`, `app/core/rate_limit.py`, `app/api/auth.py`, `app/api/tokens.py` |
+| **DB** | 11 tables + `vector` extension (migrations 0001–0015, each reversible) | `db/` image, `alembic/versions/` |
+| **Frontend** | SPA shell + views (chat, sources, git-sources, tuning, history, tokens, theme, login, shared, document viewer), feedback state machine, theming | `frontend/` (index.html shell + document.html / shared.html / login.html / doc-edit.html) |
+
+### Chat data flow (LOCKED A7/A15)
+```
+user question (+ optional client history)
+  → POST /api/chat {message, history?}
+  → embed(question)                    [aipi /v1/embeddings, model=embed; retried]
+  → cosine top-100 ∪ FTS top-30 (OR tsquery, ts_rank)   [pgvector + PG FTS]
+  → RRF fuse (k=60) → docs ranked by best fused chunk score
+  ├─ best cosine >= BOR_RELEVANCE_THRESHOLD (0.62) OR fts_hits > 0
+  │    → HIGH: locked persona + <knowledge_base> + <tuning> + full top-2
+  │      documents + <tools>  →  agent loop (ls/read/grep, round-capped)
+  │      → turbo streamed: thinking → tool → delta events → done
+  └─ else → LOW: DEFLECT_MODE prompt (weak titles only + plain-text line)
+            → turbo streamed (no tools, byte-identical path) → done
+  → query_log row + the per-turn log line (§9)
+  → final "done": {deflected, sources[] (retrieval + agent-read, deduped),
+                   suggestions[]}
+```
+
+---
+
+## 4. API Design
+
+Errors: standard JSON `{detail: str}`. Auth: `require_admin` (the
+admin session), `require_user` (admin **or** a live access token), or
+public (no dependency) — per the A10 revisions.
+
+| Method | Path | Auth | Purpose |
+|--------|------|------|---------|
+| GET | `/api/health` | public | Liveness + db up/down + version |
+| GET | `/api/config` | public | UI identity: app name, placeholder, footer text (effective `ui_settings` over env) |
+| POST | `/api/login` | public (rate-limited) | Password → signed `bor_session` cookie (204) |
+| POST | `/api/logout` | public | Clears the session cookie (204) |
+| GET | `/api/whoami` | public | `{authenticated, role: admin\|user\|anonymous}` |
+| POST | `/api/token-auth` | public | Access token (`bor_…`) → same cookie, role `user` (204) |
+| GET | `/api/suggestions` | user | Onboarding chips: `BOR_SUGGESTIONS` seed until any question is saved, then the last 3 questions asked |
+| GET | `/api/docs` | admin | Indexed document list (source, path, title, chunks, indexed_at) |
+| GET | `/api/documents/content?source=…&path=…` | user | One indexed document's full content (feeds the viewer) |
+| PATCH | `/api/documents/summary` | admin | Re-generate/edit one document's `lite` summary (phase 57) |
+| POST | `/api/chat` | user | RAG chat turn → **SSE stream** |
+| GET/POST/PUT/DELETE | `/api/steering[/<id>]` | admin | Steering notes (1–2000 chars each; injected as `<tuning>`) |
+| GET | `/api/git-sources` | admin | Source registry rows (kind: `git`\|`local`, ignore paths) |
+| POST | `/api/git-sources` | admin | Add a git URL or local directory (201) |
+| DELETE | `/api/git-sources/<id>` | admin | Remove a source (re-importable; never touches the KB until a sync) |
+| POST | `/api/git-sources/upload` | admin | Upload an archive source (202 → unpack + register; the scan is deferred to Sync — phase 90) |
+| GET | `/api/git-sources/upload/status` | admin | Upload progress |
+| POST | `/api/sync` | admin | Run the full source refresh in-process (202; 409 while one is running) |
+| GET | `/api/sync/status` | admin | Sync progress (poll every 2 s; no client timeout) |
+| GET/POST/PUT/DELETE | `/api/chats…` | admin | Saved chats (list / save / update / delete / `POST /api/chats/<id>/share`) |
+| GET | `/api/shared/<token>` | **anonymous** | One shared chat's JSON snapshot |
+| GET | `/shared/<token>` | **anonymous** | The shared-chat page |
+| POST/GET/PUT | `/api/doc-drafts[/<token>]` | admin | Save an answer as a doc draft (201); load/edit by token |
+| POST | `/api/doc-drafts/<token>/push` | admin | Commit + push to `BOR_DOCS_REPO` (A18; 409 while the repo is unconfigured) |
+| GET/POST | `/api/tokens`, POST `/api/tokens/<id>/revoke` | admin | Access-token surface (the token string appears once, at create) |
+| GET/PUT | `/api/ui-settings` | admin | The single `ui_settings` row (Theme tab; NULL = default, B1) |
+
+> Phase 79: chat, suggestions, and cited documents moved behind
+> `require_user` (admin **or** a live access token) — the only fully
+> anonymous content is the shared-chat read above.
+
+### SSE contract (`POST /api/chat`)
+```
+data: {"type":"thinking","text":"…"}\n\n        ← model reasoning (before the first delta)
+data: {"type":"tool","name":"ls","argument":null}\n\n   ← grounded turns only, one per model call
+data: {"type":"retry","attempt":1,"max_attempts":3}\n\n ← endpoint died pre-first-frame
+data: {"type":"delta","text":"Hey! "}\n\n         ← answer tokens
+...
+data: {"type":"done","deflected":false,
+       "sources":[{"source":"Homelab","path":"kubernetes.md","title":"…"}],
+       "suggestions":[]}\n\n
+```
+and on failure: `data: {"type":"error","detail":"…"}\n\n` (terminal —
+no `done`, no `query_log` row; a pre-stream DB outage is a plain 503
+JSON instead).
+
+Client rules:
+- Render deltas live; render `thinking` in the collapsible block above
+  the answer (auto-collapses on the first `delta`, toggleable after,
+  persisted with the message).
+- Render each `tool` frame as a transient "calling tool" status (the send
+  button stays busy — "Stop" — for the whole turn; Stop closes the
+  connection, which cancels the server generator).
+- `retry` frames show the transient "Communication interrupted —
+  retrying (n of N)…" status — never an error banner.
+- On `done`: append source chips (clickable → document viewer) /
+  suggestion chips, clear the busy state.
+- Guards: a 300 s pre-token guard (`TURN_TIMEOUT_MS`) turns a hung
+  stream into the error state; a button can never sit "stuck" forever
+  (the §7.4 contract).
+
+---
+
+## 5. Data Model (PostgreSQL 17)
+
+Migrations `alembic/versions/0001…0015` (idempotent
+`CREATE EXTENSION IF NOT EXISTS vector`); every migration's downgrade is
+tested (A13). Single-row tables use `id = 1` (the `kb_overview`
+precedent).
+
+### `documents`
+| Column | Type | Notes |
+|--------|------|-------|
+| id | `UUID` PK | |
+| source | `VARCHAR(120)` | source name (dir basename / repo name / archive name) |
+| path | `VARCHAR(1000)` | relative to the source root |
+| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
+| title | `VARCHAR(500)` | first markdown H1, else file stem |
+| content | `TEXT` | **full text — the RAG context** |
+| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
+| indexed_at | `TIMESTAMPTZ` | |
+| summary | `TEXT` NULL | `lite`-model summary, non-markdown docs only (phase 30) — NULL on markdown docs and on the best-effort failure path |
+| — | `UNIQUE (source, path)` | upsert key |
+
+### `chunks`
+| Column | Type | Notes |
+|--------|------|-------|
+| id | `UUID` PK | |
+| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
+| position | `INT` | 0-based order; **−1 = the summary chunk** |
+| content | `TEXT` | chunk text (format-aware) |
+| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
+| is_summary | `BOOL` default false | marks the position −1 summary chunk (phase 30) |
+| tsv | `TSVECTOR` | generated `to_tsvector('english', content) STORED` + GIN index (A7) |
+
+> No vector index in v1: sequential scan is fine at this corpus size.
+> Revisit with an HNSW index if retrieval latency grows (§13).
+
+### `query_log`
+`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT,
+deflected BOOL, sources TEXT (comma-joined source/path, retrieval +
+agent-read deduped), latency_ms INT, created_at TIMESTAMPTZ` — the
+durable record behind threshold tuning.
+
+### `steering_notes` (phase 15)
+`id UUID PK, note TEXT (1–2000 chars, API-enforced), created_at` —
+injected oldest-first into every turn's system prompt as `<tuning>`,
+char-budgeted (`BOR_STEERING_MAX_CHARS`).
+
+### `kb_overview` (phase 31)
+`id INT PK (=1), content TEXT, updated_at` — single-row `lite`-generated
+outline of the KB, regenerated when an import changed the KB; injected as
+`<knowledge_base>` (one indexed PK read per turn — no LLM call).
+
+### `git_sources` (phases 35/38/89)
+`id UUID PK, url TEXT unique, kind TEXT ('git'|'local' CHECK),
+path TEXT unique NULL (local only), ignore_paths JSONB default '[]'
+(phase 89 — normalized source-relative prefixes), added_at` — the
+admin-managed source registry; DB rows are the source of truth,
+`BOR_GIT_SOURCES` (git-only) is the empty-table fallback.
+
+### `saved_chats` (phases 50/51/53)
+`id UUID PK, title VARCHAR(500) (auto: first question), messages JSONB
+(the exact bor.chat.v1 record — raw text, never HTML),
+share_token UUID unique NULL (phase 51 — NULL = private),
+sources_version INT default 0 (phase 53 — the KB generation at save),
+created_at, updated_at`.
+
+### `sources_meta` (phase 53)
+`id INT PK (=1), version INT (KB generation), updated_at` — bumped
+exactly once per sync that changed the KB (`added + updated + pruned >
+0`); a `saved_chats` row older than it is *stale*.
+
+### `doc_drafts` (phase 59)
+`id UUID PK, token UUID unique (the URL credential — never the row id),
+title TEXT, path TEXT (in-repo file path, default docs/<slug>.md),
+body TEXT (the raw markdown answer), status TEXT ('draft'→'pushed'),
+branch TEXT NULL, commit_sha TEXT NULL, created_at, updated_at`.
+
+### `api_tokens` (phase 79)
+`id UUID PK, label VARCHAR(120) (display-only), token_hash VARCHAR(64)
+unique (SHA-256 of the FULL `bor_…` string — the plaintext exists only
+in the 201 create response, shown once), created_at, last_used_at NULL,
+revoked_at NULL (set = dead, live-checked)`.
+
+### `ui_settings` (phase 91)
+`id INT PK (=1), app_name / input_placeholder / footer_text VARCHAR(300)
+NULL, bg / surface / ink / ink_soft / line / grid_line / brand /
+brand_soft / brand_ink VARCHAR(7) NULL` — the Theme tab's single row;
+**B1 (owner-locked 2026-09-09): every column NULL/empty = "use the
+default"** (the env value for the strings — `BOR_APP_NAME` etc. — and the
+built-in palette for the colors; colors have no env fallback).
+
+### Document state transitions
+```
+unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
+                     │
+                     └──file deleted/--prune/ignored──▶ removed (chunks cascade)
+```
+
+### Chunking policy (format-aware — `app/rag/chunker.py`)
+- **Markdown:** split on `## `/`### ` headings; sub-split sections over
+  `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
+  `BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
+  preceding heading.
+- **YAML/YML:** top-level keys + `---` separators (key line kept as
+  anchor); **JSON:** pretty-printed, top-level keys; **Python:**
+  top-level defs/classes (stdlib `ast`); **txt:** paragraphs.
+- **Quadlet family + `j2`:** plain-text chunking — no format-specific
+  splitter (A9 revised 2026-08-27).
+- Every format honors the **1200-char hard cap** (aipi ~1024-token
+  request limit; ~1.4 chars/token) — oversized blocks split by line.
+
+---
+
+## 6. RAG Pipeline & Persona
+
+### Locked system prompt (sent with every chat turn — `app/rag/prompts.py`)
+```
+You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
+homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
+optimistic about the user's ability to do things.
+
+Rules:
+1. Answer ONLY from the provided document context. Cite which document(s)
+   you used, by path.
+2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
+   the docs are the value.
+3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
+   Offer 2-3 alternative questions about things you DO have notes on.
+4. Never invent facts, hosts, or steps that are not in the context.
+5. Keep answers tight: short paragraphs, bullets where helpful.
+
+<relevance>{HIGH|LOW}</relevance>
+```
+Change this text through the plan, not in code (phase 03 convention).
+> **§6 revision (2026-08-22, owner permission):** the owner's working-tree
+> persona edits are preserved — no mandated "you've got this" tagline and
+> no mandated deflection opening; the honesty gate itself is unchanged.
+
+**Prompt section order (both modes):** `<relevance>` →
+`<knowledge_base>` (when the `kb_overview` row is non-empty) →
+`<tuning>` (when steering notes exist) → mode body. An empty section
+omits itself — a no-notes/no-overview prompt is **byte-identical** to the
+pre-steering text (phase 15 convention, re-used by phase 31). Both
+sections are char-budgeted (`BOR_KB_OVERVIEW_MAX_CHARS` /
+`BOR_STEERING_MAX_CHARS`) with the shared `[…truncated…]` marker.
+
+- **HIGH mode** appends the full top-N document texts under
+  `<documents>` as `<document source="…" path="…" title="…">` blocks,
+  then the `<tools>` section (below) — the instructions are the last
+  thing the model reads.
+- **LOW mode** (deflection) appends the `DEFLECT_MODE` marker + the
+  weak-hit **titles only** (never document content) + the phase-71
+  plain-text line "Reply in plain text only — you have no tools in this
+  mode." The `DEFLECT_MODE` marker's *presence* is the E2E mock's key —
+  wording may not change without the mock.
+
+### The `<tools>` section (HIGH only — LOCKED, phase 37/70/71/72)
+Teaches the harness-aligned shapes: `ls` (lists indexed documents as
+`source: X | path: Y | title: Z`; optional `path` = a **source name**,
+not a directory); `read` (combined `source/path` **including the source
+name** — a bare path will not resolve; do **not** call `read` for a
+document already in `<documents>` — answer from the text in the prompt);
+`grep` (exact case-insensitive substring, up to 20
+`source/path:line: text` matches — a **locator, not a context-adder**;
+plain substring, NEVER a regex; for a normal pass only `pattern`).
+Discipline rules (phase 72, from live gate telemetry): exactly one tool
+call per reply (a batched second call is discarded); never repeat a
+refused or already-succeeded call; `path` arguments are document
+identities, never source names. The round cap is enforced by the agent
+loop, not re-stated in the prompt (phase 45).
+
+### Agent loop (grounded turns — `app/rag/agent.py`)
+A grounded turn runs `run_agent`: the model may call the tools as many
+times as it wants until it answers or `BOR_AGENT_MAX_ROUNDS` (default 10)
+forces one final no-tools answer. Each executed call streams a `tool`
+frame; rejected calls (bad identity shapes) get the **teaching refusals**
+(phase 72 — deterministic, model-free) instead of execution. Tool
+execution is Postgres-only (no extra LLM round trip). **Deflected turns
+never run the agent** — the direct `chat_stream` path is
+byte-identical (A8). Raw chat-template scaffolding tokens
+(`<|tool_call_start|>…`) are filtered deterministically
+(`app/rag/scaffolding.py`, phase 71) with exactly one bounded recovery;
+a wipe that survives recovery settles with the structured `error` frame.
+
+### Retrieval (hybrid — A7/A8)
+- Embed the question (`embed`, 768-d) → cosine top-100 candidates.
+- Lexical: OR tsquery over the question's tokens → FTS top-30 by
+  `ts_rank` (a stored `tsvector`, GIN-indexed).
+- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent
+  docs ranked by best chunk's fused score → top `BOR_TOP_N_DOCS` (2) →
+  full content — **never truncated**. Summary chunks (position −1)
+  resolve to the parent document like any chunk; the per-turn line
+  reports `summary_hits=`.
+- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
+  (default 0.62 — the `embed` model's live cosines cluster ~0.6–0.85)
+  **and** zero FTS hits among the candidates.
+
+---
+
+## 7. UI/UX Strategy
+
+### 7.1 Layout structure
+- **App frame:** sticky header (64px, `--header-h`) + `<main>` (flex-grow)
+  + footer. Container: `max-width: 72rem; margin-inline: auto;
+  padding-inline: 1.25rem`. `body { min-height: 100dvh }` — no fixed body
+  height (it pins the sticky navbar's travel range, phase 60).
+- **SPA shell (phase 76):** the navbar views are views of ONE shell
+  document — the router shows the view matching `location.pathname`;
+  direct URLs deep-link. Switching views keeps a live stream alive where
+  the contract requires (phase 73: hidden-tab streaming).
+- **Chat:** a *centered reading column* — `--chat-column: 46rem` base,
+  **2× (92rem) at ≥1500px wide desktops** (owner instruction 2026-08-31,
+  phase 58 — chat + shared + document view). Deliberate: chat is a
+  vertical conversation — a centered, capped column is the correct
+  pattern (NOT a layout bug); the 72rem frame + header/footer keep it
+  from reading as a hairline.
+- **Sources page:** full-width responsive **table** (min 640px,
+  horizontal scroll wrapper on small screens) + stat cards in
+  `grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
+  No skinny single-column lists anywhere: lists/tables/grids use ≥80–90%
+  of the container width.
+- **Document viewer:** near-fullscreen modal on the current page
+  (`#doc-modal`) — same-page, no new tab; back keeps the page state.
+- **Mobile (≤640px):** hamburger nav; suggestion chips horizontally
+  scrollable; composer stays reachable with `safe-area-inset-bottom`;
+  touch targets ≥44px.
+
+### 7.2 Accessibility (WCAG 2.1 AA) & theming
+- Semantic landmarks on every view: `<header>`, `<nav aria-label>`,
+  `<main>`, `<footer>`; skip-link to `#main`.
+- Every control labeled: visible `<label>` or `aria-label` (icon-only
+  buttons always get `aria-label`); form inputs have (visually-hidden)
+  labels.
+- Live regions: message stream `aria-live="polite"`; typing indicator
+  `role="status"`; banners `role="status"`; errors `role="alert"`.
+- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by
+  all animations and stills the background texture.
+- **Theming (phases 08 → 91/92):** the built-in dark palette
+  (`app/core/theming.py::BUILTIN_COLORS` — bg `#0f0a0a`, surface
+  `#1a0f0f`, ink `#f0e6e6`, ink-soft `#b8a8a8`, line `#2d1a1a`,
+  grid-line `#4a2626`, brand `#f43f5e`, brand-soft `#2d0a0a`,
+  brand-ink `#fca5a5`) renders byte-identically when nothing is set.
+  Text-on-surface pairs verify ≥4.5:1 (ink/surface ≈15.3:1,
+  ink-soft/surface ≈8.2:1, brand-ink/brand-soft ≈9.5:1, dark-on-brand
+  ≈5.4:1; light ink on brand ≈3.0:1 — never used for text). The admin
+  **Theme tab** (`/theme.html`, phase 91) edits the 9 identity colors +
+  the 3 identity strings, persists them in `ui_settings` (B1: NULL =
+  default), and injects them as an inline `<style>` **before first paint**
+  (no red flash) — the retired CSS-file `BOR_THEME` env is ignored; the
+  page's CSP carries the theme's hash (A20).
+
+### 7.3 No external dependencies
+- System font stack only (no font files to bundle, no CDN fonts).
+- Zero `<script src="https://…">` / `<link href="https://…">` — enforced
+  by an integration test on the index page and re-checked by every UI
+  phase's verification step (AGENTS.md rule 6).
+- Markdown rendering is a small local function (escape-first, then
+  transform) — XSS-safe, no library.
+- Container build minifies the frontend with esbuild (build-time only).
+
+### 7.4 Visual feedback standard (the "never stale" contract)
+| State | UI |
+|-------|----|
+| **Idle** | Send button enabled, label "Send". |
+| **Thinking (pre-token)** | 3-dot typing bubble + button busy, label "Thinking…"; elapsed hint after a few seconds. |
+| **Thinking (model reasoning)** | Collapsible `.thinking` block streams open (the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); it scrolls internally and never drags the page (phases 21/43). |
+| **Calling a tool** | Transient "calling tool" status line (same pattern as the thinking indicator); the button stays busy — "Stop" — for the whole turn (phase 37; the `read` line shows progress for big reads, phase 87). |
+| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
+| **Retrying** | `retry` frames → transient "Communication interrupted — retrying (n of N)…" status — never an error banner (phase 67). |
+| **Done (answer)** | Source chips under the bubble (mono, path-based, clickable → viewer); button re-enabled; answer actions (Tune / Save / Copy / Retry, phase 65). |
+| **Done (deflected)** | Amber bubble + "Maybe try:" suggestion chips (closest indexed titles). |
+| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled; "Retry answer" re-asks the last question (phase 49). |
+| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
+| **Stop** | "Stop" closes the SSE connection; the partial answer stays, marked stopped; no autoscroll. |
+| **Guard** | 300 s pre-token guard (`TURN_TIMEOUT_MS`, raised from 120 s) → error state (a button can never sit "stuck" forever). The guard counts only visible time — a hidden tab doesn't burn it (phase 73). |
+| **Scroll** | **No auto-follow during a turn** (owner direction 2026-08-27, phase 42 — revising the phase-18 follow-the-bottom choice): the viewport moves only on user intent — submit reveals the user's message; restoring a saved conversation lands one-shot on the latest message. |
+
+> Every failure path re-enables its controls (the "never stale" rule) —
+> pinned by unit tests on the `app.js` state machine and by the story
+> E2E suites.
+
+### 7.5 Component inventory (core ids used by tests — shell era)
+Chat: `#messages`, `#empty-state`, `#suggestions`, `.suggestion-chip`,
+`#composer`, `#message-input`, `#send-btn` / `#send-label`,
+`#typing-indicator`, `.msg.user/.msg.brain .bubble`, `.source-chip`,
+`.msg.brain.is-deflected`, `.thinking` / `.thinking-text`, `#kb-banner`,
+`#app-version`, `#doc-modal` (the viewer modal). Shell/nav: `#app-nav`
+(+ the hamburger on mobile), `#main`. Sources: `#stat-docs`,
+`#stat-chunks`, `#stat-last`, `#docs-table`, `#docs-tbody`,
+`#sources-empty`, the Sync button. Git sources: `#git-sources-table`,
+`#git-source-form`, `#archive-upload-form`, `#ignore-editor-dialog`.
+History: `#history-tbody`, `#history-gate`. Tokens: `#tokens-table`.
+Theme: `#theme-form` (the 9 color fields + 3 string fields). Login gate:
+`#auth-gate` (chat) / `#sources-gate` / `#history-gate` (soft gates).
+
+---
+
+## 8. Debugging (debugpy protocol)
+
+- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit
+  test — A14).
+- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`,
+  **non-blocking**, app continues; IDE attaches on demand.
+- Entry point: `app/core/debugging.py::configure_debugging()` called at
+  the top of `app/main.py` module import — so `uv run uvicorn
+  app.main:app`, `python -m scripts.…`, and tests all honor it.
+- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host":
+  "localhost", "port": 5678}`.
+
+---
+
+## 9. Observability
+
+- **App logs:** single-line `timestamp LEVEL logger :: message` on
+  stdout; INFO by default (`BOR_LOG_LEVEL`); noisy third-party loggers
+  capped at WARNING. Every request-critical operation logs key=value
+  context at INFO — this pairs with the UI feedback contract (§7.4).
+- **Per-chat-turn log line (required — one per completed turn):**
+  ```
+  question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
+  kb_chars=N history_msgs=N threshold=… deflected=… sources=…
+  thinking_chars=… tool_calls=N total_ms=… retries=N
+  scaffold_stripped=N
+  ```
+  Field notes: `summary_hits=` (phase 30) — hit summary chunks whose
+  parent doc is in the context; `kb_chars=` (phase 31) — the injected
+  overview length; `history_msgs=` (phase 74) — kept history messages;
+  `sources=` — retrieval docs + agent-read docs, deduped (empty on
+  deflected turns); `thinking_chars=` (phase 17) — counted even when
+  `BOR_STREAM_THINKING=0`; `tool_calls=` (phase 37) — executed calls
+  (rejected calls don't count); `retries=` (phase 67) — 0 when nothing
+  was retried; `scaffold_stripped=` (phase 71) — stripped scaffolding
+  chars across the turn's requests, 0 on clean turns. A cancelled turn
+  writes neither the line nor a `query_log` row.
+- **Importer logs:** per-file `added | updated | unchanged | pruned` +
+  per-summary `import: summary source=… path=… chars=…` + the greppable
+  summary line (cron-safe):
+  ```
+  import: summary files=… added=… updated=… unchanged=… pruned=…
+  errors=… chunks=… embed_batches=… summaries=… summary_errors=…
+  formats=md:203,yaml:267,py:14     (highest count first)
+  ```
+- **`query_log` table:** durable record of every question (score, FTS
+  hits, deflection, sources, latency) for threshold tuning and doc-gap
+  finding.
+
+---
+
+## 10. Testing Strategy (LOCKED — A16)
+
+| Layer | Tooling | Runs | Gate |
+|-------|---------|------|------|
+| Unit | pytest | `uv run pytest tests/unit` | pass |
+| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
+| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
+| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
+
+- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
+  OpenAI-compatible API. Embeddings are genuine L2-normalized
+  token-overlap vectors, so the cosine threshold behaves like production:
+  on-topic questions retrieve, off-topic questions deflect.
+  `tests/e2e/slow_llm.py` is the slow/dead-endpoint variant for the
+  retry/timeout suites. `E2E_REAL_LLM=1` switches the fixture to live
+  aipi (needs an imported KB).
+- **E2E prerequisites:** `podman compose up -d db`; Chromium via
+  `uv run playwright install chromium`.
+- DB isolation: story E2E fixtures truncate `query_log` (and re-import
+  fixtures for import-dependent stories) per test module.
+- Lint/types gate: `uv run ruff check .` and `uv run pyright` clean.
+
+---
+
+## 11. Import & Update Workflow (documented in README)
+
+```
+# the whole update loop (git sources): commit → re-run
+uv run python -m scripts.import_docs                # sync-clone/pull + walk + index
+uv run python -m scripts.import_docs --prune        # also drop deleted/ignored files
+uv run python -m scripts.import_docs --source ~/OtherProject   # extra dirs
+# or one click: the admin Sources page's "Sync sources" button (in-process)
+uv run python -m scripts.eval_retrieval "How did I install gitlab?"
+uv run python -m scripts.llm_probe                  # sanity: models + dim
+```
+
+Behavior:
+- **Sources** come from the `git_sources` registry (admin page — the
+  source of truth): `git` rows are cloned (shallow, first time) or
+  `git pull --ff-only`ed; `local` rows are walked directly (a missing
+  directory fails the run loudly); uploaded archives live under
+  `BOR_UPLOAD_DIR` (zip-bomb guarded: absolute/`..`/escaping-symlink
+  members rejected; `BOR_UPLOAD_MAX_MB` caps compressed AND extracted
+  bytes). `BOR_GIT_SOURCES` is the **git-only** fallback while the
+  registry is empty. A failed source aborts the run — no partial junk.
+- **Delta:** sha256 per `(source, path)` — unchanged files skipped (no
+  re-embedding); changed files re-chunked + re-embedded with the
+  two-phase upsert (one transaction per file — a failed embedding leaves
+  the DB untouched).
+- **Prune:** `--prune` removes docs of the imported sources whose files
+  disappeared, no longer match the format filter, or now match a
+  per-source ignore path.
+- **Summaries (phase 30):** non-markdown files get a `lite` summary
+  (best-effort — a failure still indexes the doc) + the position −1
+  summary chunk (at most one per document — replaced on re-import).
+- **After a KB-changing run:** the `lite`-generated `kb_overview` is
+  regenerated and `sources_meta.version` bumps exactly once (phase 53) —
+  which is what marks older saved chats stale.
+- **Source removal:** removing a registry row marks the source for
+  cleanup on the next sync (its documents/chunks are pruned — phase 69).
+- Formats per A9 (revised); hidden dirs/files skipped; exclusion list
+  applied; per-source ignore paths (phase 89) applied as raw
+  source-relative prefixes.
+
+---
+
+## 12. Roadmap (one story → one phase → one Playwright gate)
+
+Phases live in `.agents/phases/` — `todo/` is **empty**; `complete/`
+holds the phase directories (`00_phase.md` + task files) as read-only
+history. Next free phase number: **93**.
+
+### Shipped (phases 01–92)
+| Range | Area | Highlights |
+|-------|------|-----------|
+| 01–10 | v1 core | foundation; import story; chat RAG; honest deflection; suggestion chips; loading feedback; responsive polish; dark theme; hybrid retrieval; document viewer |
+| 11–16 | answer + access | long answers; header consistency; viewer back-nav; localStorage persistence; steering notes; single-admin auth |
+| 17–28 | UX + sources | thinking display; follow-bottom scroll (later revised, 42); shared header; mid-stream sources fix; thinking scroll fix; background animation; global tuning; git-based sources |
+| 29–36 | sources + KB | tuning nav link; document summaries; KB overview prompt; admin sync button; cache busting; git-sources admin page; summary in viewer |
+| 37–41 | agent tools | document tools `ls`/`read`; local directory sources; sync fail-fast model check; config/brand surfaces |
+| 42–49 | chat behavior | no-reply autoscroll; thinking scroll-back; markdown tables; unlimited tool rounds; quadlet/j2 import; stop generation; archive uploads; retry answer |
+| 50–58 | persistence + scale | saved chats; share links; payload limits; stale saved chats; cache-bust revalidation; save/share UX; any-extension import; summary editing; wide desktop column |
+| 59–66 | docs push + polish | save-as-doc git push; sticky navbar; UI copy; ui customization env; unambiguous doc listing; sync/upload progress; bottom chat actions; history copy |
+| 67–75 | reliability + tools | LLM retries; `grep` search tool; source removal cleanup; harness-aligned tools; scaffolding guardrails; teaching refusals; hidden-tab streaming; LLM chat history; full-session save |
+| 76–84 | shell + security | SPA nav shell; navbar refresh; static background; API access tokens; history-driven suggestion chips; login rate limit; security headers; save payload limits; docs-push error sanitization |
+| 85–92 | mobile + theming | mobile menu gate fix; history page width; big-read progress; mobile hamburger boot; per-source ignore paths; upload-without-scan; admin Theme tab; theme save + coverage |
+
+(Phase numbers 23/24/26/29–36/39/46/52 shipped in the pre-`complete/`
+workflow era — their records live in `.agents/reports/`; every one of
+their features is in the running code and covered by a story E2E suite.)
+
+### Next up (from `TODO.md` — to be captured as phases via the
+`phase-authoring` skill, one story/one phase/one E2E each)
+1. **Tree-style `ls` with folder summaries** — `ls` behaves like a
+   filesystem tree (top level = the synced projects, drill-down per
+   folder) with per-folder summaries generated at sync time, so listing
+   thousands of documents never floods the context. Run retrieval
+   accuracy + performance checks against the current architecture.
+2. **Capped `read` for huge documents** — a sensible per-read cap sized
+   for the 128k-token minimum context of the owner's LLMs; the model is
+   told the read was truncated and steered to `grep` for the rest; a
+   visual indicator in the chat tells the user the read was truncated.
+
+### Completion (per phase — unchanged)
+unit + integration green, coverage >90% on `app/`, the story's Playwright
+E2E green **in isolation**, UI verification against §7 passed, **one
+`--no-gpg-sign` commit**, phase dir moved to `complete/`.
+
+---
+
+## 13. Future (post-v1 hooks, deliberately not built)
+- HNSW index on `chunks.embedding` at scale (sequential scan is fine
+  today).
+- Watchdog auto-re-import (inotify) — until then the script / Sync button
+  is the truth.
+- More LLM providers — the OpenAI-compatible client is the seam
+  (`BOR_LLM_BASE_URL`).
+- Multi-user accounts (tokens are the stop-gap; the `require_user`
+  split is the seam).
+- Vector-index-backed `grep` (trigram/GIN) if substring search gets slow
+  on a large KB.