brain-of-reese/.agents/PLAN.md

Brain of Reese — Master Plan (minimal)

Status: Minimal re-land (2026-09-10). See §12 for current state. Section numbers and anchor IDs match the previous full plan (in git history, commit dac4a3e) so the codebase's PLAN §… / anchor comments stay valid — consult that revision or the phase records in .agents/phases/complete/ for full detail and the complete owner-permission revision log.

Rule: Every agent reads this file first. LOCKED anchors in §2 are settled — a change requires explicit owner permission, recorded as a dated revision note under the table. Never a silent deviation (AGENTS.md rule 3).

This is deliberately minimal: it captures the tech stack, the architecture an agent needs to work, and the locked decisions. It does NOT recap the shipped work — the git log, the README, and the completed phase directories are the history.

---

1. Mission

A knowledge-base chatbot ("Brain of Reese") over the owner's homelab documentation — git repos, local directories, and uploaded archives registered on the admin Sources page — embedded into Postgres + pgvector. Product feel: chippy, upbeat, and radically honest — if retrieval surfaces nothing relevant it says "I haven't done anything like that" and offers alternatives instead of hallucinating. Self-hosted LLMs only.

Deliberately out of scope: binary/non-text ingestion; real-time file watching (the import script / Sync button is the refresh loop); PR tooling for docs push (the owner opens the PR); multi-admin/per-user accounts (one admin + hand-out tokens is the model).

---

2. Tech Stack & Architectural Anchors (LOCKED)

#	Component	Decision
A1	Runtime	Python 3.12+, uv for all package management (API + scripts + tests, one venv)
A2	Web framework	FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly)
A3	Database	PostgreSQL 17 + pgvector (local build of official postgres:17, db/Containerfile), cosine (<=>) search; one system for relational + vectors
A4	Orchestration	compose.yaml, podman compose up -d (dev: db only; --profile prod adds the app container)
A5	LLM backend	OpenAI-compatible self-hosted endpoint https://aipi.reeseapps.com/v1 via the openai async client: turbo (chat, streams reasoning_content thinking), embed (embeddings), lite (one-shot: document summaries, KB overview, folder summaries)
A6	Embedding dim	768 (verified against the live endpoint); chunks.embedding is fixed at table creation — a dim mismatch must fail loudly, never silently re-embed
A7	Retrieval→context	Hybrid: cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, ts_rank), fused with RRF (k=60) → parent docs ranked by best fused chunk → full text of top-N=2 documents, never truncated on the retrieval path (owner 2026-08-24: "this should never happen"; the agent read-tool cap is a separate owner-permitted path)
A8	Honesty gate	Deflect (LOW mode) only when best cosine < BOR_RELEVANCE_THRESHOLD (0.62) and (zero FTS hits or best cosine < BOR_LEXICAL_SUPPORT_FLOOR (0.35)); an FTS hit flips HIGH only when best_cosine >= lexical_support_floor — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, owner-confirmed, TODO L2a: lexical-only hits without vector support deflect). LOW prompt carries weak-hit titles only + the DEFLECT_MODE marker (the E2E mock keys on its presence) + the plain-text no-tools line
A9	Content scope	Default md, markdown, txt, yaml, yml, json, py + Podman quadlet family + j2; BOR_IMPORT_EXTENSIONS may name any well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source ignore_paths (raw prefixes, no globs) always apply
A10	State & auth	POST /api/chat is stateless (client-provided history only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed bor_session cookie (Starlette SessionMiddleware + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; only shared chats stay anonymous. Fail-loud at boot while BOR_ADMIN_PASSWORD/BOR_SESSION_SECRET are empty
A11	Frontend	Vanilla HTML/CSS/JS in git, no CDN — everything served by FastAPI StaticFiles; system font stack; the navbar views are views of ONE shell document (frontend/index.html + router.js deep-links)
A12	Aux services	None (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted
A13	Migrations	Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; every migration ships a tested downgrade (A13 — reversible)
A14	Debugging	debugpy imported only when DEBUGPY=1 (app/core/debugging.py); never imported otherwise (unit-tested)
A15	Chat transport	SSE from POST /api/chat; event types thinking, delta, tool, retry, done, error, tool_result; no proxy buffering, no client caching on the stream
A16	Testing	Per phase: unit + integration (pytest) with >90% coverage on app/ + one dedicated Playwright E2E file, run in isolation (--no-cov); E2E uses a deterministic mock LLM by default (E2E_REAL_LLM=1 opts into live aipi)
A17	Git	Conventional Commits, always --no-gpg-sign, one atomic commit per completed phase
A18	Docs push	Save-a-answer-as-doc: server-side draft (doc_drafts, long body never in a URL — unguessable uuid4 token is the URL credential) pushed to a generic git remote (BOR_DOCS_REPO, URL or local path; no gh) on a dedicated branch, --ff-only, re-cut per push; no PR tooling; inert (hidden + 409) while the repo is unset
A19	Deploys & caching	HTML pages Cache-Control: no-cache (no etag/last-modified); /assets/* versioned (?v=<token>) + immutable, max-age=1y; the token is the deploy (git HEAD short SHA; content-hash fallback) — a deploy is a commit the browser sees without a hard refresh
A20	Security headers	Every response: Content-Security-Policy: default-src 'self' (+ the theme's inline-<style> hash on themed pages), frame-ancestors 'none', X-Content-Type-Options: nosniff; login rate-limited 10 attempts / 15 min (in-memory, A12)
A21	LLM call economics	ALL LLM calls are free (chat, vision, embed, lite — self-hosted aipi): the sync/import path may burn as many LLM/vision calls as quality requires — e.g. render + describe every sparse page of a pictorial PDF, no page caps, no call budgets (owner 2026-09-30). Latency targets bind the interactive path only (retrieval, inference, tool calling — the inference engine's job, not ours to micro-optimize): never cap, throttle, or skip work to save calls; a slower sync is acceptable, a lower-quality KB is not

UI/theming anchors (B):

#	Decision
B1	Theme ui_settings columns: NULL/empty = "use the default" — env value for the 3 strings, built-in palette for colors (colors have no env fallback)
B3	The semantic state families (--ok-*, --err-*, --accent-*) join the storable palette — 17 variables total, all NULL = built-in (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted
B4	Byte-identical contract: with no ui_settings row (or a fully default theme) the served HTML is byte-identical to the built-in default — no #bor-theme tag, caching rewrites are rewrite-only
B5	Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state text label ("text + color, never color alone")

Key recent revisions (full log in the previous plan revision / phase records): A21 — ALL LLM calls (chat, vision, embed, lite) are free for this project (owner 2026-09-30): import/sync may burn unlimited LLM/vision calls for quality (e.g. every page of a 500+ page pictorial PDF, rendered + described); sync duration is not an optimization target, interactive-path speed is the inference engine's job. A10 — single-admin cookie auth, saved chats + shares + staleness, client history budgets, access tokens. A15 — thinking, tool, unlimited tool rounds under BOR_AGENT_MAX_ROUNDS, retry, harness-aligned ls/read/grep surface. A7 — the retrieval path's never-truncated contract is unchanged; the read-tool cap is the only exception, owner-permitted 2026-09-10.

Phase 182 (verbatim text display, owner chat 2026-10-14) adds show_text(path, offset?, limit?) as the fifth agent tool — offered only when BOR_ANSWER_TEXT=1 (the default), gated exactly like show_image, with its own <tools> clause and its own kill switch — and the additive done.answer_text field above. The new settings are BOR_ANSWER_TEXT (master switch, ON), BOR_SHOW_TEXT_MAX_CHARS (per-block budget, 12 000) and BOR_SHOW_TEXT_MAX_BLOCKS (per-answer budget, 5), all following the agent_max_rounds pattern (0 inert, negative refuses startup). The principle in one sentence: the model points, the server supplies the bytes — a ```bor source/path 12-40 fence or a show_text call names a location (canonical locator source/path or source/path#start-end, 1-based and inclusive), and the displayed text is sliced from documents.content server-side, so what the user copies is the knowledge base, never a re-typed approximation. A1–A6 and A15 are otherwise UNCHANGED: no new SSE event type, no new table or column, no new model, endpoint or transport, no change to retrieval, the honesty gate, the persona text or the embedding dimension.

---

3. High-Level Architecture

Browser (SPA shell, no CDN)
  │  HTTP pages · SSE chat · JSON API
  ▼
FastAPI app (app/main.py)
  ├── middleware: SessionMiddleware → caching (A19) → SecurityHeaders (A20)
  ├── /api/*  chat, docs, git-sources, steering, sync, chats,
  │           doc-drafts, tokens, ui-settings, auth, health, suggestions
  ├── RAG pipeline (app/rag/)  embed → retrieve → gate → agent → turbo
  └── StaticFiles (frontend/)  — shell routes registered before the mount
  │            SQL (psycopg)                     OpenAI-compat
  ▼            ▼                                 ▼
Postgres 17 + pgvector              aipi.reeseapps.com/v1
(db service, podman compose)        (self-hosted: turbo / embed / lite)

Sources registry (Postgres `git_sources`, admin-managed; DB rows win,
BOR_GIT_SOURCES is the empty-table git-only fallback):
  git repos → clone/pull via scripts/git_sync.py (the ONLY git-invocation
              site; stdlib subprocess)   local dirs → walked directly
  archives  → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)

Component	Lives in
App + middlewares + auth/tokens/rate-limit/theming/caching	app/main.py, app/core/
RAG (retriever, prompts, agent, llm, chunker, importer, summarizer, overview, scaffolding, suggestions, …)	app/rag/
API routers	app/api/
Import / sync tooling	scripts/import_docs.py, app/api/sync.py, scripts/git_sync.py, scripts/eval_retrieval.py, scripts/llm_probe.py
Frontend (shell + standalone pages)	frontend/ (index.html shell; document.html, shared.html, login.html, doc-edit.html)
Migrations (all reversible)	alembic/versions/

---

4. Chat Turn & SSE Contract (§3/§4 in older comments)

POST /api/chat {message, history?}   (auth: require_user)
  → embed(question)                                   [retried, A15 ext.]
  → cosine top-100 ∪ FTS top-30 → RRF fuse (k=60)     [A7]
  ├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
  │    persona + <knowledge_base> + <tuning> + full top-2 <documents>
  │    + <tools> → agent loop (ls/read/grep, plus show_image when
  │    BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
  │    "show the user an artifact" tools, gated independently; round-capped
  │    BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
  └─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
       byte-identical direct-stream path) → turbo streamed
  → query_log row + per-turn log line (§9)

SSE frames (data: <json>\n\n): thinking (before first delta) → tool (grounded turns, one per model call) → retry (pre-first-frame restarts) → delta (answer tokens) → done {deflected, sources[], suggestions[], budget_exhausted?, answer_text[]?} (terminal). Phase 182 extends the done frame ADDITIVELY — answer_text is the list of server-resolved verbatim blocks (locator, source, path, start, end, total_lines, text, format, truncated) the answer displayed; there is no eighth event type, the key is OMITTED when no block was displayed, and with BOR_ANSWER_TEXT=0 it is absent on every turn (the frame is then byte-identical to pre-phase). The same array persists on the saved chat record, so restore and the anonymous shared page re-render the panels from the server's bytes. Failure: error (terminal — no done, no query_log row; a pre-stream DB outage is a plain 503 JSON). A tools=None request never emits a tool frame; a round-capped turn whose final reply carries no visible content answers with the budget fallback instead of erroring (phase 160, recorded revision of the phase-71 cap-path terminal, owner-confirmed 2026-10-05). BOR_STREAM_THINKING=0 suppresses thinking frames server-side (chars still counted). LLM retries (A15 extension): retry count and delay are configurable (defaults: 3 retries, 5 s flat delay), only before a request has streamed its first output frame.

Retrieval details (A7/A8) and the persona/<tools> prompt contract: see the previous plan revision §6 or app/rag/retriever.py / app/rag/prompts.py / app/rag/agent.py — the module docstrings carry the full contracts. Persona text changes through the plan, not in code (phase 03 convention); the DEFLECT_MODE marker and the mock-LLM markers (SUMMARY_MODE, KB_OVERVIEW_MODE, …) may not change without updating tests/e2e/mock_llm.py.

---

5. Data Model & Chunking (§5 in older comments)

Tables (full column detail: app/models.py — it is the living doc; each migration has a tested downgrade):

Table	Purpose
documents	one row per imported file — full content, (source, path) unique, sha256 content_hash, lite summary (non-markdown)
chunks	retrieval units; embedding VECTOR(768); position −1 = the embedded summary chunk; stored tsvector (GIN) for FTS
query_log	every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record)
steering_notes	owner tuning notes → <tuning> section of every turn (char-budgeted)
kb_overview	single row id=1: lite-generated KB outline → <knowledge_base> section (regenerated on KB change)
git_sources	source registry: kind git|local, url/path, ignore_paths JSONB
saved_chats	owner-saved conversations; messages = the raw bor.chat.v1 JSONB; share_token (NULL = private, uuid4 → /shared/<token>); sources_version (stale when < current)
sources_meta	single row id=1: the KB generation counter — bumped once per KB-changing sync
doc_drafts	save-as-doc drafts; token (uuid4) is the URL credential; draft → pushed (branch + sha)
api_tokens	access tokens; only the SHA-256 of the full bor_… string is stored; revoked_at = dead
ui_settings	single row id=1: Theme tab persistence (3 strings + 17 color variables; NULL = default, B1)

Single-row tables use id = 1 (the kb_overview precedent).

Chunking (format-aware, app/rag/chunker.py): markdown on ##/### headings (paragraph sub-split > BOR_CHUNK_TARGET_CHARS=2000, 200 overlap); YAML/JSON on top-level keys; Python on top-level defs via ast; txt on paragraphs; quadlet/j2 plain text. Every format honors the 1200-char hard cap (aipi ~1024-token request limit).

Import workflow (script and UI Sync share the importer): sha256 delta (unchanged files skip), two-phase upsert (one transaction per file), --prune drops deleted/ignored files, non-markdown files get lite summaries (best-effort/fail-soft), and a KB-changing run regenerates the kb_overview + bumps sources_meta.version exactly once. A failed source aborts the run — no partial junk.

---

6. Retrieval & Persona

(See §4's flow and the anchor rows A7/A8. The locked persona, section order — <relevance> → <knowledge_base> → <tuning> → mode body — and the <tools> teaching live in app/rag/prompts.py; the agent loop, teaching refusals, and scaffolding filter in app/rag/agent.py / app/rag/scaffolding.py. Empty prompt sections omit themselves — a no-notes/no-overview prompt is byte-identical to the pre-steering text.)

Tool surface (harness-aligned): ls (lists indexed docs source: X | path: Y | title: Z; path = a source name — drill-down tree); read (combined source/path including the source name; appends the full document — with a truncation cap); grep (case- insensitive fixed substring, ≤20 source/path:line: text matches, optional one-doc scope; a locator that adds no source/context — never a regex). Rejected calls get deterministic teaching refusals and consume a round; holder.tool_calls counts executed calls only.

---

7. UI/UX Standards

• Layout (§7.1): sticky 64px header + <main> + footer; container max-width: 72rem centered; chat is a centered 46rem column (2× = 92rem at ≥1500px desktops — deliberate, not a bug); Sources uses a full-width responsive table — no skinny single-column lists (lists/tables/grids ≥80–90% of container width); document viewer is a near-fullscreen same-page modal; mobile ≤640px: hamburger nav, ≥44px touch targets, safe-area composer.
• Accessibility (§7.2, WCAG 2.1 AA): semantic landmarks on every view, labeled controls (icon buttons get aria-label), aria-live="polite" stream, role="status"/role="alert", 3px :focus-visible, prefers-reduced-motion respected, text pairs ≥4.5:1 (state is text + color, never color alone).
• Theming (B1–B5): built-in dark palette in app/core/theming.py::BUILTIN_COLORS (the :root-drift unit test parses built-ins from styles.css — no second copy); the admin Theme tab persists the palette in ui_settings, injected pre-paint as <style id="bor-theme"> with a matching CSP hash (A20); B4 byte-identical when nothing is set.
• No CDN (§7.3): zero external <script>/<link> (integration test on the index page); markdown rendering is a small local escape-first function; esbuild minify is build-time only.
• Never-stale feedback (§7.4): every control state — idle / thinking / calling tool / streaming / retrying / done (answer or deflected) / error / KB-offline / stopped — has a defined UI, every failure path re-enables its controls, a 300 s pre-token guard (TURN_TIMEOUT_MS) turns a hung stream into the error state (counts only visible time), and no auto-follow during a turn (viewport moves only on user intent). Pinned by unit tests on the app.js state machine + the story E2E suites.
• Frontend house rules: app.js and siblings never build HTML strings (createElement + textContent); asset paths carry ?v=<token> (A19).

---

8. Debugging

DEBUGPY unset/0 → debugpy never imported (unit-tested, A14). DEBUGPY=1 → non-blocking listener on 0.0.0.0:${DEBUGPY_PORT:-5678}; IDE attaches on demand. Wired at app/main.py module import (app/core/debugging.py), so uvicorn app.main:app, python -m scripts.…, and tests all honor it.

---

9. Observability

• Logs: single-line timestamp LEVEL logger :: message on stdout, INFO default (BOR_LOG_LEVEL), third-party loggers capped at WARNING.
• Per-chat-turn line (required — AGENTS.md rule 10):

  question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
  kb_chars=N history_msgs=N threshold=… deflected=… sources=…
  thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
  retries=N scaffold_stripped=N verbatim_blocks=N
  

  (verbatim_blocks= — phase 182 — is the count of verbatim display blocks actually SENT on the done frame, 0 on every turn without them, deflected included.) (sources= = retrieval + agent-read docs, deduped; tool_calls= executed only; a cancelled turn writes neither line nor query_log row.)
• Importer: per-file added|updated|unchanged|pruned + a greppable cron-safe summary line (import: summary files=… added=… … formats=…).
• query_log is the durable tuning/gap-finding record.

---

10. Testing & Quality Gates (A16 — non-negotiable)

Gate	Command
Unit + integration	uv run pytest
Coverage >90% on app/	uv run pytest --cov=app --cov-report=term-missing
Story E2E, in isolation	uv run pytest tests/e2e/test_<story>.py -v --no-cov (DB up: podman compose up -d db)
Lint + types	uv run ruff check . && uv run pyright

E2E determinism: tests/e2e/mock_llm.py is a deterministic OpenAI-compatible mock (genuine L2-normalized token-overlap embeddings, so the cosine gate behaves like production); tests/e2e/slow_llm.py is the slow/dead variant; E2E_REAL_LLM=1 opts into live aipi. Story E2E fixtures truncate/re-import per module — suites must run in isolation. Real-model tool-calling verification uses the controlled methodology in TOOL_CALLING_TESTING.md + scripts/agent_realmodel_check.py (fixture KB: tests/fixtures/test_kb.dump.sql; the 1,000-doc live-replica snapshot restores via the restore-test-db skill).

---

11. Import & Update Workflow

uv run python -m scripts.import_docs                # sync sources + index delta
uv run python -m scripts.import_docs --prune        # also drop deleted/ignored
uv run python -m scripts.import_docs --source ~/X   # extra directories
# or one click: the admin Sources page "Sync sources" (in-process, 409 while running)
uv run python -m scripts.eval_retrieval "<question>"   # rank hybrid results (tuning)
uv run python -m scripts.llm_probe                     # models + embedding dim sanity

The loop for git sources is commit → re-run. Uploads: POST /api/git-sources/upload unpacks + registers (202); the scan is deferred to Sync. Source removal prunes on the next sync.

---

12. Current State & Roadmap

• Shipped phases: ls .agents/phases/complete/ — read-only history; each phase's 00_phase.md has the full detail. The shipped- features recap of the previous plan revision and the README cover it.
• Migrations: ls alembic/versions/ — all reversible.
• Next up (todo/): Already authored via phase-authoring. Read the directory listing in .agents/phases/todo/ for the ordered task list; each file's 00_phase.md has the full description.
• Next free phase number: max(completed phase numbers) + 1 — computed from .agents/phases/complete/.
• Post-v1 hooks (deliberately not built): HNSW index at scale; inotify auto-import; more providers (the OpenAI-compatible client is the seam); multi-user accounts (the require_user split is the seam); index-backed grep.

Per-phase completion (AGENTS.md rules 8/9): unit + integration green, coverage >90%, the phase's Playwright E2E green in isolation, UI verified against §7, one atomic --no-gpg-sign Conventional Commit, phase dir moved to complete/.

---

13. House Conventions (quick catch-up)

• Env: every setting is a BOR_-prefixed env var (or gitignored .env; see .env.example); get_settings() is lru_cached. Kill switches follow the agent_max_rounds pattern: 0 disables the feature, a negative value fails startup loudly (validator names the field).
• Fail loud, never half-configured: missing admin secrets, bad extension lists, dim mismatches, unresolvable docs branches — all refuse to start or refuse the request with a named reason.
• Shared marker: TRUNCATION_MARKER = "[…truncated…]" (app/rag/retriever.py) is the only overflow marker; char budgets are the pattern for anything sent to lite.
• Byte-identical contracts are load-bearing: deflected turns (A8), empty prompt sections, the B4 theme no-op, and the rewrite-only caching all have tests that assert byte-identity.
• git is invoked only in scripts/git_sync.py (A11 — stdlib subprocess); everything else talks Postgres.
• One story → one phase → one dedicated Playwright file (AGENTS.md rules 4/9); .agents/ is tracked; only .agents/phase-sessions/ and .agents/pipeline.log are gitignored.
• Full-history references: the previous full plan (all revision notes, complete API table, column-level data model) is in git — git show dac4a3e:.agents/PLAN.md; per-phase decisions are in .agents/phases/complete/*/00_phase.md; owner test methodologies in TOOL_CALLING_TESTING.md; skills for model testing/KB restore in .agents/skills/.