Files
brain-of-reese/.agents/PLAN.md
T
ducoterra 17dd3bfac1
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 16s
docs(plan): remove hardcoded phase/migration references to prevent staleness
Refactor §12 roadmap to point to ls commands and 00_phase.md files
instead of listing specific phase numbers and descriptions.

Remove ~20 phase-number references from anchors, revision notes,
tool surface, feedback, and sync sections — keep only the one
stable reference (phase 03 convention).

Replace migration ranges with 'ls alembic/versions/'.
Replace hardcoded retry values with 'configurable (defaults: ...)'.
Status header now points to §12 instead of listing counts.
2026-09-12 10:22:45 -04:00

364 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Brain of Reese — Master Plan (minimal)
> **Status:** Minimal re-land (2026-09-10). See §12 for current state.
> Section numbers and anchor IDs match the
> previous full plan (in git history, commit `dac4a3e`) so the codebase's
> `PLAN §…` / anchor comments stay valid — consult that revision or the
> phase records in `.agents/phases/complete/` for full detail and the
> complete owner-permission revision log.
>
> **Rule:** Every agent reads this file first. `LOCKED` anchors in §2 are
> settled — a change requires explicit owner permission, recorded as a
> dated revision note under the table. Never a silent deviation
> (AGENTS.md rule 3).
>
> **This is deliberately minimal:** it captures the tech stack, the
> architecture an agent needs to work, and the locked decisions. It does
> NOT recap the shipped work — the git log, the README, and the completed
> phase directories are the history.
---
## 1. Mission
A **knowledge-base chatbot** ("Brain of Reese") over the owner's homelab
documentation — git repos, local directories, and uploaded archives
registered on the admin Sources page — embedded into Postgres + pgvector.
Product feel: chippy, upbeat, and **radically honest** — if retrieval
surfaces nothing relevant it says *"I haven't done anything like that"*
and offers alternatives instead of hallucinating. Self-hosted LLMs only.
**Deliberately out of scope:** binary/non-text ingestion; real-time file
watching (the import script / Sync button is the refresh loop); PR
tooling for docs push (the owner opens the PR); multi-admin/per-user
accounts (one admin + hand-out tokens is the model).
---
## 2. Tech Stack & Architectural Anchors (LOCKED)
| # | Component | Decision |
|---|-----------|----------|
| A1 | Runtime | Python 3.12+, `uv` for all package management (API + scripts + tests, one venv) |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) |
| A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors |
| A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) |
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) |
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) |
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) |
| A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) |
| A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) |
| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error`, `tool_result`; no proxy buffering, no client caching on the stream |
| A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase |
| A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset |
| A19 | Deploys & caching | HTML pages `Cache-Control: no-cache` (no etag/last-modified); `/assets/*` versioned (`?v=<token>`) + `immutable, max-age=1y`; **the token is the deploy** (git `HEAD` short SHA; content-hash fallback) — a deploy is a commit the browser sees without a hard refresh |
| A20 | Security headers | Every response: `Content-Security-Policy: default-src 'self'` (+ the theme's inline-`<style>` hash on themed pages), `frame-ancestors 'none'`, `X-Content-Type-Options: nosniff`; login rate-limited 10 attempts / 15 min (in-memory, A12) |
**UI/theming anchors (B):**
| # | Decision |
|---|----------|
| B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) |
| B3 | The semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted |
| B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only |
| B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") |
**Key recent revisions (full log in the previous plan revision / phase
records):** A10 — single-admin cookie auth, saved chats + shares +
staleness, client history budgets, access tokens.
A15 — `thinking`, `tool`, unlimited tool rounds under
`BOR_AGENT_MAX_ROUNDS`, `retry`, harness-aligned
`ls`/`read`/`grep` surface. A7 — the **retrieval** path's
never-truncated contract is unchanged; the `read`-tool cap is
the only exception, owner-permitted 2026-09-10.
---
## 3. High-Level Architecture
```
Browser (SPA shell, no CDN)
│ HTTP pages · SSE chat · JSON API
▼
FastAPI app (app/main.py)
├── middleware: SessionMiddleware → caching (A19) → SecurityHeaders (A20)
├── /api/* chat, docs, git-sources, steering, sync, chats,
│ doc-drafts, tokens, ui-settings, auth, health, suggestions
├── RAG pipeline (app/rag/) embed → retrieve → gate → agent → turbo
└── StaticFiles (frontend/) — shell routes registered before the mount
│ SQL (psycopg) OpenAI-compat
▼ ▼ ▼
Postgres 17 + pgvector aipi.reeseapps.com/v1
(db service, podman compose) (self-hosted: turbo / embed / lite)
Sources registry (Postgres `git_sources`, admin-managed; DB rows win,
BOR_GIT_SOURCES is the empty-table git-only fallback):
git repos → clone/pull via scripts/git_sync.py (the ONLY git-invocation
site; stdlib subprocess) local dirs → walked directly
archives → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)
```
| Component | Lives in |
|-----------|----------|
| App + middlewares + auth/tokens/rate-limit/theming/caching | `app/main.py`, `app/core/` |
| RAG (retriever, prompts, agent, llm, chunker, importer, summarizer, overview, scaffolding, suggestions, …) | `app/rag/` |
| API routers | `app/api/` |
| Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` |
| Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) |
| Migrations (all reversible) | `alembic/versions/` |
---
## 4. Chat Turn & SSE Contract (§3/§4 in older comments)
```
POST /api/chat {message, history?} (auth: require_user)
→ embed(question) [retried, A15 ext.]
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
│ + <tools> → agent loop (ls/read/grep; round-capped
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
byte-identical direct-stream path) → turbo streamed
→ query_log row + per-turn log line (§9)
```
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
still counted). LLM retries (A15 extension): retry count and delay are
configurable (defaults: 3 retries, 5 s flat delay), **only before a
request has streamed its first output frame**.
Retrieval details (A7/A8) and the persona/`<tools>` prompt contract: see
the previous plan revision §6 or `app/rag/retriever.py` /
`app/rag/prompts.py` / `app/rag/agent.py` — the module docstrings carry
the full contracts. **Persona text changes through the plan, not in code
(phase 03 convention); the `DEFLECT_MODE` marker and the mock-LLM markers
(`SUMMARY_MODE`, `KB_OVERVIEW_MODE`, …) may not change without updating
`tests/e2e/mock_llm.py`.**
---
## 5. Data Model & Chunking (§5 in older comments)
Tables (full column detail: `app/models.py` — it is the living doc;
each migration has a tested downgrade):
| Table | Purpose |
|-------|---------|
| `documents` | one row per imported file — full content, `(source, path)` unique, sha256 `content_hash`, `lite` `summary` (non-markdown) |
| `chunks` | retrieval units; `embedding VECTOR(768)`; `position −1` = the embedded summary chunk; stored `tsvector` (GIN) for FTS |
| `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) |
| `steering_notes` | owner tuning notes → `<tuning>` section of every turn (char-budgeted) |
| `kb_overview` | single row `id=1`: `lite`-generated KB outline → `<knowledge_base>` section (regenerated on KB change) |
| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB |
| `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/<token>`); `sources_version` (stale when < current) |
| `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync |
| `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) |
| `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead |
| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 17 color variables; NULL = default, B1) |
Single-row tables use `id = 1` (the `kb_overview` precedent).
**Chunking** (format-aware, `app/rag/chunker.py`): markdown on
`##`/`###` headings (paragraph sub-split > `BOR_CHUNK_TARGET_CHARS`=2000,
200 overlap); YAML/JSON on top-level keys; Python on top-level defs via
`ast`; txt on paragraphs; quadlet/j2 plain text. Every format honors the
**1200-char hard cap** (aipi ~1024-token request limit).
**Import workflow** (script and UI Sync share the importer): sha256
delta (unchanged files skip), two-phase upsert (one transaction per
file), `--prune` drops deleted/ignored files, non-markdown files get
`lite` summaries (best-effort/fail-soft), and a KB-changing run
regenerates the `kb_overview` + bumps `sources_meta.version` exactly
once. A failed source aborts the run — no partial junk.
---
## 6. Retrieval & Persona
(See §4's flow and the anchor rows A7/A8. The locked persona, section
order — `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body —
and the `<tools>` teaching live in `app/rag/prompts.py`; the agent loop,
teaching refusals, and scaffolding filter in `app/rag/agent.py` /
`app/rag/scaffolding.py`. Empty prompt sections omit themselves — a
no-notes/no-overview prompt is byte-identical to the pre-steering text.)
Tool surface (harness-aligned): **`ls`** (lists indexed docs
`source: X | path: Y | title: Z`; `path` = a source name — drill-down
tree); **`read`** (combined `source/path` including the source name;
appends the full document — with a truncation cap); **`grep`** (case-
insensitive **fixed substring**, ≤20 `source/path:line: text` matches,
optional one-doc scope; a locator that adds no source/context — never a
regex).
Rejected calls get deterministic teaching refusals and consume a round;
`holder.tool_calls` counts executed calls only.
---
## 7. UI/UX Standards
- **Layout (§7.1):** sticky 64px header + `<main>` + footer; container
`max-width: 72rem` centered; chat is a **centered 46rem column** (2×
= 92rem at ≥1500px desktops — deliberate, not a bug); Sources uses a
**full-width responsive table** — no skinny single-column lists
(lists/tables/grids ≥80–90% of container width); document viewer is a
near-fullscreen same-page modal; mobile ≤640px: hamburger nav,
≥44px touch targets, safe-area composer.
- **Accessibility (§7.2, WCAG 2.1 AA):** semantic landmarks on every
view, labeled controls (icon buttons get `aria-label`),
`aria-live="polite"` stream, `role="status"`/`role="alert"`,
3px `:focus-visible`, `prefers-reduced-motion` respected, text pairs
≥4.5:1 (state is **text + color, never color alone**).
- **Theming (B1–B5):** built-in dark palette in
`app/core/theming.py::BUILTIN_COLORS` (the `:root`-drift unit test
parses built-ins from `styles.css` — no second copy); the admin Theme
tab persists the palette in `ui_settings`, injected pre-paint as
`<style id="bor-theme">` with a matching CSP hash (A20); **B4
byte-identical** when nothing is set.
- **No CDN (§7.3):** zero external `<script>`/`<link>` (integration test
on the index page); markdown rendering is a small local
escape-first function; esbuild minify is build-time only.
- **Never-stale feedback (§7.4):** every control state — idle /
thinking / calling tool / streaming / retrying / done (answer or
deflected) / error / KB-offline / stopped — has a defined UI, every
failure path re-enables its controls, a 300 s pre-token guard
(`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts
only visible time), and **no auto-follow during a turn**
(viewport moves only on user intent). Pinned by unit tests
on the `app.js` state machine + the story E2E suites.
- Frontend house rules: `app.js` and siblings **never build HTML
strings** (createElement + textContent); asset paths carry
`?v=<token>` (A19).
---
## 8. Debugging
`DEBUGPY` unset/`0` → `debugpy` never imported (unit-tested, A14).
`DEBUGPY=1` → non-blocking listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`;
IDE attaches on demand. Wired at `app/main.py` module import
(`app/core/debugging.py`), so `uvicorn app.main:app`, `python -m
scripts.…`, and tests all honor it.
---
## 9. Observability
- Logs: single-line `timestamp LEVEL logger :: message` on stdout, INFO
default (`BOR_LOG_LEVEL`), third-party loggers capped at WARNING.
- **Per-chat-turn line (required — AGENTS.md rule 10):**
```
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
thinking_chars=… tool_calls=N total_ms=… retries=N scaffold_stripped=N
```
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
executed only; a cancelled turn writes neither line nor `query_log`
row.)
- Importer: per-file `added|updated|unchanged|pruned` + a greppable
cron-safe summary line (`import: summary files=… added=… … formats=…`).
- `query_log` is the durable tuning/gap-finding record.
---
## 10. Testing & Quality Gates (A16 — non-negotiable)
| Gate | Command |
|------|---------|
| Unit + integration | `uv run pytest` |
| Coverage **>90%** on `app/` | `uv run pytest --cov=app --cov-report=term-missing` |
| Story E2E, **in isolation** | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` (DB up: `podman compose up -d db`) |
| Lint + types | `uv run ruff check . && uv run pyright` |
E2E determinism: `tests/e2e/mock_llm.py` is a deterministic
OpenAI-compatible mock (genuine L2-normalized token-overlap embeddings,
so the cosine gate behaves like production); `tests/e2e/slow_llm.py` is
the slow/dead variant; `E2E_REAL_LLM=1` opts into live aipi. Story E2E
fixtures truncate/re-import per module — suites must run in isolation.
Real-model tool-calling verification uses the controlled methodology in
`TOOL_CALLING_TESTING.md` + `scripts/agent_realmodel_check.py`
(fixture KB: `tests/fixtures/test_kb.dump.sql`; the 1,000-doc live-replica
snapshot restores via the `restore-test-db` skill).
---
## 11. Import & Update Workflow
```bash
uv run python -m scripts.import_docs # sync sources + index delta
uv run python -m scripts.import_docs --prune # also drop deleted/ignored
uv run python -m scripts.import_docs --source ~/X # extra directories
# or one click: the admin Sources page "Sync sources" (in-process, 409 while running)
uv run python -m scripts.eval_retrieval "<question>" # rank hybrid results (tuning)
uv run python -m scripts.llm_probe # models + embedding dim sanity
```
The loop for git sources is *commit → re-run*. Uploads:
`POST /api/git-sources/upload` unpacks + registers (202); the scan is
deferred to Sync. Source removal prunes on the next sync.
---
## 12. Current State & Roadmap
- **Shipped phases:** `ls .agents/phases/complete/` — read-only
history; each phase's `00_phase.md` has the full detail. The shipped-
features recap of the previous plan revision and the README cover it.
- **Migrations:** `ls alembic/versions/` — all reversible.
- **Next up (`todo/`):** Already authored via `phase-authoring`.
Read the directory listing in `.agents/phases/todo/` for the ordered
task list; each file's `00_phase.md` has the full description.
- **Next free phase number:** `max(completed phase numbers) + 1` —
computed from `.agents/phases/complete/`.
- **Post-v1 hooks (deliberately not built):** HNSW index at scale;
inotify auto-import; more providers (the OpenAI-compatible client is
the seam); multi-user accounts (the `require_user` split is the seam);
index-backed `grep`.
**Per-phase completion** (AGENTS.md rules 8/9): unit + integration
green, coverage >90%, the phase's Playwright E2E green in isolation, UI
verified against §7, one atomic `--no-gpg-sign` Conventional Commit,
phase dir moved to `complete/`.
---
## 13. House Conventions (quick catch-up)
- **Env:** every setting is a `BOR_`-prefixed env var (or gitignored
`.env`; see `.env.example`); `get_settings()` is `lru_cache`d.
Kill switches follow the `agent_max_rounds` pattern: `0` disables the
feature, a negative value fails startup **loudly** (validator names
the field).
- **Fail loud, never half-configured:** missing admin secrets, bad
extension lists, dim mismatches, unresolvable docs branches — all
refuse to start or refuse the request with a named reason.
- **Shared marker:** `TRUNCATION_MARKER = "[…truncated…]"`
(`app/rag/retriever.py`) is the only overflow marker; char budgets are
the pattern for anything sent to `lite`.
- **Byte-identical contracts are load-bearing:** deflected turns
(A8), empty prompt sections, the B4 theme no-op, and the
rewrite-only caching all have tests that assert byte-identity.
- **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib
subprocess); everything else talks Postgres.
- **One story → one phase → one dedicated Playwright file** (AGENTS.md
rules 4/9); `.agents/` is tracked; only `.agents/phase-sessions/` and
`.agents/pipeline.log` are gitignored.
- **Full-history references:** the previous full plan (all revision
notes, complete API table, column-level data model) is in git —
`git show dac4a3e:.agents/PLAN.md`; per-phase decisions are in
`.agents/phases/complete/*/00_phase.md`; owner test methodologies in
`TOOL_CALLING_TESTING.md`; skills for model testing/KB restore in
`.agents/skills/`.