diff --git a/.agents/PLAN.md b/.agents/PLAN.md index 18da1fd..21757e3 100644 --- a/.agents/PLAN.md +++ b/.agents/PLAN.md @@ -1,7 +1,7 @@ # Brain of Reese — Master Plan (minimal) -> **Status:** Minimal re-land (2026-09-10). Phases 01–92 shipped; -> `todo/` holds 93–95 (see §12). Section numbers and anchor IDs match the +> **Status:** Minimal re-land (2026-09-10). See §12 for current state. +> Section numbers and anchor IDs match the > previous full plan (in git history, commit `dac4a3e`) so the codebase's > `PLAN §…` / anchor comments stay valid — consult that revision or the > phase records in `.agents/phases/complete/` for full detail and the @@ -43,17 +43,17 @@ accounts (one admin + hand-out tokens is the model). | A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) | | A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors | | A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) | -| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview; phase 94 adds sync-time folder summaries) | +| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) | | A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed | -| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path — phase 95) | +| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) | | A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line | | A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply | | A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty | -| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links, phase 76) | +| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) | | A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted | | A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) | | A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) | -| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error` (+ `tool_result`, owner-permitted extension — phase 95, not yet landed); no proxy buffering, no client caching on the stream | +| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error`, `tool_result`; no proxy buffering, no client caching on the stream | | A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | | A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase | | A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset | @@ -65,17 +65,17 @@ accounts (one admin + hand-out tokens is the model). | # | Decision | |---|----------| | B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) | -| B3 | **Revised (owner 2026-09-10, phase 93):** the semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted | +| B3 | The semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted | | B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only | | B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") | **Key recent revisions (full log in the previous plan revision / phase -records):** A10 — single-admin cookie auth (16), saved chats + shares + -staleness (50/51/53), client history budgets (74), access tokens (79). -A15 — `thinking` (17), `tool` (37), unlimited tool rounds under -`BOR_AGENT_MAX_ROUNDS` (45), `retry` (67), harness-aligned -`ls`/`read`/`grep` surface (70). A7 — the **retrieval** path's -never-truncated contract is unchanged; the phase-95 `read`-tool cap is +records):** A10 — single-admin cookie auth, saved chats + shares + +staleness, client history budgets, access tokens. +A15 — `thinking`, `tool`, unlimited tool rounds under +`BOR_AGENT_MAX_ROUNDS`, `retry`, harness-aligned +`ls`/`read`/`grep` surface. A7 — the **retrieval** path's +never-truncated contract is unchanged; the `read`-tool cap is the only exception, owner-permitted 2026-09-10. --- @@ -111,7 +111,7 @@ BOR_GIT_SOURCES is the empty-table git-only fallback): | API routers | `app/api/` | | Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` | | Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) | -| Migrations (0001–0015, all reversible) | `alembic/versions/` | +| Migrations (all reversible) | `alembic/versions/` | --- @@ -119,7 +119,7 @@ BOR_GIT_SOURCES is the empty-table git-only fallback): ``` POST /api/chat {message, history?} (auth: require_user) - → embed(question) [retried, phase 67 / A15 ext.] + → embed(question) [retried, A15 ext.] → cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7] ├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0): │ persona + + + full top-2 @@ -136,9 +136,9 @@ restarts) → `delta` (answer tokens) → `done` `{deflected, sources[], suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no `query_log` row; a pre-stream DB outage is a plain 503 JSON). `BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars -still counted). LLM retries (phase 67, A15 extension): `BOR_LLM_RETRIES`=3 × flat -`BOR_LLM_RETRY_DELAY`=5 s, **only before a request has streamed its -first output frame**. +still counted). LLM retries (A15 extension): retry count and delay are +configurable (defaults: 3 retries, 5 s flat delay), **only before a +request has streamed its first output frame**. Retrieval details (A7/A8) and the persona/`` prompt contract: see the previous plan revision §6 or `app/rag/retriever.py` / @@ -153,7 +153,7 @@ the full contracts. **Persona text changes through the plan, not in code ## 5. Data Model & Chunking (§5 in older comments) Tables (full column detail: `app/models.py` — it is the living doc; -migrations 0001–0015, each with a tested downgrade): +each migration has a tested downgrade): | Table | Purpose | |-------|---------| @@ -162,12 +162,12 @@ migrations 0001–0015, each with a tested downgrade): | `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) | | `steering_notes` | owner tuning notes → `` section of every turn (char-budgeted) | | `kb_overview` | single row `id=1`: `lite`-generated KB outline → `` section (regenerated on KB change) | -| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB (phase 89) | +| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB | | `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/`); `sources_version` (stale when < current) | | `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync | | `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) | | `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead | -| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 9 identity colors; NULL = default, B1 — grows to 17 vars in phase 93) | +| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 17 color variables; NULL = default, B1) | Single-row tables use `id = 1` (the `kb_overview` precedent). @@ -195,13 +195,13 @@ teaching refusals, and scaffolding filter in `app/rag/agent.py` / `app/rag/scaffolding.py`. Empty prompt sections omit themselves — a no-notes/no-overview prompt is byte-identical to the pre-steering text.) -Tool surface (harness-aligned, phase 70): **`ls`** (lists indexed docs -`source: X | path: Y | title: Z`; `path` = a source name — becomes a -drill-down tree in phase 94), **`read`** (combined `source/path` -including the source name; appends the full document — gets a -truncation cap in phase 95), **`grep`** (case-insensitive **fixed -substring**, ≤20 `source/path:line: text` matches, optional one-doc -scope; a locator that adds no source/context — never a regex). +Tool surface (harness-aligned): **`ls`** (lists indexed docs +`source: X | path: Y | title: Z`; `path` = a source name — drill-down +tree); **`read`** (combined `source/path` including the source name; +appends the full document — with a truncation cap); **`grep`** (case- +insensitive **fixed substring**, ≤20 `source/path:line: text` matches, +optional one-doc scope; a locator that adds no source/context — never a +regex). Rejected calls get deterministic teaching refusals and consume a round; `holder.tool_calls` counts executed calls only. @@ -235,8 +235,8 @@ Rejected calls get deterministic teaching refusals and consume a round; deflected) / error / KB-offline / stopped — has a defined UI, every failure path re-enables its controls, a 300 s pre-token guard (`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts - only visible time — phase 73), and **no auto-follow during a turn** - (viewport moves only on user intent, phase 42). Pinned by unit tests + only visible time), and **no auto-follow during a turn** + (viewport moves only on user intent). Pinned by unit tests on the `app.js` state machine + the story E2E suites. - Frontend house rules: `app.js` and siblings **never build HTML strings** (createElement + textContent); asset paths carry @@ -307,30 +307,21 @@ uv run python -m scripts.llm_probe # models + embedding dim The loop for git sources is *commit → re-run*. Uploads: `POST /api/git-sources/upload` unpacks + registers (202); the scan is -deferred to Sync (phase 90). Source removal prunes on the next sync -(phase 69). +deferred to Sync. Source removal prunes on the next sync. --- ## 12. Current State & Roadmap -- **Shipped: phases 01–92** (`.agents/phases/complete/` — read-only - history; the shipped-features recap of the previous plan revision and - the README cover it). Migrations through 0015. -- **Next up (`todo/`, in order — already authored via - `phase-authoring`):** - 1. `93_theme_semantic_completion` — 8 semantic state colors join the - Theme-tab palette (17 total, B3 revised) + surface panels behind - every page head. - 2. `94_ls_tree_drilldown` — `ls` becomes a filesystem-style drill-down - tree with sync-time `lite` folder summaries (`folder_summaries` - table, migration 0017) + real-model battery. - 3. `95_read_truncation_cap` — capped `read` (default 128 000 chars - ≈ 32k tokens, `BOR_READ_MAX_CHARS`), honest truncation notice to - the LLM + visible "(truncated — showing N of M chars)" marker - (new optional `tool_result` SSE frame — A15 extended to seven - event types). -- **Next free phase number: 96.** +- **Shipped phases:** `ls .agents/phases/complete/` — read-only + history; each phase's `00_phase.md` has the full detail. The shipped- + features recap of the previous plan revision and the README cover it. +- **Migrations:** `ls alembic/versions/` — all reversible. +- **Next up (`todo/`):** Already authored via `phase-authoring`. + Read the directory listing in `.agents/phases/todo/` for the ordered + task list; each file's `00_phase.md` has the full description. +- **Next free phase number:** `max(completed phase numbers) + 1` — + computed from `.agents/phases/complete/`. - **Post-v1 hooks (deliberately not built):** HNSW index at scale; inotify auto-import; more providers (the OpenAI-compatible client is the seam); multi-user accounts (the `require_user` split is the seam); @@ -357,7 +348,7 @@ phase dir moved to `complete/`. (`app/rag/retriever.py`) is the only overflow marker; char budgets are the pattern for anything sent to `lite`. - **Byte-identical contracts are load-bearing:** deflected turns - (A8), empty prompt sections, the B4 theme no-op, and the phase-33/54 + (A8), empty prompt sections, the B4 theme no-op, and the rewrite-only caching all have tests that assert byte-identity. - **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib subprocess); everything else talks Postgres.