feat(ui): dark tech theme — emoji-free chrome, subtle animated CSS background, WCAG AA dark palette

.agent/PLAN.md

@@ -3,6 +3,9 @@
 > **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
 > **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
 > Anchors table are settled — do not re-litigate them in a phase.
+> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
+> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
+> theme (Phase 08); clickable document viewer (Phase 10). See roadmap §12.
 
 ---
 
@@ -19,7 +22,8 @@ that"* and offers alternatives instead of hallucinating.
 
 ### In scope (v1)
 - Chat UI (mobile-friendly, well-styled, no auth, no CDN).
-- RAG over `*.md` files **only** from `~/Homelab` + `~/Deployments`
+- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
+  by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
   (and any future directory the importer is pointed at).
 - Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
   `embed` (embeddings, **768 dims — verified**).
@@ -31,7 +35,7 @@ that"* and offers alternatives instead of hallucinating.
 
 ### Out of scope (v1)
 - Auth / multi-user (API is stateless under `/api` so it can be added later).
-- Non-markdown content, file uploads, caching layer, message persistence.
+- Binary / non-text content, file uploads, caching layer, message persistence.
 - Real-time document watching (manual re-import for now).
 
 ---
@@ -46,9 +50,9 @@ that"* and offers alternatives instead of hallucinating.
 | A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
 | A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
 | A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
-| A7 | Retrieval→context | Cosine **top-K=4 chunks** → map to parent documents → feed the **full text of top-N=2 documents** (deduped, capped at 24k chars) to the LLM | User requirement: whole-document context; mapping via `chunks.document_id → documents.path` | LOCKED |
-| A8 | Honesty gate | If best cosine similarity < `BOR_RELEVANCE_THRESHOLD` (0.30) → **deflection mode**: LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions | Required product behavior; threshold is tunable without code change | LOCKED |
-| A9 | Content scope | **`*.md` only**, with an exclusion list for non-content dirs (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) | Simplicity per user; prevents indexing dependency license files (~1.6k junk files in `~/Homelab/.venv`) | LOCKED |
+| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped, capped at 24k chars) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved | LOCKED (revised 2026-08-21) |
+| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
+| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
 | A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
 | A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
 | A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
@@ -84,7 +88,8 @@ that"* and offers alternatives instead of hallucinating.
                                        (self-hosted: turbo, embed)
 
    Offline tooling (same repo, same venv):
-     scripts/import_docs.py   → walks *.md dirs, chunks, embeds, upserts
+     scripts/import_docs.py   → walks A9-format dirs, chunks, embeds, upserts
+     scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
      scripts/llm_probe.py     → verifies models + embedding dim
 ```
 
@@ -93,7 +98,7 @@ that"* and offers alternatives instead of hallucinating.
 |-----------|----------------|----------|
 | **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
 | **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
-| **Importer** | Directory walk (exclusions), sha256 delta detection, markdown chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
+| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
 | **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
 | **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
 
@@ -102,12 +107,12 @@ that"* and offers alternatives instead of hallucinating.
 user question
   → POST /api/chat {message}
   → embed(question)                      [aipi /v1/embeddings, model=embed]
-  → SELECT chunks ORDER BY embedding <=> $1 LIMIT 4   [pgvector cosine]
-  → best_score = max(1 - distance)
-  ├─ best_score >= 0.30  →  top-2 documents' FULL content
+  → cosine top-30  +  FTS top-30 (OR tsquery, ts_rank)  [pgvector + PG FTS]
+  → RRF fuse (k=60) → docs ranked by best fused chunk score
+  ├─ best cosine >= 0.62 OR fts_hits > 0  →  top-2 documents' FULL content
   │                          → system prompt (persona + HONESTY rules + docs)
   │                          → turbo, stream=True  → SSE deltas
-  └─ best_score <  0.30  →  DEFLECT_MODE system prompt (weak hits as topics)
+  └─ else  →  DEFLECT_MODE system prompt (weak hits as topics)
                              → turbo, stream=True  → SSE deltas (honest reply)
   → query_log row (question, score, deflected, sources, latency)
   → final SSE "done" event: {deflected, sources[], suggestions[]}
@@ -124,6 +129,7 @@ All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
 | GET | `/api/health` | Liveness + db up/down + version | 01 |
 | GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
 | GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
+| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
 | POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
 
 ### SSE contract (`POST /api/chat`)
@@ -165,12 +171,13 @@ Created by `alembic/versions/0001_initial_schema.py` (idempotent
 | position | `INT` | 0-based order within the doc |
 | content | `TEXT` | chunk text (heading-aware) |
 | embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
+| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
 
 > No vector index in v1: sequential scan is fine at this corpus size
 > (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
 
 ### `query_log`
-`id UUID PK, question TEXT, top_score FLOAT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
+`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
 
 ### Document state transitions
 ```
@@ -185,6 +192,12 @@ than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
 `BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
 preceding heading in the text for retrieval quality.
 
+**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
+`---` separators (key line kept as anchor); `json` pretty-printed, split on
+top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
+`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
+hard cap (aipi ~1024-token request limit).
+
 ---
 
 ## 6. RAG Pipeline & Persona
@@ -213,12 +226,15 @@ Rules:
   model can suggest real alternatives (marker used by the E2E mock:
   `DEFLECT_MODE` appears in the system prompt).
 
-### Retrieval
-- Embed the question (`embed`, 768-d) → `ORDER BY embedding <=> $1 LIMIT 4`.
-- `score = 1 − cosine_distance`. Gate on `max(score) >= 0.30`.
-- Distinct parent docs ranked by best chunk score → top 2 → full content,
-  concatenated, truncated to `BOR_MAX_CONTEXT_CHARS` (24k) with a
-  `[…truncated…]` marker.
+### Retrieval (hybrid — A7/A8, revised 2026-08-21)
+- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
+- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
+- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
+  ranked by best chunk's fused score → top 2 → full content, concatenated,
+  truncated to `BOR_MAX_CONTEXT_CHARS` (24k) with a `[…truncated…]` marker.
+- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
+  (default 0.62, calibrated against the `embed` model's measured 0.41–0.84
+  distribution) **and** zero FTS hits among the candidates.
 
 ---
 
@@ -246,9 +262,14 @@ Rules:
   buttons always get `aria-label`); form input has a (visually-hidden) label.
 - Live regions: message stream `aria-live="polite"`; typing indicator
   `role="status"`; banner `role="status"`; errors `role="alert"`.
-- Contrast (verified pairs): ink `#1c2130` on `#fff` ≈14.9:1; ink-soft
-  `#4a5168` ≈7.6:1; white on brand `#4f46e5` ≈6.3:1; deflection text
-  `#92400e` on `#fff7e8` ≈8.7:1. All ≥4.5:1.
+- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
+  `#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
+  surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
+  on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
+  brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
+  (border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
+  all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
+  layer.
 - `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
   typing/spinner animations.
 
@@ -277,7 +298,10 @@ Rules:
 `#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
 `.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
 `#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
-`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`.
+`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
+(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
+`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
+path links).
 
 ---
 
@@ -298,7 +322,7 @@ Rules:
 - **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
   uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
 - **Per-chat-turn log line (required):**
-  `question=… embed_ms=… top_score=… threshold=… deflected=… sources=… total_ms=…`
+  `question=… embed_ms=… top_score=… fts_hits=… threshold=… deflected=… sources=… total_ms=…`
 - **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
   (counts, embedding batches, total time).
 - **`query_log` table:** durable record of every question (score, deflection,
@@ -333,13 +357,17 @@ Rules:
 # first import (and any future refresh):
 uv run python -m scripts.import_docs                # defaults: ~/Homelab ~/Deployments
 uv run python -m scripts.import_docs --source ~/OtherProject   # extra dirs
-uv run python -m scripts.import_docs --prune        # also drop deleted files
+uv run python -m scripts.import_docs --prune        # drop deleted / filtered-out files
+uv run python -m scripts.eval_retrieval "How did I install gitlab?"
 uv run python -m scripts.llm_probe                  # sanity: models + dim
 ```
 Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
 (no re-embedding); changed files are re-chunked + re-embedded (chunks
-replaced atomically); `--prune` removes docs whose files disappeared.
-Only `*.md` (A9) with the exclusion list (A9).
+replaced atomically); `--prune` removes docs whose files disappeared or no
+longer match the format filter. Formats per A9 (revised): `md, markdown,
+txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
+directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
+ranks live hybrid results for a question (retrieval tuning).
 
 ---
 
@@ -354,6 +382,9 @@ Only `*.md` (A9) with the exclusion list (A9).
 | 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
 | 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
 | 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
+| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
+| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
+| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
 
 Completion = unit+integration green, coverage >90%, story E2E green in
 isolation, UI verification passed, **one `--no-gpg-sign` commit**.
@@ -365,4 +396,4 @@ isolation, UI verification passed, **one `--no-gpg-sign` commit**.
 - HNSW index on `chunks.embedding` at scale.
 - Conversation persistence (messages tables).
 - Watchdog auto-re-import (inotify) — until then the script is the truth.
-- More sources: any directory of `*.md` via `--source`.
+- More sources: any directory of A9-format files via `--source`.