feat(rag): feed whole matched documents to the LLM — no context truncation (A7 revised)
This commit is contained in:
+481
@@ -0,0 +1,481 @@
|
|||||||
|
# Brain of Reese — Master Plan
|
||||||
|
|
||||||
|
> **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
|
||||||
|
> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
|
||||||
|
> Anchors table are settled — do not re-litigate them in a phase.
|
||||||
|
> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
|
||||||
|
> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
|
||||||
|
> theme (Phase 08); clickable document viewer (Phase 10); thinking display
|
||||||
|
> (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll
|
||||||
|
> (Phase 18, owner choice 2026-08-23); shared header (Phase 19);
|
||||||
|
> whole-document context (Phase 24 — A7's 24k context cap removed,
|
||||||
|
> owner permission 2026-08-24). See roadmap §12.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Mission
|
||||||
|
|
||||||
|
A **knowledge base chatbot** that embeds the `~/Homelab` and `~/Deployments`
|
||||||
|
projects into a Postgres vector database and lets anyone ask *Reese* (the
|
||||||
|
bot) questions about them.
|
||||||
|
|
||||||
|
**Product feel:** a chippy, upbeat assistant that is optimistic about the
|
||||||
|
user's ability ("you've got this") and **radically honest** — if retrieval
|
||||||
|
didn't surface anything relevant it says *"I haven't done anything like
|
||||||
|
that"* and offers alternatives instead of hallucinating.
|
||||||
|
|
||||||
|
### In scope (v1)
|
||||||
|
- Chat UI (mobile-friendly, well-styled, no auth, no CDN).
|
||||||
|
- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
|
||||||
|
by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
|
||||||
|
(and any future directory the importer is pointed at).
|
||||||
|
- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
|
||||||
|
`embed` (embeddings, **768 dims — verified**).
|
||||||
|
- Postgres 17 + pgvector, cosine similarity, chunk→document mapping so the
|
||||||
|
LLM receives the **entire relevant document** as context.
|
||||||
|
- Idempotent import/update script, documented in the README.
|
||||||
|
- Ample server logging + explicit UI loading/progress feedback (never a
|
||||||
|
stale submit button).
|
||||||
|
|
||||||
|
### Out of scope (v1)
|
||||||
|
- Auth / multi-user (API is stateless under `/api` so it can be added later).
|
||||||
|
- Binary / non-text content, file uploads, caching layer, message persistence.
|
||||||
|
- Real-time document watching (manual re-import for now).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Architectural Anchors (LOCKED DECISIONS)
|
||||||
|
|
||||||
|
| # | Component | Decision | Rationale | Status |
|
||||||
|
|---|-----------|----------|-----------|--------|
|
||||||
|
| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
|
||||||
|
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
|
||||||
|
| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
|
||||||
|
| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
|
||||||
|
| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
|
||||||
|
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
|
||||||
|
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
|
||||||
|
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
|
||||||
|
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
|
||||||
|
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
|
||||||
|
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
|
||||||
|
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
|
||||||
|
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3 | Standard, reversible, reviewable schema history | LOCKED |
|
||||||
|
| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
|
||||||
|
| A15 | Chat transport | **SSE streaming** from `POST /api/chat` (deltas + final `done` event with metadata) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED |
|
||||||
|
| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
|
||||||
|
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
|
||||||
|
|
||||||
|
> **A10 revision (phase 16, owner permission 2026-08-22):** single-admin
|
||||||
|
> signed-cookie auth — public: chat / documents / suggestions / health;
|
||||||
|
> admin-only: docs catalog + steering. The row above keeps the original v1
|
||||||
|
> decision text; the public API surface stays stateless (the signed
|
||||||
|
> session cookie is the only session state) — recorded as a revision,
|
||||||
|
> not a silent deviation.
|
||||||
|
>
|
||||||
|
> **A10 UI revision (phase 19, owner permission 2026-08-23):** the
|
||||||
|
> "Sources" nav link is hidden from anonymous users on all pages — the
|
||||||
|
> soft-gate page and the API split above are unchanged.
|
||||||
|
>
|
||||||
|
> **A7 revision (phase 24, owner permission 2026-08-24):** the
|
||||||
|
> `[…truncated…]` cap on document context is removed —
|
||||||
|
> `select_documents` always returns the full top-N texts;
|
||||||
|
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
|
||||||
|
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
|
||||||
|
> marker.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. High-Level Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────┐
|
||||||
|
│ Podman Compose │
|
||||||
|
Browser │ ┌──────────────────────────────────────┐ │
|
||||||
|
┌──────────┐ HTTP │ │ brain-of-reese/app (FastAPI) │ │
|
||||||
|
│ index.html│◄──────┼─►│ • static frontend (no CDN) │ │
|
||||||
|
│ app.js │ SSE │ │ • /api/chat /api/suggestions │ │
|
||||||
|
└──────────┘ │ │ • /api/health /api/docs │ │
|
||||||
|
│ │ • RAG pipeline (embed→retrieve→gen) │ │
|
||||||
|
│ └──────┬──────────────────┬───────────┘ │
|
||||||
|
│ │ SQL (psycopg) │ OpenAI-compat│
|
||||||
|
│ ┌──────▼──────┐ ┌───────▼────────────┐ │
|
||||||
|
│ │ db: │ └─────────┬──────────┘ │
|
||||||
|
│ │ postgres:17 │ │ │
|
||||||
|
│ │ + pgvector │ │ │
|
||||||
|
│ └─────────────┘ │ │
|
||||||
|
└──────────────────────────────┼────────────┘
|
||||||
|
▼
|
||||||
|
https://aipi.reeseapps.com/v1
|
||||||
|
(self-hosted: turbo, embed)
|
||||||
|
|
||||||
|
Offline tooling (same repo, same venv):
|
||||||
|
scripts/import_docs.py → walks A9-format dirs, chunks, embeds, upserts
|
||||||
|
scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
|
||||||
|
scripts/llm_probe.py → verifies models + embedding dim
|
||||||
|
```
|
||||||
|
|
||||||
|
### Component breakdown
|
||||||
|
| Component | Responsibility | Lives in |
|
||||||
|
|-----------|----------------|----------|
|
||||||
|
| **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
|
||||||
|
| **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
|
||||||
|
| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
|
||||||
|
| **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
|
||||||
|
| **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
|
||||||
|
|
||||||
|
### Chat data flow
|
||||||
|
```
|
||||||
|
user question
|
||||||
|
→ POST /api/chat {message}
|
||||||
|
→ embed(question) [aipi /v1/embeddings, model=embed]
|
||||||
|
→ cosine top-30 + FTS top-30 (OR tsquery, ts_rank) [pgvector + PG FTS]
|
||||||
|
→ RRF fuse (k=60) → docs ranked by best fused chunk score
|
||||||
|
├─ best cosine >= 0.62 OR fts_hits > 0 → top-2 documents' FULL content
|
||||||
|
│ → system prompt (persona + HONESTY rules + docs)
|
||||||
|
│ → turbo, stream=True → SSE deltas
|
||||||
|
└─ else → DEFLECT_MODE system prompt (weak hits as topics)
|
||||||
|
→ turbo, stream=True → SSE deltas (honest reply)
|
||||||
|
→ query_log row (question, score, deflected, sources, latency)
|
||||||
|
→ final SSE "done" event: {deflected, sources[], suggestions[]}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. API Design
|
||||||
|
|
||||||
|
All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
|
||||||
|
|
||||||
|
| Method | Path | Purpose | Story |
|
||||||
|
|--------|------|---------|-------|
|
||||||
|
| GET | `/api/health` | Liveness + db up/down + version | 01 |
|
||||||
|
| GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
|
||||||
|
| GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
|
||||||
|
| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
|
||||||
|
| POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
|
||||||
|
|
||||||
|
### SSE contract (`POST /api/chat`)
|
||||||
|
```
|
||||||
|
data: {"type":"thinking","text":"…"}\n\n
|
||||||
|
data: {"type":"thinking","text":"…"}\n\n
|
||||||
|
data: {"type":"delta","text":"Hey! "}\n\n
|
||||||
|
data: {"type":"delta","text":"Good "}\n\n
|
||||||
|
...
|
||||||
|
data: {"type":"done","deflected":false,"sources":[{"source":"Homelab","path":"kubernetes.md","title":"Kubernetes Homelab Cluster"}],"suggestions":[]}\n\n
|
||||||
|
```
|
||||||
|
Client rules: render deltas as they arrive; render `thinking` text in a
|
||||||
|
collapsible block above the answer; auto-collapse on the first `delta`;
|
||||||
|
tolerate interleaved `thinking` events (append — never reopen once the
|
||||||
|
answer started); the `done` shape is unchanged (thinking never travels on
|
||||||
|
`done`); on `done` append source chips / suggestion chips and clear the
|
||||||
|
busy state; on HTTP/stream error show the error banner + retry (never a
|
||||||
|
stuck button).
|
||||||
|
|
||||||
|
> **SSE revision (phase 17, owner permission 2026-08-23):** the contract
|
||||||
|
> gains one event type — `{"type":"thinking","text":"…"}` — carrying the
|
||||||
|
> model's reasoning ahead of the `delta` events (the `turbo` model emits
|
||||||
|
> `delta.reasoning_content` chunks before the first content chunk, verified
|
||||||
|
> live 2026-08-23; `BOR_STREAM_THINKING=0` suppresses the frames
|
||||||
|
> server-side). `delta` and `done` shapes are unchanged — a recorded
|
||||||
|
> extension of A15, not a silent deviation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Data Model (PostgreSQL 17)
|
||||||
|
|
||||||
|
Created by `alembic/versions/0001_initial_schema.py` (idempotent
|
||||||
|
`CREATE EXTENSION IF NOT EXISTS vector`).
|
||||||
|
|
||||||
|
### `documents`
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|--------|------|-------|
|
||||||
|
| id | `UUID` PK | |
|
||||||
|
| source | `VARCHAR(120)` | source dir basename, e.g. `Homelab` |
|
||||||
|
| path | `VARCHAR(1000)` | relative to source dir, e.g. `ansible/roles/k3s.md` |
|
||||||
|
| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
|
||||||
|
| title | `VARCHAR(500)` | first markdown H1, else file stem |
|
||||||
|
| content | `TEXT` | **full markdown — the RAG context** |
|
||||||
|
| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
|
||||||
|
| indexed_at | `TIMESTAMPTZ` | |
|
||||||
|
| — | `UNIQUE (source, path)` | upsert key |
|
||||||
|
|
||||||
|
### `chunks`
|
||||||
|
| Column | Type | Notes |
|
||||||
|
|--------|------|-------|
|
||||||
|
| id | `UUID` PK | |
|
||||||
|
| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
|
||||||
|
| position | `INT` | 0-based order within the doc |
|
||||||
|
| content | `TEXT` | chunk text (heading-aware) |
|
||||||
|
| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
|
||||||
|
| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
|
||||||
|
|
||||||
|
> No vector index in v1: sequential scan is fine at this corpus size
|
||||||
|
> (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
|
||||||
|
|
||||||
|
### `query_log`
|
||||||
|
`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
|
||||||
|
|
||||||
|
### Document state transitions
|
||||||
|
```
|
||||||
|
unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
|
||||||
|
│
|
||||||
|
└──file deleted + --prune──▶ removed (chunks cascade)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Chunking policy (markdown-aware)
|
||||||
|
Split on `## `/`### ` headings into sections; sub-split any section longer
|
||||||
|
than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
|
||||||
|
`BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
|
||||||
|
preceding heading in the text for retrieval quality.
|
||||||
|
|
||||||
|
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
|
||||||
|
`---` separators (key line kept as anchor); `json` pretty-printed, split on
|
||||||
|
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
|
||||||
|
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
|
||||||
|
hard cap (aipi ~1024-token request limit).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. RAG Pipeline & Persona
|
||||||
|
|
||||||
|
### Locked system prompt (sent with every chat turn)
|
||||||
|
```
|
||||||
|
You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
|
||||||
|
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
|
||||||
|
optimistic about the user's ability to do things ("you've got this").
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
1. Answer ONLY from the provided document context. Cite which document(s)
|
||||||
|
you used, by path.
|
||||||
|
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
|
||||||
|
the docs are the value.
|
||||||
|
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
|
||||||
|
Start your answer with a variant of: "I haven't done anything like that."
|
||||||
|
Then offer 2-3 alternative questions about things you DO have notes on.
|
||||||
|
4. Never invent facts, hosts, or steps that are not in the context.
|
||||||
|
5. Keep answers tight: short paragraphs, bullets where helpful.
|
||||||
|
|
||||||
|
<relevance>{HIGH|LOW}</relevance>
|
||||||
|
```
|
||||||
|
- `HIGH` mode appends the full document text under `<documents>…</documents>`.
|
||||||
|
- `LOW` mode (deflection) appends only the **titles** of the weak hits so the
|
||||||
|
model can suggest real alternatives (marker used by the E2E mock:
|
||||||
|
`DEFLECT_MODE` appears in the system prompt).
|
||||||
|
|
||||||
|
### Retrieval (hybrid — A7/A8, revised 2026-08-21)
|
||||||
|
- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
|
||||||
|
- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
|
||||||
|
- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
|
||||||
|
ranked by best chunk's fused score → top 2 → full content, concatenated
|
||||||
|
— **never truncated** (A7 revised, phase 24, owner permission
|
||||||
|
2026-08-24).
|
||||||
|
- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
|
||||||
|
(default 0.62, calibrated against the `embed` model's measured 0.41–0.84
|
||||||
|
distribution) **and** zero FTS hits among the candidates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. UI/UX Strategy
|
||||||
|
|
||||||
|
### 7.1 Layout structure
|
||||||
|
- **App frame:** sticky header (64px) + `<main>` (flex-grow) + footer.
|
||||||
|
Container: `max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem`.
|
||||||
|
- **Shared header (Phase 19, owner permission 2026-08-23):** the bar is a
|
||||||
|
shared contract across chat / sources / viewer — one bar per page, same
|
||||||
|
controls (brand + nav [Chat, Sources — admin only] + New Chat + Sign in
|
||||||
|
/ Sign out on chat & sources; the viewer bar = back + title + the same
|
||||||
|
actions in `.doc-header-actions`). One shared module
|
||||||
|
(`frontend/assets/header.js`) toggles the existing controls on each
|
||||||
|
page, so they can never "disappear" between pages again; the heights
|
||||||
|
stay pinned at 64px / 58px (phase 12).
|
||||||
|
- **Chat:** a *centered column capped at 46rem*. This is deliberate: chat is
|
||||||
|
a vertical conversation — a centered, capped column is the correct pattern
|
||||||
|
(NOT a layout bug). The 72rem frame + header/footer ensure the column
|
||||||
|
never reads as a hairline in a sea of whitespace.
|
||||||
|
- **Sources page:** full-width responsive **table** (min 640px, horizontal
|
||||||
|
scroll wrapper on small screens) + stat cards in
|
||||||
|
`grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
|
||||||
|
No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of
|
||||||
|
the container width.
|
||||||
|
- **Mobile (≤640px):** suggestion chips become a horizontally scrollable row;
|
||||||
|
composer stays reachable with `safe-area-inset-bottom`; touch targets ≥44px.
|
||||||
|
|
||||||
|
### 7.2 Accessibility (WCAG 2.1 AA)
|
||||||
|
- Semantic landmarks on every page: `<header>`, `<nav aria-label>`,
|
||||||
|
`<main>`, `<footer>`; skip-link to `#main`.
|
||||||
|
- Every control labeled: visible `<label>` or `aria-label` (icon-only
|
||||||
|
buttons always get `aria-label`); form input has a (visually-hidden) label.
|
||||||
|
- Live regions: message stream `aria-live="polite"`; typing indicator
|
||||||
|
`role="status"`; banner `role="status"`; errors `role="alert"`.
|
||||||
|
- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
|
||||||
|
`#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
|
||||||
|
surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
|
||||||
|
on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
|
||||||
|
brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
|
||||||
|
(border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
|
||||||
|
all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
|
||||||
|
layer.
|
||||||
|
- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
|
||||||
|
typing/spinner animations.
|
||||||
|
|
||||||
|
### 7.3 No external dependencies
|
||||||
|
- System font stack only (no font files to bundle, no CDN fonts).
|
||||||
|
- Zero `<script src="https://…">` / `<link href="https://…">` — enforced by
|
||||||
|
an integration test (`tests/integration/test_api.py::test_index_html_served_locally`)
|
||||||
|
and re-checked by every UI phase's verification step.
|
||||||
|
- Markdown rendering is a ~60-line local function (escape-first, then
|
||||||
|
transform) — XSS-safe, no library.
|
||||||
|
|
||||||
|
### 7.4 Visual feedback standard (the "never stale" contract)
|
||||||
|
| State | UI |
|
||||||
|
|-------|----|
|
||||||
|
| **Idle** | Send button enabled, label "Send". |
|
||||||
|
| **Thinking (pre-token)** | 3-dot typing bubble + button disabled with spinner, label "Thinking…". |
|
||||||
|
| **Thinking (model reasoning)** | Collapsible `.thinking` block streams open (replaces the typing dots as the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); 120s guard clears on the first `thinking` *or* `delta` event. |
|
||||||
|
| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
|
||||||
|
| **Done (answer)** | Source chips under the bubble (mono, path-based); button re-enabled. |
|
||||||
|
| **Done (deflected)** | Amber-bordered bubble + "Maybe try:" suggestion chips. |
|
||||||
|
| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled. |
|
||||||
|
| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
|
||||||
|
| **Guard** | 120s client-side timeout → error state (a button can never sit "stuck" forever). |
|
||||||
|
| **Scroll (follow-the-bottom, phase 18)** | The page auto-scrolls only while the user is pinned to the bottom (≤200px band, `NEAR_BOTTOM_PX` — the composer zone; submitting reveals the user's message through the same gate, which holds in real use); scrolling up holds the viewport for the rest of the turn (thinking and answer alike); restore lands one-shot on the latest message. |
|
||||||
|
|
||||||
|
> The **Thinking (model reasoning)** row is a phase-17 addition (owner
|
||||||
|
> permission 2026-08-23) — see the §4 SSE revision.
|
||||||
|
>
|
||||||
|
> The **Scroll** row is a phase-18 addition (owner choice 2026-08-23 —
|
||||||
|
> option 1: follow-the-bottom, no "↓ new content" pill).
|
||||||
|
|
||||||
|
### 7.5 Component inventory (ids used by tests)
|
||||||
|
`#messages` (stream), `#empty-state`, `#suggestions`, `.suggestion-chip`,
|
||||||
|
`#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
|
||||||
|
`.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
|
||||||
|
`#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
|
||||||
|
`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
|
||||||
|
(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
|
||||||
|
`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
|
||||||
|
path links); thinking (phase 17, owner permission 2026-08-23):
|
||||||
|
`.thinking`, `.thinking-text` (collapsible thinking block; plain
|
||||||
|
`<summary>`, no id); auth (phase 16, owner permission 2026-08-22):
|
||||||
|
`#sign-in-link`, `#sign-out-btn`, `#sources-gate`; shared header (phase
|
||||||
|
19, owner permission 2026-08-23): `#nav-sources` (Sources nav link,
|
||||||
|
hidden for anonymous), `#new-chat-btn` + `#sign-in-link` + `#sign-out-btn`
|
||||||
|
on the sources and viewer pages (ids shared with chat),
|
||||||
|
`.doc-header-actions` (viewer).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Debugging (debugpy protocol)
|
||||||
|
|
||||||
|
- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit test).
|
||||||
|
- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`, **non-blocking**,
|
||||||
|
app continues; IDE attaches on demand.
|
||||||
|
- Entry point: `app/core/debugging.py::configure_debugging()` called at the top
|
||||||
|
of `app/main.py` module import — so `uv run uvicorn app.main:app`,
|
||||||
|
`python -m scripts.…`, and tests all honor it.
|
||||||
|
- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Observability
|
||||||
|
|
||||||
|
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
|
||||||
|
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
|
||||||
|
- **Per-chat-turn log line (required):**
|
||||||
|
`question=… embed_ms=… top_score=… fts_hits=… tuning=N threshold=… deflected=… sources=… thinking_chars=… total_ms=…`
|
||||||
|
(`thinking_chars=` counts the turn's reasoning chars — phase 17, owner
|
||||||
|
permission 2026-08-23 — and is counted even when `BOR_STREAM_THINKING=0`
|
||||||
|
suppresses the frames.)
|
||||||
|
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
|
||||||
|
(counts, embedding batches, total time).
|
||||||
|
- **`query_log` table:** durable record of every question (score, deflection,
|
||||||
|
sources, latency) for tuning the threshold and finding gaps in the docs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Testing Strategy (LOCKED — A16)
|
||||||
|
|
||||||
|
| Layer | Tooling | Runs | Gate |
|
||||||
|
|-------|---------|------|------|
|
||||||
|
| Unit | pytest | `uv run pytest tests/unit` | pass |
|
||||||
|
| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
|
||||||
|
| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
|
||||||
|
| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
|
||||||
|
|
||||||
|
- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
|
||||||
|
OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap
|
||||||
|
vectors, so the cosine threshold behaves like production: on-topic
|
||||||
|
questions retrieve, off-topic questions deflect. `E2E_REAL_LLM=1` switches
|
||||||
|
the app fixture to live aipi (needs imported KB).
|
||||||
|
- **E2E prerequisites:** `podman compose up -d db`; Chromium installed via
|
||||||
|
`uv run playwright install chromium`.
|
||||||
|
- DB isolation: story E2E fixtures truncate `query_log` (and re-import
|
||||||
|
fixtures for import-dependent stories) per test module.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Import & Update Workflow (documented in README)
|
||||||
|
|
||||||
|
```
|
||||||
|
# first import (and any future refresh):
|
||||||
|
uv run python -m scripts.import_docs # defaults: ~/Homelab ~/Deployments
|
||||||
|
uv run python -m scripts.import_docs --source ~/OtherProject # extra dirs
|
||||||
|
uv run python -m scripts.import_docs --prune # drop deleted / filtered-out files
|
||||||
|
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
|
||||||
|
uv run python -m scripts.llm_probe # sanity: models + dim
|
||||||
|
```
|
||||||
|
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
|
||||||
|
(no re-embedding); changed files are re-chunked + re-embedded (chunks
|
||||||
|
replaced atomically); `--prune` removes docs whose files disappeared or no
|
||||||
|
longer match the format filter. Formats per A9 (revised): `md, markdown,
|
||||||
|
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
||||||
|
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
|
||||||
|
ranks live hybrid results for a question (retrieval tuning).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Roadmap (one story → one phase → one Playwright gate)
|
||||||
|
|
||||||
|
| Phase | File | Story | Playwright gate |
|
||||||
|
|-------|------|-------|-----------------|
|
||||||
|
| 01 | `01_infrastructure.md` | — (foundation) | `tests/e2e/test_smoke.py` |
|
||||||
|
| 02 | `02_story_import_documents.md` | `import-documents.md` | `tests/e2e/test_import_documents.py` |
|
||||||
|
| 03 | `03_story_chat_rag.md` | `chat-rag-answer.md` | `tests/e2e/test_chat_rag.py` |
|
||||||
|
| 04 | `04_story_honest_deflection.md` | `honest-deflection.md` | `tests/e2e/test_honest_deflection.py` |
|
||||||
|
| 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
|
||||||
|
| 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
|
||||||
|
| 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
|
||||||
|
| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
|
||||||
|
| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
|
||||||
|
| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
|
||||||
|
| 17 | `17_thinking_display.md` | `thinking-display.md` | `tests/e2e/test_thinking_display.py` |
|
||||||
|
| 18 | `18_follow_bottom_scroll.md` | `follow-bottom-scroll.md` | `tests/e2e/test_follow_bottom_scroll.py` |
|
||||||
|
| 19 | `19_shared_header.md` | `shared-header.md` | `tests/e2e/test_shared_header.py` |
|
||||||
|
| 24 | `24_whole_document_context.md` | `whole-document-context.md` | `tests/e2e/test_whole_document_context.py` |
|
||||||
|
|
||||||
|
> Row 17 (thinking display) added 2026-08-23 with owner permission — the
|
||||||
|
> A15 SSE extension recorded in §4.
|
||||||
|
>
|
||||||
|
> Row 18 (follow-the-bottom scroll) added 2026-08-23 with owner choice —
|
||||||
|
> option 1: follow-the-bottom, no "↓ new content" pill (UI-behavior-only
|
||||||
|
> change; no anchor revised).
|
||||||
|
>
|
||||||
|
> Row 19 (shared header) added 2026-08-23 with owner permission —
|
||||||
|
> Sign in/Sign out + New Chat on every page via one shared module, and
|
||||||
|
> the phase-16 "Sources" link UX choice revised: the nav link is hidden
|
||||||
|
> for anonymous (the soft-gate page and the A10 API split are unchanged).
|
||||||
|
>
|
||||||
|
> Row 24 (whole-document context) added 2026-08-24 with owner permission
|
||||||
|
> — A7's 24k context cap removed (documents are never truncated)
|
||||||
|
|
||||||
|
Completion = unit+integration green, coverage >90%, story E2E green in
|
||||||
|
isolation, UI verification passed, **one `--no-gpg-sign` commit**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Future (post-v1 hooks, deliberately not built)
|
||||||
|
- Auth (stateless API makes this a drop-in: sessions → Valkey).
|
||||||
|
- HNSW index on `chunks.embedding` at scale.
|
||||||
|
- Conversation persistence (messages tables).
|
||||||
|
- Watchdog auto-re-import (inotify) — until then the script is the truth.
|
||||||
|
- More sources: any directory of A9-format files via `--source`.
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# Story: Whole-Document Context — a matched document is never truncated
|
||||||
|
|
||||||
|
**Phase:** `24_whole_document_context` · **E2E:** `tests/e2e/test_whole_document_context.py`
|
||||||
|
|
||||||
|
## Bug report (TODO.md L3–L4, verbatim)
|
||||||
|
|
||||||
|
> *"Documents are truncated for some reason? This should never happen"*
|
||||||
|
>
|
||||||
|
> *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
|
||||||
|
|
||||||
|
## Narrative
|
||||||
|
|
||||||
|
As **a user asking about an indexed note**, I want **the entire document
|
||||||
|
in the LLM's context** whenever retrieval matches one of its chunks — so
|
||||||
|
the answer is grounded in the whole note, not a silently cut-down version
|
||||||
|
of it.
|
||||||
|
|
||||||
|
- **Given** a question whose chunk matches a document
|
||||||
|
- **When** the turn assembles context
|
||||||
|
- **Then** the LLM receives the **entire** parent document — never a
|
||||||
|
`[…truncated…]`-cut version — so the answer is grounded in the whole
|
||||||
|
note.
|
||||||
|
|
||||||
|
## Owner-confirmed (2026-08-24, roadmap D1–D5)
|
||||||
|
1. **D1 — no cap at all (revises LOCKED A7):** `select_documents` returns
|
||||||
|
the full top-N document texts, always. The `max_context_chars` setting
|
||||||
|
and `BOR_MAX_CONTEXT_CHARS` env var are removed. If a future KB ever
|
||||||
|
makes the prompt too large for the model, the existing `LLMError` → SSE
|
||||||
|
`error` path surfaces it loudly — no silent partial context. The
|
||||||
|
emergency-valve variant (raised cap + warning log) was **explicitly
|
||||||
|
rejected**.
|
||||||
|
2. **D2 — `top_n_docs = 2` unchanged** (the TODO is about truncation, not
|
||||||
|
about how many documents).
|
||||||
|
3. **D3 — no viewer/import changes** — both already serve full content
|
||||||
|
(verified diagnosis above).
|
||||||
|
4. **D4 — E2E evidence via a deterministic mock tail-echo** (repo pattern,
|
||||||
|
cf. the phase-15 tuning-note echo); the big documents are seeded
|
||||||
|
directly in the DB inside the E2E test — `tests/fixtures/docs/` must
|
||||||
|
not grow, because other suites pin `summary.added == 8`.
|
||||||
|
5. **D5 — no `query_log` schema change** (no new columns, no migration).
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
1. No budget parameter in `select_documents` — it never truncates
|
||||||
|
(`TRUNCATION_MARKER` remains for the steering section only).
|
||||||
|
2. `BOR_MAX_CONTEXT_CHARS` gone from settings/env/README
|
||||||
|
(`app/config.py`, `.env.example`, `README.md`).
|
||||||
|
3. The story E2E's three tests green in isolation
|
||||||
|
(`uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`).
|
||||||
|
4. Steering-note truncation (phase 15, `BOR_STEERING_MAX_CHARS` + shared
|
||||||
|
marker) unchanged.
|
||||||
|
5. Unit + integration green, `app/` coverage >90%, one
|
||||||
|
`--no-gpg-sign` commit.
|
||||||
|
|
||||||
|
## Playwright Mapping Rule
|
||||||
|
**Test Scenario → `tests/e2e/test_whole_document_context.py`** (mock LLM
|
||||||
|
tail-echo, oversized documents seeded directly in the DB):
|
||||||
|
1. `test_whole_document_over_old_cap_reaches_llm` — a 30 000-char document
|
||||||
|
(past the old 24 000 cap): its tail sentinel (last line) is echoed in
|
||||||
|
the rendered answer, `[…truncated…]` is absent, the source chip renders,
|
||||||
|
`query_log` row `deflected == False`.
|
||||||
|
2. `test_second_document_of_over_cap_pair_reaches_llm` — two ~16 000-char
|
||||||
|
documents (32 000 combined — the exact case the old budget cut): the
|
||||||
|
second, lower-ranked document's tail sentinel is echoed (doc 1's is not
|
||||||
|
— it pins the rank order), both source chips render.
|
||||||
|
3. `test_small_document_path_unchanged` — regression: standard fixtures via
|
||||||
|
the real importer → grounded answer with the `kubernetes.md` chip, no
|
||||||
|
`[…truncated…]` (the under-cap path is byte-identical to before).
|
||||||
@@ -21,7 +21,6 @@ BOR_STREAM_THINKING=1 # stream the model's thinking as `thinking` SS
|
|||||||
# --- RAG tuning ---
|
# --- RAG tuning ---
|
||||||
BOR_TOP_N_DOCS=2
|
BOR_TOP_N_DOCS=2
|
||||||
BOR_RELEVANCE_THRESHOLD=0.62 # answer when best cosine >= this OR an FTS hit; else honest deflection
|
BOR_RELEVANCE_THRESHOLD=0.62 # answer when best cosine >= this OR an FTS hit; else honest deflection
|
||||||
BOR_MAX_CONTEXT_CHARS=24000 # cap on total document text sent to the LLM
|
|
||||||
BOR_MAX_OUTPUT_TOKENS=32768 # max answer length in tokens (answers must not be cut off)
|
BOR_MAX_OUTPUT_TOKENS=32768 # max answer length in tokens (answers must not be cut off)
|
||||||
BOR_STEERING_MAX_CHARS=8000 # char budget for the <tuning> (steering notes) prompt section
|
BOR_STEERING_MAX_CHARS=8000 # char budget for the <tuning> (steering notes) prompt section
|
||||||
BOR_CHUNK_TARGET_CHARS=2000
|
BOR_CHUNK_TARGET_CHARS=2000
|
||||||
|
|||||||
@@ -374,7 +374,6 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
|||||||
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
|
||||||
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
|
||||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||||
| `BOR_MAX_CONTEXT_CHARS` | `24000` | cap on total document text sent to the LLM |
|
|
||||||
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
||||||
| `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
|
| `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
|
||||||
| `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |
|
| `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |
|
||||||
|
|||||||
+2
-4
@@ -126,9 +126,7 @@ def plan_turn(
|
|||||||
best_cosine = max((c.cosine for c in chunks), default=0.0)
|
best_cosine = max((c.cosine for c in chunks), default=0.0)
|
||||||
fts_hits = sum(1 for c in chunks if c.fts_hit)
|
fts_hits = sum(1 for c in chunks if c.fts_hit)
|
||||||
if best_cosine >= settings.relevance_threshold or fts_hits > 0:
|
if best_cosine >= settings.relevance_threshold or fts_hits > 0:
|
||||||
docs = select_documents(
|
docs = select_documents(chunks, n=settings.top_n_docs)
|
||||||
chunks, n=settings.top_n_docs, max_chars=settings.max_context_chars
|
|
||||||
)
|
|
||||||
return TurnPlan(
|
return TurnPlan(
|
||||||
best_cosine,
|
best_cosine,
|
||||||
fts_hits,
|
fts_hits,
|
||||||
@@ -144,7 +142,7 @@ def plan_turn(
|
|||||||
fts_hits,
|
fts_hits,
|
||||||
True,
|
True,
|
||||||
build_deflect_prompt(titles, notes=steering),
|
build_deflect_prompt(titles, notes=steering),
|
||||||
select_documents(chunks, n=settings.top_n_docs, max_chars=settings.max_context_chars),
|
select_documents(chunks, n=settings.top_n_docs),
|
||||||
derive_suggestions(titles, settings.suggestions),
|
derive_suggestions(titles, settings.suggestions),
|
||||||
len(steering),
|
len(steering),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -55,7 +55,6 @@ class Settings(BaseSettings):
|
|||||||
# default never discriminated. LOW only fires when best cosine < this
|
# default never discriminated. LOW only fires when best cosine < this
|
||||||
# AND no candidate chunk matches the question lexically (see A8).
|
# AND no candidate chunk matches the question lexically (see A8).
|
||||||
relevance_threshold: float = 0.62
|
relevance_threshold: float = 0.62
|
||||||
max_context_chars: int = 24_000
|
|
||||||
#: Maximum output tokens a chat answer may use (owner instruction
|
#: Maximum output tokens a chat answer may use (owner instruction
|
||||||
#: 2026-08-22: answers must run to their natural end — the old hard
|
#: 2026-08-22: answers must run to their natural end — the old hard
|
||||||
#: 700-token cap cut long answers off mid-sentence).
|
#: 700-token cap cut long answers off mid-sentence).
|
||||||
|
|||||||
+15
-23
@@ -10,10 +10,12 @@
|
|||||||
fused score ranks; :meth:`select_documents` and :func:`weak_hit_titles`
|
fused score ranks; :meth:`select_documents` and :func:`weak_hit_titles`
|
||||||
keep working off ``score``.
|
keep working off ``score``.
|
||||||
|
|
||||||
The product requirement is unchanged (LOCKED A7): the LLM receives the
|
The product requirement (LOCKED A7, revised 2026-08-24): the LLM receives
|
||||||
**entire relevant document**, not just the chunk — chunk hits map back to
|
the **entire relevant document**, not just the chunk — chunk hits map back
|
||||||
their parents, dedupe, rank by best fused score, and the combined context
|
to their parents, dedupe, rank by best fused score, and the full text of
|
||||||
is capped at ``BOR_MAX_CONTEXT_CHARS``.
|
the top-N documents is always fed through, never truncated. If a future KB
|
||||||
|
ever makes the prompt too large for the model, the ``LLMError`` → SSE
|
||||||
|
``error`` path surfaces it loudly — no silent partial context.
|
||||||
|
|
||||||
Deterministic tie-break for equal fused scores:
|
Deterministic tie-break for equal fused scores:
|
||||||
``(−fused, −cosine, document.path, chunk.position)``.
|
``(−fused, −cosine, document.path, chunk.position)``.
|
||||||
@@ -31,7 +33,9 @@ from sqlalchemy.orm import Session
|
|||||||
from app.config import get_settings
|
from app.config import get_settings
|
||||||
from app.models import Chunk, Document
|
from app.models import Chunk, Document
|
||||||
|
|
||||||
#: Marker appended when the context budget is exceeded (PLAN §6).
|
#: Shared overflow marker — now used by the steering (<tuning>) section
|
||||||
|
#: only (phase 15; imported by ``app.rag.prompts``). The document context
|
||||||
|
#: path never truncates (A7 revised, owner permission 2026-08-24).
|
||||||
TRUNCATION_MARKER = "[…truncated…]"
|
TRUNCATION_MARKER = "[…truncated…]"
|
||||||
|
|
||||||
#: Alphanumeric tokens of a question (``to_tsquery`` input, OR-joined).
|
#: Alphanumeric tokens of a question (``to_tsquery`` input, OR-joined).
|
||||||
@@ -254,19 +258,17 @@ def weak_hit_titles(chunks: Sequence[RetrievedChunk]) -> list[str]:
|
|||||||
def select_documents(
|
def select_documents(
|
||||||
chunks: Sequence[RetrievedChunk],
|
chunks: Sequence[RetrievedChunk],
|
||||||
n: int | None = None,
|
n: int | None = None,
|
||||||
max_chars: int | None = None,
|
|
||||||
) -> list[Document]:
|
) -> list[Document]:
|
||||||
"""Map chunk hits to distinct parent documents, ranked by best fused score.
|
"""Map chunk hits to distinct parent documents, ranked by best fused score.
|
||||||
|
|
||||||
At most *n* documents are returned (default ``BOR_TOP_N_DOCS``). The
|
At most *n* documents are returned (default ``BOR_TOP_N_DOCS``). The
|
||||||
returned rows carry the full document content; if the combined content
|
returned rows carry the full document content, byte-identical — a
|
||||||
would exceed *max_chars* (default ``BOR_MAX_CONTEXT_CHARS``), the
|
matched parent document is **never truncated** (A7 revised, owner
|
||||||
lowest-ranked overflowing document is truncated in place with the
|
permission 2026-08-24). There is deliberately no context budget: an
|
||||||
``[…truncated…]`` marker so the assembled context never exceeds the
|
oversized prompt must fail loudly through the ``LLMError`` → SSE
|
||||||
budget (PLAN §6).
|
``error`` path, never arrive as silent partial context.
|
||||||
"""
|
"""
|
||||||
top_n = n if n is not None else get_settings().top_n_docs
|
top_n = n if n is not None else get_settings().top_n_docs
|
||||||
budget = max_chars if max_chars is not None else get_settings().max_context_chars
|
|
||||||
|
|
||||||
docs: list[Document] = []
|
docs: list[Document] = []
|
||||||
seen: set[uuid.UUID] = set()
|
seen: set[uuid.UUID] = set()
|
||||||
@@ -275,14 +277,4 @@ def select_documents(
|
|||||||
continue
|
continue
|
||||||
seen.add(rc.document.id)
|
seen.add(rc.document.id)
|
||||||
docs.append(rc.document)
|
docs.append(rc.document)
|
||||||
docs = docs[:top_n]
|
return docs[:top_n]
|
||||||
|
|
||||||
remaining = budget
|
|
||||||
for doc in docs:
|
|
||||||
if len(doc.content) <= remaining:
|
|
||||||
remaining -= len(doc.content)
|
|
||||||
else:
|
|
||||||
keep = max(0, remaining - len(TRUNCATION_MARKER))
|
|
||||||
doc.content = doc.content[:keep] + TRUNCATION_MARKER
|
|
||||||
remaining = 0
|
|
||||||
return docs
|
|
||||||
|
|||||||
@@ -26,6 +26,11 @@ Implements just enough of the aipi surface:
|
|||||||
- system prompt containing ``<tuning>`` (phase 15, steering notes) ->
|
- system prompt containing ``<tuning>`` (phase 15, steering notes) ->
|
||||||
the composed answer ends with `` (tuning: <first note line>)`` —
|
the composed answer ends with `` (tuning: <first note line>)`` —
|
||||||
makes prompt injection observable in the UI deterministically.
|
makes prompt injection observable in the UI deterministically.
|
||||||
|
- user message containing ``show the end of your notes`` (phase 24,
|
||||||
|
whole-document context) -> the answer quotes the **last 160 chars of
|
||||||
|
the document context** — a tail echo, byte-stable across runs, so a
|
||||||
|
sentinel placed at the *end* of a document appears in the rendered
|
||||||
|
answer iff the whole document was in the prompt.
|
||||||
|
|
||||||
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
``max_tokens`` is honored deterministically (token ≈ whitespace word),
|
||||||
like a real endpoint: an answer longer than the cap is truncated. This
|
like a real endpoint: an answer longer than the cap is truncated. This
|
||||||
@@ -104,6 +109,13 @@ THINKING_TRIGGER = "think out loud"
|
|||||||
SLOW_PRETOKEN_TRIGGER = "think out loud then hesitate"
|
SLOW_PRETOKEN_TRIGGER = "think out loud then hesitate"
|
||||||
PRE_CONTENT_PAUSE_S = 4.0
|
PRE_CONTENT_PAUSE_S = 4.0
|
||||||
|
|
||||||
|
#: Phase 24 (whole-document-context story): a user message containing this
|
||||||
|
#: substring (case-insensitive) gets an answer quoting the TAIL of the
|
||||||
|
#: document context (see the module docstring). Verified 2026-08-24: no
|
||||||
|
#: existing E2E question or fixture file contains the phrase, so every
|
||||||
|
#: other suite is unaffected.
|
||||||
|
END_OF_NOTES_TRIGGER = "show the end of your notes"
|
||||||
|
|
||||||
|
|
||||||
def long_answer() -> str:
|
def long_answer() -> str:
|
||||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||||
@@ -151,6 +163,16 @@ def compose_answer(body: dict[str, Any]) -> str:
|
|||||||
"kubernetes, backups, or deploying a new service — I know those inside out. "
|
"kubernetes, backups, or deploying a new service — I know those inside out. "
|
||||||
"You've got this!"
|
"You've got this!"
|
||||||
)
|
)
|
||||||
|
elif END_OF_NOTES_TRIGGER in user.lower():
|
||||||
|
# Whole-document-context story (phase 24): echo the tail of the
|
||||||
|
# context. Byte-stable across runs — a sentinel on the document's
|
||||||
|
# last line appears in the answer iff the whole document was in
|
||||||
|
# the prompt. (The tail includes the closing </documents> —
|
||||||
|
# harmless for the E2E sentinel assertions.)
|
||||||
|
answer = (
|
||||||
|
f"…and the very end of my notes reads: “{_context(body)[-160:]}” "
|
||||||
|
"(Deterministic mock answer for E2E.)"
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
ctx = _context(body)
|
ctx = _context(body)
|
||||||
snippet = ctx[:220].replace("\n", " ").strip()
|
snippet = ctx[:220].replace("\n", " ").strip()
|
||||||
|
|||||||
@@ -0,0 +1,315 @@
|
|||||||
|
"""Phase 24 E2E (Playwright): a matched document reaches the LLM whole.
|
||||||
|
|
||||||
|
Story: ``.agent/user_stories/whole-document-context.md``
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
|
||||||
|
|
||||||
|
The mock LLM's tail-echo trigger (``END_OF_NOTES_TRIGGER``, see
|
||||||
|
``tests/e2e/mock_llm.py``) makes the model quote the last 160 chars of
|
||||||
|
the document context. A sentinel placed on the *last line* of a document
|
||||||
|
therefore appears in the rendered answer **iff the entire document was in
|
||||||
|
the prompt** — which is what makes the no-truncation contract (A7 revised,
|
||||||
|
owner permission 2026-08-24: matched parent documents are never cut)
|
||||||
|
provable end-to-end.
|
||||||
|
|
||||||
|
The oversized documents are seeded directly via SQLAlchemy (a
|
||||||
|
``documents`` row + 2–3 ``chunks`` rows whose embeddings are the mock's
|
||||||
|
own deterministic bag-of-words vectors, so the question's live mock
|
||||||
|
embedding genuinely overlaps — no fixture files added:
|
||||||
|
``tests/fixtures/docs/`` stays at its 8 files, other suites pin
|
||||||
|
``summary.added == 8``).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import hashlib
|
||||||
|
from collections.abc import Callable, Sequence
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from threading import Thread
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from playwright.sync_api import Page, expect
|
||||||
|
from sqlalchemy import select, text
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.config import Settings
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import Chunk, Document, QueryLog
|
||||||
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
|
from app.rag.llm import LLMClient
|
||||||
|
from app.rag.retriever import TRUNCATION_MARKER
|
||||||
|
from tests.e2e.mock_llm import embed_text
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||||
|
|
||||||
|
#: The pre-phase-24 ``BOR_MAX_CONTEXT_CHARS`` default — the budget this
|
||||||
|
#: suite proves is gone from the document path.
|
||||||
|
OLD_CONTEXT_CAP = 24_000
|
||||||
|
|
||||||
|
QUESTION = "Show the end of your notes about the gitlab install playbook, please."
|
||||||
|
SMALL_QUESTION = "How is my Kubernetes cluster set up?"
|
||||||
|
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||||
|
|
||||||
|
|
||||||
|
# --- Content builders (deterministic, token-controlled) -------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _repeated(line: str, min_chars: int) -> str:
|
||||||
|
""""line" (newline-terminated) repeated until at least min_chars chars."""
|
||||||
|
unit = line + "\n"
|
||||||
|
return unit * max(1, -(-min_chars // len(unit)))
|
||||||
|
|
||||||
|
|
||||||
|
def _doc_slices(content: str, n: int) -> list[str]:
|
||||||
|
"""Even slices of *content* (the last slice keeps the final line)."""
|
||||||
|
step = len(content) // n
|
||||||
|
return [content[i * step : (i + 1) * step] for i in range(n - 1)] + [
|
||||||
|
content[(n - 1) * step :]
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _gitlab_30k_doc(sentinel: str) -> str:
|
||||||
|
"""A ~30 000-char document (past the old 24k cap): a body of repeated
|
||||||
|
"gitlab install playbook" lines — the same tokens the question carries,
|
||||||
|
so hybrid retrieval genuinely hits — whose LAST line is a unique
|
||||||
|
sentinel only a tail echo can surface."""
|
||||||
|
body = _repeated(
|
||||||
|
"gitlab install playbook: run the gitlab install playbook on the homelab host.",
|
||||||
|
OLD_CONTEXT_CAP + 6_000,
|
||||||
|
)
|
||||||
|
return body + sentinel + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _pair_doc(strong_line: str, filler_line: str, sentinel: str) -> tuple[str, list[str]]:
|
||||||
|
"""A ~16 000-char document: a ~2 000-char first chunk carrying the
|
||||||
|
question's key tokens, a ~14 000-char low-overlap remainder, and a
|
||||||
|
unique sentinel as the last line. Returns (content, chunk_texts)."""
|
||||||
|
chunk0 = _repeated(strong_line, 2_000)
|
||||||
|
chunk1 = _repeated(filler_line, 14_000)
|
||||||
|
return chunk0 + chunk1 + sentinel + "\n", [chunk0, chunk1]
|
||||||
|
|
||||||
|
|
||||||
|
# --- DB seeding (TRUNCATE-then-seed, cf. test_chat_rag.py) -----------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_doc(
|
||||||
|
db: Session,
|
||||||
|
source: str,
|
||||||
|
path: str,
|
||||||
|
title: str,
|
||||||
|
content: str,
|
||||||
|
chunk_texts: Sequence[str],
|
||||||
|
) -> None:
|
||||||
|
"""One ``documents`` row + one ``chunks`` row per chunk text.
|
||||||
|
|
||||||
|
Each chunk's embedding is the mock's own ``embed_text`` vector, so the
|
||||||
|
app's live mock embedding of the question genuinely overlaps.
|
||||||
|
"""
|
||||||
|
doc = Document(
|
||||||
|
source=source,
|
||||||
|
path=path,
|
||||||
|
full_path=f"/tmp/{path}",
|
||||||
|
title=title,
|
||||||
|
content=content,
|
||||||
|
content_hash=hashlib.sha256(content.encode("utf-8")).hexdigest(),
|
||||||
|
indexed_at=datetime.now(UTC),
|
||||||
|
)
|
||||||
|
db.add(doc)
|
||||||
|
db.flush()
|
||||||
|
db.add_all(
|
||||||
|
Chunk(document_id=doc.id, position=i, content=chunk, embedding=embed_text(chunk))
|
||||||
|
for i, chunk in enumerate(chunk_texts)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
||||||
|
"""Truncate the KB (and query log), then optionally run *seed*."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(text("TRUNCATE chunks, documents, query_log"))
|
||||||
|
db.commit()
|
||||||
|
if seed is not None:
|
||||||
|
seed(db)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
# --- Importer + thread helpers (test_chat_rag.py pattern) ------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||||
|
kwargs: dict[str, Any] = {
|
||||||
|
"_env_file": None,
|
||||||
|
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
|
||||||
|
}
|
||||||
|
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
return await import_sources([FIXTURES], LLMClient(settings))
|
||||||
|
|
||||||
|
|
||||||
|
def _run_in_thread(coro: Any) -> Any:
|
||||||
|
"""Run a coroutine on a worker thread.
|
||||||
|
|
||||||
|
Playwright's sync API keeps an asyncio loop running on the test thread,
|
||||||
|
so ``asyncio.run`` cannot be called directly from a test body.
|
||||||
|
"""
|
||||||
|
box: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def runner() -> None:
|
||||||
|
try:
|
||||||
|
box["value"] = asyncio.run(coro)
|
||||||
|
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||||
|
box["error"] = e
|
||||||
|
|
||||||
|
t = Thread(target=runner)
|
||||||
|
t.start()
|
||||||
|
t.join()
|
||||||
|
if "error" in box:
|
||||||
|
raise box["error"]
|
||||||
|
return box["value"]
|
||||||
|
|
||||||
|
|
||||||
|
def _ask(page: Page, app_url: str, question: str) -> Any:
|
||||||
|
"""Submit *question* and wait for the streamed brain bubble."""
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
page.goto(app_url)
|
||||||
|
page.fill("#message-input", question)
|
||||||
|
page.click("#send-btn")
|
||||||
|
bubble = page.locator(".msg.brain .bubble")
|
||||||
|
bubble.first.wait_for(state="visible", timeout=30_000)
|
||||||
|
return bubble.first
|
||||||
|
|
||||||
|
|
||||||
|
def _last_query_log() -> QueryLog:
|
||||||
|
with SessionLocal() as db:
|
||||||
|
rows = db.scalars(select(QueryLog)).all()
|
||||||
|
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||||
|
return rows[0]
|
||||||
|
|
||||||
|
|
||||||
|
# --- Story tests -------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_whole_document_over_old_cap_reaches_llm(
|
||||||
|
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
"""A 30k document (past the old 24k cap) reaches the LLM whole: its
|
||||||
|
tail sentinel — the last 160 chars of the context — is echoed back."""
|
||||||
|
sentinel = "WHOLE-DOC-TAIL-GITLAB-30K"
|
||||||
|
content = _gitlab_30k_doc(sentinel)
|
||||||
|
assert len(content) > OLD_CONTEXT_CAP # this is the point of the test
|
||||||
|
|
||||||
|
def seed(db: Session) -> None:
|
||||||
|
_seed_doc(
|
||||||
|
db,
|
||||||
|
"Homelab",
|
||||||
|
"gitlab-30k.md",
|
||||||
|
"GitLab Install Playbook (30k)",
|
||||||
|
content,
|
||||||
|
_doc_slices(content, 3),
|
||||||
|
)
|
||||||
|
|
||||||
|
_reset_db(seed)
|
||||||
|
bubble = _ask(page, app_url, QUESTION)
|
||||||
|
|
||||||
|
# The tail sentinel exists only on the document's last line — its
|
||||||
|
# presence proves the entire 30k document was in the LLM prompt.
|
||||||
|
expect(bubble).to_contain_text(sentinel, timeout=30_000)
|
||||||
|
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||||
|
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||||
|
|
||||||
|
# Grounded: the document's source chip renders under the bubble.
|
||||||
|
chip = page.locator(".msg.brain .source-chip", has_text="gitlab-30k.md")
|
||||||
|
expect(chip).to_have_count(1)
|
||||||
|
expect(chip.first).to_contain_text("Homelab/gitlab-30k.md")
|
||||||
|
|
||||||
|
# Button recovers (never stale) and the turn was grounded.
|
||||||
|
expect(page.locator("#send-btn")).to_be_enabled()
|
||||||
|
expect(page.locator("#send-label")).to_have_text("Send")
|
||||||
|
row = _last_query_log()
|
||||||
|
assert row.question == QUESTION
|
||||||
|
assert row.deflected is False
|
||||||
|
assert "Homelab/gitlab-30k.md" in row.sources
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_document_of_over_cap_pair_reaches_llm(
|
||||||
|
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
"""Two ~16k documents (32k combined — under the old budget the
|
||||||
|
lower-ranked one was truncated in place): the SECOND document, the
|
||||||
|
last block inside <documents>, reaches the LLM whole — its sentinel is
|
||||||
|
the one the tail echo surfaces."""
|
||||||
|
sentinel_a = "WHOLE-DOC-TAIL-PAIR-A"
|
||||||
|
sentinel_b = "WHOLE-DOC-TAIL-PAIR-B"
|
||||||
|
# Doc A's first chunk carries the question's key tokens → it ranks
|
||||||
|
# first in both candidate lists → it comes first in <documents>.
|
||||||
|
content_a, chunks_a = _pair_doc(
|
||||||
|
"gitlab install playbook: run the gitlab install playbook end to end.",
|
||||||
|
"the server room keeps a steady temperature and the racks are labelled.",
|
||||||
|
sentinel_a,
|
||||||
|
)
|
||||||
|
# Doc B's first chunk has only a weaker overlap ("playbook", "notes")
|
||||||
|
# → it ranks second → it is the LAST block inside <documents>.
|
||||||
|
content_b, chunks_b = _pair_doc(
|
||||||
|
"playbook notes: the playbook notes track what changed and where.",
|
||||||
|
"the rack elevation drawing shows cable trays and pdu positions.",
|
||||||
|
sentinel_b,
|
||||||
|
)
|
||||||
|
assert len(content_a) + len(content_b) > OLD_CONTEXT_CAP # 32k > 24k
|
||||||
|
|
||||||
|
def seed(db: Session) -> None:
|
||||||
|
_seed_doc(
|
||||||
|
db, "Homelab", "gitlab-install-playbook.md",
|
||||||
|
"GitLab Install Playbook", content_a, chunks_a,
|
||||||
|
)
|
||||||
|
_seed_doc(
|
||||||
|
db, "Homelab", "playbook-notes.md",
|
||||||
|
"Playbook Notes", content_b, chunks_b,
|
||||||
|
)
|
||||||
|
|
||||||
|
_reset_db(seed)
|
||||||
|
bubble = _ask(page, app_url, QUESTION)
|
||||||
|
|
||||||
|
# The tail echo quotes doc B's sentinel (the last block's tail) — doc B
|
||||||
|
# was in the prompt whole, past the old budget. Doc A's sentinel sits
|
||||||
|
# mid-prompt, so it must NOT be in the quoted tail: that is what pins
|
||||||
|
# the rank order (A first, B last inside <documents>).
|
||||||
|
expect(bubble).to_contain_text(sentinel_b, timeout=30_000)
|
||||||
|
expect(bubble).not_to_contain_text(sentinel_a)
|
||||||
|
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||||
|
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||||
|
|
||||||
|
# Both documents are cited (top-2), in rank order.
|
||||||
|
chips = page.locator(".msg.brain .source-chip")
|
||||||
|
expect(chips).to_have_count(2)
|
||||||
|
expect(page.locator(".msg.brain .source-chip",
|
||||||
|
has_text="gitlab-install-playbook.md")).to_have_count(1)
|
||||||
|
expect(page.locator(".msg.brain .source-chip",
|
||||||
|
has_text="playbook-notes.md")).to_have_count(1)
|
||||||
|
|
||||||
|
row = _last_query_log()
|
||||||
|
assert row.question == QUESTION
|
||||||
|
assert row.deflected is False
|
||||||
|
assert "Homelab/gitlab-install-playbook.md" in row.sources
|
||||||
|
assert "Homelab/playbook-notes.md" in row.sources
|
||||||
|
|
||||||
|
|
||||||
|
def test_small_document_path_unchanged(
|
||||||
|
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
"""Regression: the standard (small) fixtures still take the grounded
|
||||||
|
path, byte-identical to before — no marker, kubernetes.md cited."""
|
||||||
|
_reset_db(None)
|
||||||
|
summary = _run_in_thread(_import_fixtures(mock_llm))
|
||||||
|
assert summary.added == 8 # A9 formats (fixture set unchanged)
|
||||||
|
|
||||||
|
bubble = _ask(page, app_url, SMALL_QUESTION)
|
||||||
|
expect(bubble).to_contain_text(SMALL_QUESTION, timeout=30_000)
|
||||||
|
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||||
|
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||||
|
|
||||||
|
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||||
|
expect(chip).to_have_count(1)
|
||||||
|
row = _last_query_log()
|
||||||
|
assert row.deflected is False
|
||||||
|
assert "docs/homelab/kubernetes.md" in row.sources
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Unit: retriever — ordering, dedup, and the context cap (fake rows).
|
"""Unit: retriever — ordering and dedup, no context cap (A7 revised) (fake rows).
|
||||||
|
|
||||||
The SQL side of :func:`app.rag.retriever.retrieve` is exercised by the
|
The SQL side of :func:`app.rag.retriever.retrieve` is exercised by the
|
||||||
chat integration tests against real Postgres; the pure mapping logic in
|
chat integration tests against real Postgres; the pure mapping logic in
|
||||||
@@ -47,14 +47,14 @@ def test_ranks_by_best_chunk_score_not_first_hit() -> None:
|
|||||||
_chunk(a, 0.9, position=2), # a's best chunk comes last
|
_chunk(a, 0.9, position=2), # a's best chunk comes last
|
||||||
_chunk(c, 0.5),
|
_chunk(c, 0.5),
|
||||||
]
|
]
|
||||||
docs = select_documents(chunks, n=3, max_chars=10_000)
|
docs = select_documents(chunks, n=3)
|
||||||
assert [d.path for d in docs] == ["a.md", "b.md", "c.md"]
|
assert [d.path for d in docs] == ["a.md", "b.md", "c.md"]
|
||||||
|
|
||||||
|
|
||||||
def test_dedups_to_one_document_per_hit_set() -> None:
|
def test_dedups_to_one_document_per_hit_set() -> None:
|
||||||
a = _doc("a.md", "A" * 50)
|
a = _doc("a.md", "A" * 50)
|
||||||
chunks = [_chunk(a, 0.2), _chunk(a, 0.7), _chunk(a, 0.5)]
|
chunks = [_chunk(a, 0.2), _chunk(a, 0.7), _chunk(a, 0.5)]
|
||||||
docs = select_documents(chunks, n=2, max_chars=10_000)
|
docs = select_documents(chunks, n=2)
|
||||||
assert len(docs) == 1
|
assert len(docs) == 1
|
||||||
assert docs[0] is a
|
assert docs[0] is a
|
||||||
|
|
||||||
@@ -62,40 +62,31 @@ def test_dedups_to_one_document_per_hit_set() -> None:
|
|||||||
def test_caps_at_n_documents() -> None:
|
def test_caps_at_n_documents() -> None:
|
||||||
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(4)]
|
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(4)]
|
||||||
chunks = [_chunk(d, 0.5 - 0.1 * i) for i, d in enumerate(docs_in)]
|
chunks = [_chunk(d, 0.5 - 0.1 * i) for i, d in enumerate(docs_in)]
|
||||||
out = select_documents(chunks, n=2, max_chars=10_000)
|
out = select_documents(chunks, n=2)
|
||||||
assert [d.path for d in out] == ["d0.md", "d1.md"]
|
assert [d.path for d in out] == ["d0.md", "d1.md"]
|
||||||
|
|
||||||
|
|
||||||
def test_combined_content_capped_with_truncation_marker() -> None:
|
def test_content_never_truncated_even_past_old_budget() -> None:
|
||||||
big = _doc("big.md", "B" * 100)
|
"""Whole documents, never truncated (A7 revised, owner permission 2026-08-24).
|
||||||
small = _doc("small.md", "S" * 100)
|
|
||||||
|
Two documents of 20 000 + 15 000 chars — 35 000 combined, well past
|
||||||
|
the old 24 000 context budget — come back with content
|
||||||
|
**byte-identical** to the originals, and the truncation marker is
|
||||||
|
absent from both.
|
||||||
|
"""
|
||||||
|
big = _doc("big.md", "B" * 20_000)
|
||||||
|
small = _doc("small.md", "S" * 15_000)
|
||||||
chunks = [_chunk(big, 0.9), _chunk(small, 0.6)]
|
chunks = [_chunk(big, 0.9), _chunk(small, 0.6)]
|
||||||
out = select_documents(chunks, n=2, max_chars=150)
|
out = select_documents(chunks, n=2)
|
||||||
# Best doc stays intact; the overflowing one is truncated in place.
|
assert [d.path for d in out] == ["big.md", "small.md"]
|
||||||
assert out[0].content == "B" * 100
|
assert out[0].content == "B" * 20_000
|
||||||
assert out[1].content.endswith(TRUNCATION_MARKER)
|
assert out[1].content == "S" * 15_000
|
||||||
assert out[1].content.startswith("S")
|
assert TRUNCATION_MARKER not in out[0].content
|
||||||
assert len(out[0].content) + len(out[1].content) <= 150
|
assert TRUNCATION_MARKER not in out[1].content
|
||||||
|
|
||||||
|
|
||||||
def test_single_doc_over_budget_is_truncated_to_budget() -> None:
|
|
||||||
big = _doc("big.md", "Z" * 200)
|
|
||||||
out = select_documents([_chunk(big, 0.9)], n=2, max_chars=50)
|
|
||||||
assert len(out[0].content) == 50
|
|
||||||
assert out[0].content.endswith(TRUNCATION_MARKER)
|
|
||||||
|
|
||||||
|
|
||||||
def test_under_budget_no_truncation() -> None:
|
|
||||||
a = _doc("a.md", "A" * 80)
|
|
||||||
b = _doc("b.md", "B" * 60)
|
|
||||||
out = select_documents([_chunk(b, 0.5), _chunk(a, 0.9)], n=2, max_chars=200)
|
|
||||||
assert [d.path for d in out] == ["a.md", "b.md"]
|
|
||||||
assert a.content == "A" * 80 and b.content == "B" * 60
|
|
||||||
assert TRUNCATION_MARKER not in a.content + b.content
|
|
||||||
|
|
||||||
|
|
||||||
def test_empty_hits_yield_no_documents() -> None:
|
def test_empty_hits_yield_no_documents() -> None:
|
||||||
assert select_documents([], n=2, max_chars=24_000) == []
|
assert select_documents([], n=2) == []
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
Reference in New Issue
Block a user