feat(auth): single-admin password login (signed cookie) — gate tuning + Sources catalog, keep chat and document viewer public
This commit is contained in:
Binary file not shown.
@@ -1,76 +0,0 @@
|
||||
# Phase 02 — Story: Import Documents
|
||||
|
||||
**Story:** `.agent/user_stories/import-documents.md`
|
||||
**Context:** `.agent/PLAN.md` §5 (data model), §9 (logging), §11 (import workflow)
|
||||
|
||||
## Goal
|
||||
The importer (`scripts/import_docs.py`) + `GET /api/docs` + the Sources page
|
||||
rendering the indexed documents — the knowledge base becomes refreshable.
|
||||
|
||||
## Implementation steps
|
||||
1. `app/rag/__init__.py`, `app/rag/chunker.py` — markdown-aware chunker
|
||||
(PLAN §5 policy: heading splits, 2000-char target, 200 overlap, keep
|
||||
nearest heading). Pure functions, fully unit-testable.
|
||||
2. `app/rag/llm.py` — `LLMClient` (openai async) with `embed(texts) ->
|
||||
list[list[float]]` (batched, `BOR_EMBED_BATCH_SIZE`) and a
|
||||
`embed_one`; dimension check vs `settings.embedding_dim` with a loud,
|
||||
actionable error. (Chat streaming is added in Phase 03 on this client.)
|
||||
3. `app/rag/importer.py` — the core: directory walk (exclusion list, PLAN
|
||||
A9; `*.md` only), sha256 delta vs `documents.content_hash`,
|
||||
upsert-or-skip, two-phase chunk replace (insert doc → replace chunks →
|
||||
embed → commit), `--prune` support, per-file + summary logging.
|
||||
4. `scripts/import_docs.py` — CLI wrapper (argparse): repeatable
|
||||
`--source` (default `~/Homelab` `~/Deployments`, `expanduser`),
|
||||
`--prune`, `--limit`.
|
||||
5. `app/api/docs.py` — `GET /api/docs` → `{"documents": [DocSummary]}`
|
||||
(include `chunks` count via `func.count`); mount in `app/main.py`
|
||||
**before** the static mount.
|
||||
6. `frontend/assets/sources.js` + `sources.html` polish — wire the real
|
||||
endpoint (already scaffolded to expect this shape); keep the empty state.
|
||||
7. Update `README.md` §Knowledge Base Import with the final commands +
|
||||
exclusion list + "update your docs → re-run the script" workflow.
|
||||
|
||||
## UI Verification
|
||||
Compare `/sources.html` against the story's "UI Visualization & Structure":
|
||||
stat cards `auto-fit minmax(170px,1fr)`; full-width table (≥85% container);
|
||||
mono path column with `title` ellipsis; empty state with the exact command;
|
||||
`<caption class="visually-hidden">`, `scope="col"`, scroll wrapper
|
||||
`role="region" tabindex="0"`. No CDN refs. Take a 1280px and 375px
|
||||
screenshot pass before finishing.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: chunker (heading splits, overlap, short-doc single chunk, code
|
||||
fences kept intact), exclusion walk (temp tree with `.venv` junk),
|
||||
delta logic (unchanged/changed/pruned via tmp Postgres or in-memory fakes
|
||||
— real DB preferred since compose runs locally).
|
||||
- Integration: `GET /api/docs` empty shape + populated shape; importer
|
||||
end-to-end against `tests/fixtures/docs/` into a test schema.
|
||||
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — **>90%**
|
||||
on `app/` (importer + chunker + client are the bulk; test them hard).
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite (DB must be up: `podman compose up -d db`):
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
|
||||
```
|
||||
|
||||
The test file implements the story's Playwright Mapping Rule (seed via the
|
||||
import function against `tests/fixtures/docs/` with the mock LLM; assert
|
||||
Sources page rows, layout width, and the empty state).
|
||||
|
||||
## Success criteria
|
||||
- [ ] `uv run python -m scripts.import_docs` (fixtures) imports all 3 docs,
|
||||
re-run reports `unchanged`
|
||||
- [ ] `GET /api/docs` + Sources page show the docs (real run: `~/Homelab`
|
||||
+ `~/Deployments` counts logged)
|
||||
- [ ] unit + integration green, coverage >90%
|
||||
- [ ] UI verification passed (screenshots attached to the phase record)
|
||||
- [ ] story E2E green in isolation
|
||||
- [ ] README import section updated
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(kb): markdown importer with sha256 deltas, chunking, batched embeddings, and Sources page"
|
||||
```
|
||||
@@ -1,74 +0,0 @@
|
||||
# Phase 03 — Story: Chat RAG Answer (happy path)
|
||||
|
||||
**Story:** `.agent/user_stories/chat-rag-answer.md`
|
||||
**Context:** `.agent/PLAN.md` §3 (data flow), §4 (SSE contract), §6 (persona), §9 (logging)
|
||||
|
||||
## Goal
|
||||
The core product loop: question → embed → cosine top-4 → full top-2
|
||||
documents → `turbo` (streamed) → chippy grounded answer with source chips.
|
||||
|
||||
## Implementation steps
|
||||
1. `app/rag/retriever.py` — `retrieve(db, question_embedding) ->
|
||||
list[RetrievedChunk]` (score = 1 − distance, `ORDER BY embedding <=> $1
|
||||
LIMIT BOR_TOP_K_CHUNKS`) + `select_documents(chunks, n) -> list[Document]`
|
||||
(distinct by `document_id`, ranked by best chunk score, cap content at
|
||||
`BOR_MAX_CONTEXT_CHARS` with `[…truncated…]`).
|
||||
2. `app/rag/prompts.py` — locked persona + HONESTY GATE prompt builder
|
||||
(PLAN §6 verbatim, `<relevance>HIGH|LOW</relevance>`, `<documents>`
|
||||
block; LOW mode includes the `DEFLECT_MODE` marker + weak-hit titles).
|
||||
3. `app/rag/llm.py` — add `chat_stream(messages) -> AsyncIterator[str]`
|
||||
(openai async, `stream=True`, `model=turbo`, temperature 0.4,
|
||||
max_tokens ~700).
|
||||
4. `app/api/chat.py` — `POST /api/chat` (ChatRequest) → `StreamingResponse`
|
||||
(SSE): emit `delta` events from the stream, then the `done` event
|
||||
(deflected, sources, suggestions); insert `query_log` row (deflected=
|
||||
false this phase); per-turn log line (PLAN §9); structured error events
|
||||
(`{"type":"error","detail":…}`) on LLM/DB failure.
|
||||
5. `frontend/assets/app.js` — replace the placeholder handler: `fetch` +
|
||||
`ReadableStream` SSE parser; render deltas live into a brain bubble
|
||||
(reuse the typing-indicator → streaming handoff); on `done`, append
|
||||
`.source-chip`s under the bubble; on error, show the banner (full
|
||||
state machine is Phase 06 — keep it simple-correct here).
|
||||
6. Tune `settings.suggestions` if the real Homelab import revealed better
|
||||
defaults (optional here; Phase 05 owns the chips).
|
||||
|
||||
## UI Verification
|
||||
Against the story's "UI Visualization & Structure": bubbles right/left
|
||||
(brand vs surface, ≥4.5:1 text), avatar 🧠, source chips mono/brand-soft
|
||||
with `source/path` and ellipsis, safe markdown (paste an answer containing
|
||||
`<script>alert(1)</script>` from the mock to prove it's escaped). Chat
|
||||
column 46rem centered. 1280px + 375px screenshot pass.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: retriever ordering/dedup/cap (fake rows), prompt builder (HIGH
|
||||
contains documents + `HIGH`, LOW contains `DEFLECT_MODE` + titles only,
|
||||
persona rules present verbatim), SSE event serialization.
|
||||
- Integration: `/api/chat` against the mock LLM with a seeded temp schema —
|
||||
assert SSE delta sequence, `done` payload (sources non-empty,
|
||||
deflected false), `query_log` row, error event when LLM unreachable.
|
||||
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — **>90%**.
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: streamed grounded answer + `kubernetes.md`
|
||||
source chip + button recovery; DB `query_log` assertion; raw SSE shape
|
||||
check via `httpx`.
|
||||
|
||||
## Success criteria
|
||||
- [ ] end-to-end: question → streamed chippy answer citing `kubernetes.md`
|
||||
- [ ] `query_log` row per turn; per-turn log line in stdout
|
||||
- [ ] LLM-down path shows error banner, no stuck button
|
||||
- [ ] unit + integration green, coverage >90%
|
||||
- [ ] UI verification passed
|
||||
- [ ] story E2E green in isolation
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(rag): stream grounded chat answers via pgvector cosine retrieval with source citations"
|
||||
```
|
||||
@@ -1,66 +0,0 @@
|
||||
# Phase 04 — Story: Honest Deflection
|
||||
|
||||
**Story:** `.agent/user_stories/honest-deflection.md`
|
||||
**Context:** `.agent/PLAN.md` §4, §6 (honesty gate), §9
|
||||
|
||||
## Goal
|
||||
When retrieval finds nothing relevant, Brain says so — plainly, chippily —
|
||||
and offers real alternatives. No hallucinated confidence.
|
||||
|
||||
## Implementation steps
|
||||
1. `app/api/chat.py` — apply the gate: `best_score < settings.relevance_
|
||||
threshold` ⇒ build LOW prompt (`DEFLECT_MODE`, weak-hit titles only),
|
||||
else HIGH prompt. Set `deflected` on the `done` event + `query_log`.
|
||||
2. Deflection `suggestions[]`: ask `turbo` (same stream) to include 2–3
|
||||
alternative questions; simplest robust approach — have the LLM emit them
|
||||
inline in the answer AND have the server derive 2–3 chips from the
|
||||
weak-hit document titles (deterministic fallback if the model doesn't
|
||||
produce a parsable list). Ship the deterministic title-derived chips as
|
||||
the v1 behavior; model-generated list is a bonus if trivially parseable.
|
||||
3. `frontend/assets/app.js` — on `done.deflected`: add `.is-deflected`
|
||||
class to the bubble, render "Maybe try:" chips below it (same
|
||||
`.suggestion-chip` component; clicking fills the input — full submit
|
||||
behavior lands with Phase 05's chip component; wire what exists).
|
||||
4. `README.md` — document `BOR_RELEVANCE_THRESHOLD` tuning + the
|
||||
deflection behavior in Troubleshooting.
|
||||
|
||||
## UI Verification
|
||||
Against the story: amber bubble (`#fff7e8` bg / `#f59e0b` border) distinct
|
||||
from normal answers; "Maybe try:" chips ≥44px, brand-soft/brand-ink;
|
||||
contrast pairs verified (ink on accent-bg ≥ 9:1, accent-ink ≥ 8:1);
|
||||
chip group has an accessible name; mobile wraps cleanly.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: gate boundary with a fake retriever — score exactly 0.30 → HIGH;
|
||||
0.2999 → LOW; LOW prompt contains `DEFLECT_MODE` + titles, no full docs;
|
||||
HIGH unaffected. Suggestions derivation (2–3, non-empty, derived from
|
||||
titles).
|
||||
- Integration: mock LLM — off-topic question ("sourdough") ⇒ `done`
|
||||
`deflected: true`, `query_log.deflected=true`, weak `top_score` stored;
|
||||
on-topic question ⇒ `deflected: false`.
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: off-topic question ⇒ `.is-deflected` bubble
|
||||
matching /haven't done anything like that/i + ≥2 "Maybe try:" chips; chip
|
||||
click behavior; (unit boundary test lives in pytest, not here).
|
||||
|
||||
## Success criteria
|
||||
- [ ] off-topic question never gets a confident fake answer
|
||||
- [ ] deflected bubble visually distinct + alternative chips render
|
||||
- [ ] `query_log.deflected` accurate; threshold env-tunable
|
||||
- [ ] unit + integration green, coverage >90%
|
||||
- [ ] UI verification passed
|
||||
- [ ] story E2E green in isolation
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(rag): honest deflection gate with amber UI state and alternative-question chips"
|
||||
```
|
||||
@@ -1,59 +0,0 @@
|
||||
# Phase 05 — Story: Suggestion Chips
|
||||
|
||||
**Story:** `.agent/user_stories/suggestion-chips.md`
|
||||
**Context:** `.agent/PLAN.md` §7 (UI/UX), story file for chip spec
|
||||
|
||||
## Goal
|
||||
Zero-friction onboarding: 3–4 real example questions on first load,
|
||||
clickable → filled → submitted, keyboard-first, mobile-scrollable.
|
||||
|
||||
## Implementation steps
|
||||
1. `app/config.py` — confirm `suggestions` is env-overridable
|
||||
(`BOR_SUGGESTIONS` as JSON list via pydantic-settings) and tune the
|
||||
defaults against the *actually imported* Homelab/Deployments topics
|
||||
(read a sample of `documents` titles; pick questions real answers
|
||||
exist for).
|
||||
2. `app.js` — extract a `renderChips(container, items, {onSelect})` helper;
|
||||
real `<button type="button" class="suggestion-chip" role="listitem">`
|
||||
inside `#suggestions[role="list"]`; onboarding `onSelect` = fill
|
||||
`#message-input` + focus + `composer.requestSubmit()`. Reuse the same
|
||||
helper for deflection chips (Phase 04) with the same submit behavior.
|
||||
3. Empty-state lifecycle: first user message hides `#empty-state` (already
|
||||
done in `addMessage`) — verify chips don't linger in the conversation.
|
||||
4. Mobile CSS check: chip row `nowrap + overflow-x auto` at ≤640px (tokens
|
||||
already exist — verify, don't duplicate).
|
||||
|
||||
## UI Verification
|
||||
Against the story: pills 999px radius, ≥44px, brand-soft/brand-ink (≥6:1),
|
||||
hover/active states; desktop centered wrap vs mobile single scroll row;
|
||||
Tab order reaches chips before the composer input is required; screen
|
||||
reader: group labeled "Suggested questions". Screenshot pass 1280px + 375px.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: `GET /api/suggestions` honors `BOR_SUGGESTIONS` env
|
||||
override (JSON list); default list has ≥3 non-empty strings.
|
||||
- Coverage: **>90%** on `app/` (JS is covered by E2E).
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: chips render (≥3, role=list); chip click
|
||||
submits (user bubble with exact chip text + mock reply); keyboard Tab+Enter
|
||||
activates; 375px chip row is a horizontal scroll row.
|
||||
|
||||
## Success criteria
|
||||
- [ ] onboarding chips render from the API; click = one-tap question
|
||||
- [ ] keyboard + SR usable; mobile scroll row
|
||||
- [ ] deflection chips share the component + submit behavior
|
||||
- [ ] unit + integration green, coverage >90%
|
||||
- [ ] story E2E green in isolation
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(ui): onboarding suggestion chips with one-tap submit, keyboard access, and mobile scroll row"
|
||||
```
|
||||
@@ -1,64 +0,0 @@
|
||||
# Phase 06 — Story: Loading Feedback & Progress
|
||||
|
||||
**Story:** `.agent/user_stories/loading-feedback.md`
|
||||
**Context:** `.agent/PLAN.md` §7.4 ("never stale" contract), §9
|
||||
|
||||
## Goal
|
||||
An unambiguous state machine — `idle → thinking → streaming → done |
|
||||
error → idle` — so the user always knows what's happening, and a stale
|
||||
Send button is impossible.
|
||||
|
||||
## Implementation steps
|
||||
1. `app.js` — formalize the state machine (single `setUiState(state)`
|
||||
function driving: typing indicator, send button disabled/spinner/label,
|
||||
`#send-status` live text). Replace ad-hoc busy handling from Phase 03.
|
||||
2. Pre-token: typing indicator (`role="status"`,
|
||||
`aria-label="Brain of Reese is thinking"`); after 10s pre-token, update
|
||||
the label with elapsed seconds (setInterval, cleared on state change).
|
||||
3. Streaming: first `delta` removes the typing indicator and starts
|
||||
appending to the answer bubble; button stays busy.
|
||||
4. Error paths: `{"type":"error"}` SSE event, non-2xx response, or
|
||||
**120s client-side timeout** (clear on first delta) → red banner
|
||||
`role="alert"` ("Try again — if this persists, check the LLM is
|
||||
reachable") + state → idle.
|
||||
5. `prefers-reduced-motion`: CSS already slows animations — verify; add a
|
||||
static fallback for the dots if needed.
|
||||
6. Server: confirm the per-turn log line includes `embed_ms` and
|
||||
`total_ms` (add if Phase 03 omitted it).
|
||||
|
||||
## UI Verification
|
||||
Walk the full state machine by hand (dev server + mock LLM slow path):
|
||||
submit → indicator + "Thinking…" disabled button → live tokens → done
|
||||
(enabled, focused input). Kill the mock mid-stream → banner + recovery.
|
||||
Contrast of disabled button + spinner OK; reduced-motion pass.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: SSE error event serialization; timeout constant
|
||||
exported/testable; (JS logic is E2E-covered).
|
||||
- Coverage: **>90%** on `app/`.
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_loading_feedback.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping (mock LLM's 3s "pretend to think slowly"
|
||||
warm-up + a fixture that stops the mock): typing indicator visible during
|
||||
pre-token and gone by answer; button disabled→"Thinking…"→enabled "Send";
|
||||
streaming appends (two-timestamp length check); LLM-down ⇒ `role=alert`
|
||||
banner + button recovered.
|
||||
|
||||
## Success criteria
|
||||
- [ ] every in-flight state has a visible indicator; button never zombies
|
||||
- [ ] error + 120s timeout paths both recover cleanly
|
||||
- [ ] reduced-motion respected
|
||||
- [ ] unit + integration green, coverage >90%
|
||||
- [ ] story E2E green in isolation
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(ui): explicit chat state machine — typing indicator, streaming progress, timeout and error recovery"
|
||||
```
|
||||
@@ -1,57 +0,0 @@
|
||||
# Phase 07 — Story: Responsive, Polished, Accessible UI
|
||||
|
||||
**Story:** `.agent/user_stories/responsive-polish.md`
|
||||
**Context:** `.agent/PLAN.md` §7 (the whole UI/UX strategy)
|
||||
|
||||
## Goal
|
||||
The final visual + accessibility audit pass across chat and Sources. No
|
||||
new features — enforce PLAN §7 end-to-end and fix every deviation.
|
||||
|
||||
## Implementation steps
|
||||
1. Viewport sweep (360 / 375 / 768 / 1280 / 1600) on both pages: fix
|
||||
overflow, pinched columns, dead whitespace. Chat column stays ≤46rem
|
||||
centered; Sources table full-width with horizontal scroll <640px.
|
||||
2. A11y sweep (both pages): landmarks, skip link, labels on every input,
|
||||
`aria-label` on every icon-only control, `:focus-visible` outline on
|
||||
every focusable, `aria-live` regions intact, no contrast <4.5:1
|
||||
(compute, don't eyeball — use the E2E helper).
|
||||
3. Reduced-motion + long-content pass (60-char paths, long answers).
|
||||
4. No-CDN re-verification on **both** pages (extend the integration test
|
||||
to `/sources.html` if it only covers `/`).
|
||||
5. Final README polish pass: screenshots section (optional), quickstart
|
||||
sanity, "Update your documents" workflow prominent.
|
||||
|
||||
## UI Verification
|
||||
This phase IS the verification: the E2E below is the acceptance test.
|
||||
Additionally, manual screenshot pass at 1280px + 375px for both pages,
|
||||
reviewed against PLAN §7.1–7.4 before committing.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: no-CDN check extended to Sources page; health unchanged.
|
||||
- Coverage: **>90%** on `app/` (final state of the whole app).
|
||||
- Whole suite green: `uv run pytest` (unit+integration) — the entire
|
||||
repo must be green at this phase.
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_responsive_polish.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: no horizontal overflow at 5 viewports (both
|
||||
pages); chat column capped + centered at 1600px; Sources table ≥80%
|
||||
container at 1280px; landmarks/labels/skip-link sweep; WCAG contrast
|
||||
pairs ≥4.5:1 (computed); reduced-motion honored.
|
||||
|
||||
## Success criteria
|
||||
- [ ] all six mapping tests pass at every viewport
|
||||
- [ ] zero known a11y deviations against PLAN §7.2
|
||||
- [ ] whole pytest suite green + coverage >90%
|
||||
- [ ] README polished
|
||||
- [ ] committed (this commit marks v1.0 feature-complete)
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(ui): responsive + WCAG AA polish pass across chat and sources — v1 feature complete"
|
||||
```
|
||||
@@ -1,164 +0,0 @@
|
||||
# Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
|
||||
|
||||
**Story:** `.agent/user_stories/retrieval-quality.md`
|
||||
**Context:** `.agent/PLAN.md` §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)
|
||||
|
||||
## Goal
|
||||
Fix "RAG retrieval is terrible": ingest the full text-format set (not
|
||||
just `.md`), purge vendored-cache junk from the index, and replace
|
||||
pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused)
|
||||
retrieval so name-your-tool questions find the right document.
|
||||
|
||||
## Owner permission (recorded per phase protocol)
|
||||
> "I'm giving you explicit permission to update the locked decisions and
|
||||
> proceed with writing all 3 of these phases" — Reese, 2026-08-21.
|
||||
|
||||
This phase revises anchors **A9** (content scope: `*.md` only →
|
||||
`md, markdown, txt, yaml, yml, json, py` + hidden-dir skip), **A7**
|
||||
(pure-cosine top-4 → hybrid RRF retrieval; the whole-document context
|
||||
contract is preserved), **A8** (gate: LOW only when best cosine <
|
||||
threshold **and** zero FTS hits; threshold re-tuned 0.30 → 0.62 default).
|
||||
`PLAN.md` anchors were updated 2026-08-21 under this permission.
|
||||
|
||||
## Evidence (measured 2026-08-21 against the live KB + `embed` model)
|
||||
- "How did I install gitlab?": the best `gitlab.md` chunk ranks **7th**
|
||||
(cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache
|
||||
README (`.esphome/.espressif/…/esp-tflite-micro/README.md`, 0.838) and
|
||||
generic templates (`project_readme_template.md`, `templates/…/foobar.
|
||||
md`, 0.81–0.82). The LLM therefore answered from junk docs and honestly
|
||||
reported "no notes on gitlab".
|
||||
- Corpus cosine range: **0.41–0.84** — the old 0.30 gate never
|
||||
discriminated.
|
||||
- FTS: `plainto_tsquery('english','gitlab')` matches **exactly**
|
||||
gitlab.md's 4 chunks and nothing else.
|
||||
- ~470 of 672 indexed docs live under dot-prefixed path components
|
||||
(vendored caches) that A9's exclusion list doesn't cover.
|
||||
|
||||
## Dependencies
|
||||
Phases 01–07 (02 importer, 03 retriever, 04 gate especially).
|
||||
Independent of 08 (backend + E2E only). Phase 10 builds on the new
|
||||
multi-format corpus.
|
||||
|
||||
## Implementation steps
|
||||
1. **Config** (`app/config.py`): `import_extensions` (csv, default
|
||||
`md,markdown,txt,yaml,yml,json,py`; env `BOR_IMPORT_EXTENSIONS`),
|
||||
`hybrid_vector_candidates` (30, `BOR_HYBRID_VECTOR_CANDIDATES`),
|
||||
`hybrid_lexical_candidates` (30, `BOR_HYBRID_LEXICAL_CANDIDATES`),
|
||||
`rrf_k` (60, `BOR_RRF_K`), `relevance_threshold` default **0.62**
|
||||
(re-tuned; env override stays). In `tests/e2e/conftest.py`'s
|
||||
`app_server` fixture set `BOR_RELEVANCE_THRESHOLD=0.30` — the mock's
|
||||
token-overlap embeddings need their own calibration; this keeps
|
||||
stories 02–07's E2E suites green.
|
||||
2. **Chunker** (`app/rag/chunker.py`): add a `chunk_document(content,
|
||||
path)` dispatcher by lowercased suffix + per-format functions —
|
||||
**stdlib only, no new dependencies**:
|
||||
- `yaml`/`yml`: split on `---` document separators and top-level keys
|
||||
(indent-0 `key:` lines); every chunk keeps its key line as anchor.
|
||||
- `json`: `json.dumps(obj, indent=2)` then split at top-level keys
|
||||
(track brace depth); unparseable JSON → paragraph packing.
|
||||
- `py`: stdlib `ast` top-level node line ranges → split at
|
||||
defs/classes; oversized functions fall back to line packing.
|
||||
- `txt`: paragraph packing (reuse `_paragraph_blocks`/`_pack_blocks`).
|
||||
- All formats honor `HARD_MAX_CHARS` (1200 — the aipi ~1024-token
|
||||
request cap) and the target/overlap settings; the `md` path stays
|
||||
byte-for-byte unchanged (existing chunker tests must stay green).
|
||||
3. **Importer** (`app/rag/importer.py`, `scripts/import_docs.py`):
|
||||
extension filter (case-insensitive, config-driven); **skip any path
|
||||
containing a dot-prefixed component** (hidden dirs); chunker dispatch
|
||||
by suffix; `--prune` now also drops docs whose files **no longer
|
||||
match the filter** (this is how the ~470 junk docs leave the index);
|
||||
summary log gains per-format counts
|
||||
(`formats=md:203,yaml:267,…`).
|
||||
4. **Migration `0002_hybrid_retrieval.py`** (alembic):
|
||||
- `chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english',
|
||||
content)) STORED` + `CREATE INDEX … USING gin (chunks.tsv)`.
|
||||
- `query_log.fts_hits INT` (nullable; pre-existing rows stay NULL).
|
||||
5. **Retriever** (`app/rag/retriever.py`) — hybrid path:
|
||||
- `retrieve(db, question, question_embedding)`: vector top-N (cosine,
|
||||
as today) ∪ lexical top-N — `to_tsquery('english', <OR-joined
|
||||
stemmed tokens of the question>)` (skip pure-stopword/no-token
|
||||
questions → empty lexical list), ordered by `ts_rank` — fused with
|
||||
RRF: `score = Σ 1/(k + rank)` over the lists a chunk appears in
|
||||
(single-list chunks get one term; k from config).
|
||||
- `RetrievedChunk` gains `cosine` (for the gate) and `fts_hit: bool`
|
||||
alongside `score` (now the fused score, used for ranking);
|
||||
`select_documents` / `weak_hit_titles` keep working off `score`.
|
||||
- Deterministic tie-break: `(−fused, −cosine, document.path,
|
||||
chunk.position)`.
|
||||
6. **Chat flow + gate** (`app/api/chat.py`): pass the raw question into
|
||||
`retrieve`; **LOW only when `best_cosine < threshold and fts_hits ==
|
||||
0`** (`fts_hits` = count of lexical candidates matched); per-turn log
|
||||
line gains `fts_hits=…` (PLAN §9); `query_log` row stores `fts_hits`.
|
||||
7. **Eval script** `scripts/eval_retrieval.py`:
|
||||
`uv run python -m scripts.eval_retrieval "q1" "q2" …` (or
|
||||
`--from-file questions.txt`) — embeds via aipi, runs the hybrid
|
||||
search, prints top-5 docs per question with cosine/fts/fused scores +
|
||||
the gate verdict. Requires `AIPI_KEY` in the environment (same
|
||||
convention as `llm_probe.py`).
|
||||
8. **Re-import the live KB** (one-time; expect ~15–40 min of embedding
|
||||
batches — the importer logs per file):
|
||||
`uv run python -m scripts.import_docs --prune`. Expect ~470
|
||||
hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then
|
||||
verify with the eval script:
|
||||
- "How did I install gitlab?" → top doc `active/container_gitlab/
|
||||
gitlab.md` (the compose yaml should land in the top-2).
|
||||
- "How is my Kubernetes cluster set up?" → kubernetes docs.
|
||||
- "sourdough starter" → LOW (deflect).
|
||||
If the gitlab case isn't #1, iterate the **fusion** (k, candidate
|
||||
counts, token handling) — not the threshold — until it is, and record
|
||||
the final numbers in the phase report.
|
||||
9. **E2E fixtures** (`tests/fixtures/docs/`): add
|
||||
`homelab/container_gitlab/gitlab.md` (H1 "Gitlab", docker install
|
||||
steps, "gitlab" repeated), `homelab/container_gitlab/
|
||||
gitlab-compose.yaml` (`services: gitlab: …`), a `.py` note, a `.json`
|
||||
note, a `.txt` note, and `.hidden/junk.md` (must never be imported).
|
||||
Follow the existing in-process seeding pattern from
|
||||
`tests/e2e/test_import_documents.py`.
|
||||
10. **README**: import workflow section — supported formats, hidden-dir
|
||||
skip, `scripts/eval_retrieval.py`, threshold tuning; note that the
|
||||
Sources count drops after the prune (intended cleanup).
|
||||
|
||||
## Testing & Quality
|
||||
- **Unit:** chunker per format (yaml top-level + `---` split, json
|
||||
top-level keys + pretty-print + unparseable fallback, py ast split +
|
||||
oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md
|
||||
output unchanged; importer (hidden-dir skip, extension filter,
|
||||
prune-when-filtered-out, per-format summary); retriever (RRF math:
|
||||
both-lists / one-list / tie-break; OR tsquery construction incl.
|
||||
no-token and stopword-only questions; gate: `cosine ≥ T` → HIGH;
|
||||
`cosine < T` + `fts>0` → HIGH; `cosine < T` + `fts=0` → LOW; boundary
|
||||
exactly `T` → HIGH).
|
||||
- **Integration:** `/api/chat` hybrid against a seeded temp schema —
|
||||
keyword question grounded + `fts_hits` in `query_log`; off-topic
|
||||
deflected with `fts_hits=0`; migration up clean.
|
||||
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
|
||||
**>90%** on `app/`.
|
||||
- **No regressions:** existing story E2E suites (02–07) green in
|
||||
isolation after the change (the conftest threshold override is what
|
||||
keeps them green — verify each one).
|
||||
|
||||
## Playwright Execution Phase
|
||||
Run ONLY this story's suite:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov
|
||||
```
|
||||
|
||||
Implements the story mapping: multi-format fixture import (hidden doc
|
||||
excluded, `/api/docs` counts); "How did I install gitlab?" → grounded,
|
||||
not deflected, gitlab chip, `query_log` row; keyword-only question beats
|
||||
vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble +
|
||||
≥2 chips.
|
||||
|
||||
## Success criteria
|
||||
- [ ] live eval: "How did I install gitlab?" → `gitlab.md` is the top doc
|
||||
- [ ] zero dot-prefixed path components in `documents` after re-import
|
||||
- [ ] off-topic still deflects; on-topic still grounds (new + existing E2E)
|
||||
- [ ] unit + integration green, coverage >90%, ruff + pyright green
|
||||
- [ ] README documents formats / hidden-dir skip / eval / tuning
|
||||
- [ ] committed
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"
|
||||
```
|
||||
@@ -1,52 +0,0 @@
|
||||
# Phase 11 — Long Answers (No Truncation)
|
||||
|
||||
**Story:** `.agent/user_stories/long-answers.md`
|
||||
**Context:** owner report 2026-08-22 — "responses keep getting cut off.
|
||||
It should be allowed to respond up to 32768 tokens."
|
||||
|
||||
## Goal
|
||||
Remove the hard 700-token output cap on chat answers; the model may
|
||||
respond up to **32 768** tokens (`BOR_MAX_OUTPUT_TOKENS`, default
|
||||
32 768).
|
||||
|
||||
## Diagnosis
|
||||
`app/rag/llm.py::chat_stream` calls `chat.completions.create(...,
|
||||
max_tokens=700, ...)`. Long answers die mid-sentence at ~700 tokens.
|
||||
|
||||
## Implementation steps
|
||||
1. **Config** (`app/config.py`): `max_output_tokens: int = 32_768` in the
|
||||
RAG-tuning section (env `BOR_MAX_OUTPUT_TOKENS`); document in
|
||||
`.env.example`.
|
||||
2. **LLM client** (`app/rag/llm.py`): `chat_stream` passes
|
||||
`max_tokens=self.settings.max_output_tokens`.
|
||||
3. **E2E mock** (`tests/e2e/mock_llm.py`):
|
||||
- Honor `max_tokens` deterministically: token ≈ whitespace word; if
|
||||
the composed answer is longer, truncate to the first N words.
|
||||
(With the old 700 cap a long answer loses its tail — the mock now
|
||||
behaves like the real endpoint.)
|
||||
- New trigger: user message containing `write a long answer` →
|
||||
deterministic ~4 000-word numbered answer ending in a unique final
|
||||
line (`LONG-ANSWER-END`).
|
||||
- No behavior change for existing (short) answers: they fit under any
|
||||
sane cap.
|
||||
4. **Tests:**
|
||||
- Unit: settings default + env override (`test_config.py`);
|
||||
`chat_stream` forwards the configured `max_tokens` (fake client in
|
||||
`test_llm_client.py`).
|
||||
- E2E: `tests/e2e/test_long_answers.py` per the story mapping.
|
||||
|
||||
## Locked decisions
|
||||
None touched. A5 (aipi endpoint) unchanged; `turbo` accepts the larger
|
||||
cap per owner instruction.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit + integration green; `uv run pytest --cov=app --cov-report=term-missing`
|
||||
**>90%**; E2E in isolation:
|
||||
`uv run pytest tests/e2e/test_long_answers.py -v --no-cov`.
|
||||
- No regressions: `test_chat_rag.py` + `test_chat_api.py` green
|
||||
(short answers unaffected by the mock's new `max_tokens` honoring).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add -A .agent/ app/ tests/ .env.example && git commit --no-gpg-sign -m "fix(rag): lift chat output cap to 32768 tokens — long answers no longer cut off"
|
||||
```
|
||||
Reference in New Issue
Block a user