Compare commits
13
Commits
6bf7f456d4
...
281f3555c3
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
281f3555c3 | ||
|
|
914097abcf | ||
|
|
c564e317ed | ||
|
|
9518d9d5d1 | ||
|
|
32b7bfd4b3 | ||
|
|
ea8e041189 | ||
|
|
aba8615177 | ||
|
|
820753948e | ||
|
|
619bf2187a | ||
|
|
114b115034 | ||
|
|
ece93a7c8f | ||
|
|
6832957ab0 | ||
|
|
1a60ecbd8b |
-521
@@ -1,521 +0,0 @@
|
|||||||
# Brain of Reese — Master Plan
|
|
||||||
|
|
||||||
> **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
|
|
||||||
> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
|
|
||||||
> Anchors table are settled — do not re-litigate them in a phase.
|
|
||||||
> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
|
|
||||||
> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
|
|
||||||
> theme (Phase 08); clickable document viewer (Phase 10); thinking display
|
|
||||||
> (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll
|
|
||||||
> (Phase 18, owner choice 2026-08-23); shared header (Phase 19);
|
|
||||||
> whole-document context (Phase 24 — A7's 24k context cap removed,
|
|
||||||
> owner permission 2026-08-24). See roadmap §12.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Mission
|
|
||||||
|
|
||||||
A **knowledge base chatbot** that embeds the `~/Homelab` and `~/Deployments`
|
|
||||||
projects into a Postgres vector database and lets anyone ask *Reese* (the
|
|
||||||
bot) questions about them.
|
|
||||||
|
|
||||||
**Product feel:** a chippy, upbeat assistant that is optimistic about the
|
|
||||||
user's ability ("you've got this") and **radically honest** — if retrieval
|
|
||||||
didn't surface anything relevant it says *"I haven't done anything like
|
|
||||||
that"* and offers alternatives instead of hallucinating.
|
|
||||||
|
|
||||||
### In scope (v1)
|
|
||||||
- Chat UI (mobile-friendly, well-styled, no auth, no CDN).
|
|
||||||
- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
|
|
||||||
by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
|
|
||||||
(and any future directory the importer is pointed at).
|
|
||||||
- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
|
|
||||||
`embed` (embeddings, **768 dims — verified**).
|
|
||||||
- Postgres 17 + pgvector, cosine similarity, chunk→document mapping so the
|
|
||||||
LLM receives the **entire relevant document** as context.
|
|
||||||
- Idempotent import/update script, documented in the README.
|
|
||||||
- Ample server logging + explicit UI loading/progress feedback (never a
|
|
||||||
stale submit button).
|
|
||||||
|
|
||||||
### Out of scope (v1)
|
|
||||||
- Auth / multi-user (API is stateless under `/api` so it can be added later).
|
|
||||||
- Binary / non-text content, file uploads, caching layer, message persistence.
|
|
||||||
- Real-time document watching (manual re-import for now).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Architectural Anchors (LOCKED DECISIONS)
|
|
||||||
|
|
||||||
| # | Component | Decision | Rationale | Status |
|
|
||||||
|---|-----------|----------|-----------|--------|
|
|
||||||
| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
|
|
||||||
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
|
|
||||||
| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
|
|
||||||
| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
|
|
||||||
| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
|
|
||||||
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
|
|
||||||
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
|
|
||||||
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
|
|
||||||
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) |
|
|
||||||
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
|
|
||||||
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
|
|
||||||
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
|
|
||||||
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3 | Standard, reversible, reviewable schema history | LOCKED |
|
|
||||||
| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
|
|
||||||
| A15 | Chat transport | **SSE streaming** from `POST /api/chat` (deltas + final `done` event with metadata) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED |
|
|
||||||
| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
|
|
||||||
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
|
|
||||||
|
|
||||||
> **A10 revision (phase 16, owner permission 2026-08-22):** single-admin
|
|
||||||
> signed-cookie auth — public: chat / documents / suggestions / health;
|
|
||||||
> admin-only: docs catalog + steering. The row above keeps the original v1
|
|
||||||
> decision text; the public API surface stays stateless (the signed
|
|
||||||
> session cookie is the only session state) — recorded as a revision,
|
|
||||||
> not a silent deviation.
|
|
||||||
>
|
|
||||||
> **A10 UI revision (phase 19, owner permission 2026-08-23):** the
|
|
||||||
> "Sources" nav link is hidden from anonymous users on all pages — the
|
|
||||||
> soft-gate page and the API split above are unchanged.
|
|
||||||
>
|
|
||||||
> **A7 revision (phase 24, owner permission 2026-08-24):** the
|
|
||||||
> `[…truncated…]` cap on document context is removed —
|
|
||||||
> `select_documents` always returns the full top-N texts;
|
|
||||||
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
|
|
||||||
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
|
|
||||||
> marker.
|
|
||||||
>
|
|
||||||
> **A9 revision (phase 47, owner permission 2026-08-27):** the format
|
|
||||||
> set extends with the Podman quadlet family (`container, network,
|
|
||||||
> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja
|
|
||||||
> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md`
|
|
||||||
> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the
|
|
||||||
> hidden-dir/exclusion invariants are unchanged.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. High-Level Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
┌────────────────────────────────────────────┐
|
|
||||||
│ Podman Compose │
|
|
||||||
Browser │ ┌──────────────────────────────────────┐ │
|
|
||||||
┌──────────┐ HTTP │ │ brain-of-reese/app (FastAPI) │ │
|
|
||||||
│ index.html│◄──────┼─►│ • static frontend (no CDN) │ │
|
|
||||||
│ app.js │ SSE │ │ • /api/chat /api/suggestions │ │
|
|
||||||
└──────────┘ │ │ • /api/health /api/docs │ │
|
|
||||||
│ │ • RAG pipeline (embed→retrieve→gen) │ │
|
|
||||||
│ └──────┬──────────────────┬───────────┘ │
|
|
||||||
│ │ SQL (psycopg) │ OpenAI-compat│
|
|
||||||
│ ┌──────▼──────┐ ┌───────▼────────────┐ │
|
|
||||||
│ │ db: │ └─────────┬──────────┘ │
|
|
||||||
│ │ postgres:17 │ │ │
|
|
||||||
│ │ + pgvector │ │ │
|
|
||||||
│ └─────────────┘ │ │
|
|
||||||
└──────────────────────────────┼────────────┘
|
|
||||||
▼
|
|
||||||
https://aipi.reeseapps.com/v1
|
|
||||||
(self-hosted: turbo, embed)
|
|
||||||
|
|
||||||
Offline tooling (same repo, same venv):
|
|
||||||
scripts/import_docs.py → walks A9-format dirs, chunks, embeds, upserts
|
|
||||||
scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
|
|
||||||
scripts/llm_probe.py → verifies models + embedding dim
|
|
||||||
```
|
|
||||||
|
|
||||||
### Component breakdown
|
|
||||||
| Component | Responsibility | Lives in |
|
|
||||||
|-----------|----------------|----------|
|
|
||||||
| **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
|
|
||||||
| **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
|
|
||||||
| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
|
|
||||||
| **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
|
|
||||||
| **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
|
|
||||||
|
|
||||||
### Chat data flow
|
|
||||||
```
|
|
||||||
user question
|
|
||||||
→ POST /api/chat {message}
|
|
||||||
→ embed(question) [aipi /v1/embeddings, model=embed]
|
|
||||||
→ cosine top-30 + FTS top-30 (OR tsquery, ts_rank) [pgvector + PG FTS]
|
|
||||||
→ RRF fuse (k=60) → docs ranked by best fused chunk score
|
|
||||||
├─ best cosine >= 0.62 OR fts_hits > 0 → top-2 documents' FULL content
|
|
||||||
│ → system prompt (persona + HONESTY rules + docs)
|
|
||||||
│ → turbo, stream=True → SSE deltas
|
|
||||||
└─ else → DEFLECT_MODE system prompt (weak hits as topics)
|
|
||||||
→ turbo, stream=True → SSE deltas (honest reply)
|
|
||||||
→ query_log row (question, score, deflected, sources, latency)
|
|
||||||
→ final SSE "done" event: {deflected, sources[], suggestions[]}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. API Design
|
|
||||||
|
|
||||||
All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
|
|
||||||
|
|
||||||
| Method | Path | Purpose | Story |
|
|
||||||
|--------|------|---------|-------|
|
|
||||||
| GET | `/api/health` | Liveness + db up/down + version | 01 |
|
|
||||||
| GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
|
|
||||||
| GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
|
|
||||||
| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
|
|
||||||
| POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
|
|
||||||
|
|
||||||
### SSE contract (`POST /api/chat`)
|
|
||||||
```
|
|
||||||
data: {"type":"thinking","text":"…"}\n\n
|
|
||||||
data: {"type":"thinking","text":"…"}\n\n
|
|
||||||
data: {"type":"delta","text":"Hey! "}\n\n
|
|
||||||
data: {"type":"delta","text":"Good "}\n\n
|
|
||||||
...
|
|
||||||
data: {"type":"done","deflected":false,"sources":[{"source":"Homelab","path":"kubernetes.md","title":"Kubernetes Homelab Cluster"}],"suggestions":[]}\n\n
|
|
||||||
```
|
|
||||||
Client rules: render deltas as they arrive; render `thinking` text in a
|
|
||||||
collapsible block above the answer; auto-collapse on the first `delta`;
|
|
||||||
tolerate interleaved `thinking` events (append — never reopen once the
|
|
||||||
answer started); the `done` shape is unchanged (thinking never travels on
|
|
||||||
`done`); on `done` append source chips / suggestion chips and clear the
|
|
||||||
busy state; on HTTP/stream error show the error banner + retry (never a
|
|
||||||
stuck button).
|
|
||||||
|
|
||||||
> **SSE revision (phase 17, owner permission 2026-08-23):** the contract
|
|
||||||
> gains one event type — `{"type":"thinking","text":"…"}` — carrying the
|
|
||||||
> model's reasoning ahead of the `delta` events (the `turbo` model emits
|
|
||||||
> `delta.reasoning_content` chunks before the first content chunk, verified
|
|
||||||
> live 2026-08-23; `BOR_STREAM_THINKING=0` suppresses the frames
|
|
||||||
> server-side). `delta` and `done` shapes are unchanged — a recorded
|
|
||||||
> extension of A15, not a silent deviation.
|
|
||||||
>
|
|
||||||
> **SSE revision (phase 37, owner permission 2026-08-26):** the contract
|
|
||||||
> gains a second event type — `{"type":"tool","name":"…","argument":…}` —
|
|
||||||
> carrying the model's document tool calls on grounded turns (phase 37:
|
|
||||||
> `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS`
|
|
||||||
> / `BOR_AGENT_READ_CALLS` (removed in phase 45 — see the revision note
|
|
||||||
> below); `argument` is `"source/path"` for
|
|
||||||
> `read_document`, null otherwise). Client rule: render each `tool` frame
|
|
||||||
> as a "calling tool" line/state (task 05); `delta` and `done` shapes are
|
|
||||||
> unchanged — the read document is reflected in `done.sources` instead
|
|
||||||
> (deduped) — a recorded extension of A15, not a silent deviation.
|
|
||||||
>
|
|
||||||
> **SSE revision (phase 45, owner permission 2026-08-27):** the phase-37
|
|
||||||
> per-turn tool budgets are **removed** (owner: "allow the LLM to make
|
|
||||||
> as many tool calls as it wants — `TODO.md` L8): `BOR_AGENT_LIST_CALLS`
|
|
||||||
> / `BOR_AGENT_READ_CALLS` no longer exist; `BOR_AGENT_MAX_ROUNDS`
|
|
||||||
> (default 10) caps the tool rounds and `0` disables the tools
|
|
||||||
> entirely (the pre-phase-37 path). The `tool` event shape and the
|
|
||||||
> `done` shape are unchanged — a recorded revision of the phase-37
|
|
||||||
> note's budget wording, not a silent deviation.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Data Model (PostgreSQL 17)
|
|
||||||
|
|
||||||
Created by `alembic/versions/0001_initial_schema.py` (idempotent
|
|
||||||
`CREATE EXTENSION IF NOT EXISTS vector`).
|
|
||||||
|
|
||||||
### `documents`
|
|
||||||
| Column | Type | Notes |
|
|
||||||
|--------|------|-------|
|
|
||||||
| id | `UUID` PK | |
|
|
||||||
| source | `VARCHAR(120)` | source dir basename, e.g. `Homelab` |
|
|
||||||
| path | `VARCHAR(1000)` | relative to source dir, e.g. `ansible/roles/k3s.md` |
|
|
||||||
| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
|
|
||||||
| title | `VARCHAR(500)` | first markdown H1, else file stem |
|
|
||||||
| content | `TEXT` | **full markdown — the RAG context** |
|
|
||||||
| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
|
|
||||||
| indexed_at | `TIMESTAMPTZ` | |
|
|
||||||
| — | `UNIQUE (source, path)` | upsert key |
|
|
||||||
|
|
||||||
### `chunks`
|
|
||||||
| Column | Type | Notes |
|
|
||||||
|--------|------|-------|
|
|
||||||
| id | `UUID` PK | |
|
|
||||||
| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
|
|
||||||
| position | `INT` | 0-based order within the doc |
|
|
||||||
| content | `TEXT` | chunk text (heading-aware) |
|
|
||||||
| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
|
|
||||||
| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
|
|
||||||
|
|
||||||
> No vector index in v1: sequential scan is fine at this corpus size
|
|
||||||
> (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
|
|
||||||
|
|
||||||
### `query_log`
|
|
||||||
`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
|
|
||||||
|
|
||||||
### Document state transitions
|
|
||||||
```
|
|
||||||
unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
|
|
||||||
│
|
|
||||||
└──file deleted + --prune──▶ removed (chunks cascade)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Chunking policy (markdown-aware)
|
|
||||||
Split on `## `/`### ` headings into sections; sub-split any section longer
|
|
||||||
than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
|
|
||||||
`BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
|
|
||||||
preceding heading in the text for retrieval quality.
|
|
||||||
|
|
||||||
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
|
|
||||||
`---` separators (key line kept as anchor); `json` pretty-printed, split on
|
|
||||||
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
|
|
||||||
`txt` on paragraphs; markdown unchanged. The quadlet family
|
|
||||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`)
|
|
||||||
and `j2` (Jinja templates) are plain-text chunked — no format-specific
|
|
||||||
splitter (A9 revised 2026-08-27, phase 47). Every format honors the
|
|
||||||
1200-char hard cap (aipi ~1024-token request limit).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. RAG Pipeline & Persona
|
|
||||||
|
|
||||||
### Locked system prompt (sent with every chat turn)
|
|
||||||
```
|
|
||||||
You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
|
|
||||||
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
|
|
||||||
optimistic about the user's ability to do things ("you've got this").
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
1. Answer ONLY from the provided document context. Cite which document(s)
|
|
||||||
you used, by path.
|
|
||||||
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
|
|
||||||
the docs are the value.
|
|
||||||
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
|
|
||||||
Start your answer with a variant of: "I haven't done anything like that."
|
|
||||||
Then offer 2-3 alternative questions about things you DO have notes on.
|
|
||||||
4. Never invent facts, hosts, or steps that are not in the context.
|
|
||||||
5. Keep answers tight: short paragraphs, bullets where helpful.
|
|
||||||
|
|
||||||
<relevance>{HIGH|LOW}</relevance>
|
|
||||||
```
|
|
||||||
- `HIGH` mode appends the full document text under `<documents>…</documents>`.
|
|
||||||
- `LOW` mode (deflection) appends only the **titles** of the weak hits so the
|
|
||||||
model can suggest real alternatives (marker used by the E2E mock:
|
|
||||||
`DEFLECT_MODE` appears in the system prompt).
|
|
||||||
|
|
||||||
### Retrieval (hybrid — A7/A8, revised 2026-08-21)
|
|
||||||
- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
|
|
||||||
- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
|
|
||||||
- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
|
|
||||||
ranked by best chunk's fused score → top 2 → full content, concatenated
|
|
||||||
— **never truncated** (A7 revised, phase 24, owner permission
|
|
||||||
2026-08-24).
|
|
||||||
- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
|
|
||||||
(default 0.62, calibrated against the `embed` model's measured 0.41–0.84
|
|
||||||
distribution) **and** zero FTS hits among the candidates.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. UI/UX Strategy
|
|
||||||
|
|
||||||
### 7.1 Layout structure
|
|
||||||
- **App frame:** sticky header (64px) + `<main>` (flex-grow) + footer.
|
|
||||||
Container: `max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem`.
|
|
||||||
- **Shared header (Phase 19, owner permission 2026-08-23):** the bar is a
|
|
||||||
shared contract across chat / sources / viewer — one bar per page, same
|
|
||||||
controls (brand + nav [Chat, Sources — admin only] + New Chat + Sign in
|
|
||||||
/ Sign out on chat & sources; the viewer bar = back + title + the same
|
|
||||||
actions in `.doc-header-actions`). One shared module
|
|
||||||
(`frontend/assets/header.js`) toggles the existing controls on each
|
|
||||||
page, so they can never "disappear" between pages again; the heights
|
|
||||||
stay pinned at 64px / 58px (phase 12).
|
|
||||||
- **Chat:** a *centered column capped at 46rem*. This is deliberate: chat is
|
|
||||||
a vertical conversation — a centered, capped column is the correct pattern
|
|
||||||
(NOT a layout bug). The 72rem frame + header/footer ensure the column
|
|
||||||
never reads as a hairline in a sea of whitespace.
|
|
||||||
- **Sources page:** full-width responsive **table** (min 640px, horizontal
|
|
||||||
scroll wrapper on small screens) + stat cards in
|
|
||||||
`grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
|
|
||||||
No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of
|
|
||||||
the container width.
|
|
||||||
- **Mobile (≤640px):** suggestion chips become a horizontally scrollable row;
|
|
||||||
composer stays reachable with `safe-area-inset-bottom`; touch targets ≥44px.
|
|
||||||
|
|
||||||
### 7.2 Accessibility (WCAG 2.1 AA)
|
|
||||||
- Semantic landmarks on every page: `<header>`, `<nav aria-label>`,
|
|
||||||
`<main>`, `<footer>`; skip-link to `#main`.
|
|
||||||
- Every control labeled: visible `<label>` or `aria-label` (icon-only
|
|
||||||
buttons always get `aria-label`); form input has a (visually-hidden) label.
|
|
||||||
- Live regions: message stream `aria-live="polite"`; typing indicator
|
|
||||||
`role="status"`; banner `role="status"`; errors `role="alert"`.
|
|
||||||
- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
|
|
||||||
`#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
|
|
||||||
surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
|
|
||||||
on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
|
|
||||||
brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
|
|
||||||
(border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
|
|
||||||
all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
|
|
||||||
layer.
|
|
||||||
- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
|
|
||||||
typing/spinner animations.
|
|
||||||
|
|
||||||
### 7.3 No external dependencies
|
|
||||||
- System font stack only (no font files to bundle, no CDN fonts).
|
|
||||||
- Zero `<script src="https://…">` / `<link href="https://…">` — enforced by
|
|
||||||
an integration test (`tests/integration/test_api.py::test_index_html_served_locally`)
|
|
||||||
and re-checked by every UI phase's verification step.
|
|
||||||
- Markdown rendering is a ~60-line local function (escape-first, then
|
|
||||||
transform) — XSS-safe, no library.
|
|
||||||
|
|
||||||
### 7.4 Visual feedback standard (the "never stale" contract)
|
|
||||||
| State | UI |
|
|
||||||
|-------|----|
|
|
||||||
| **Idle** | Send button enabled, label "Send". |
|
|
||||||
| **Thinking (pre-token)** | 3-dot typing bubble + button disabled with spinner, label "Thinking…". |
|
|
||||||
| **Thinking (model reasoning)** | Collapsible `.thinking` block streams open (replaces the typing dots as the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); 120s guard clears on the first `thinking` *or* `delta` event. |
|
|
||||||
| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
|
|
||||||
| **Done (answer)** | Source chips under the bubble (mono, path-based); button re-enabled. |
|
|
||||||
| **Done (deflected)** | Amber-bordered bubble + "Maybe try:" suggestion chips. |
|
|
||||||
| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled. |
|
|
||||||
| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
|
|
||||||
| **Guard** | 120s client-side timeout → error state (a button can never sit "stuck" forever). |
|
|
||||||
| **Scroll (follow-the-bottom, phase 18)** | The page auto-scrolls only while the user is pinned to the bottom (≤200px band, `NEAR_BOTTOM_PX` — the composer zone; submitting reveals the user's message through the same gate, which holds in real use); scrolling up holds the viewport for the rest of the turn (thinking and answer alike); restore lands one-shot on the latest message. |
|
|
||||||
|
|
||||||
> The **Thinking (model reasoning)** row is a phase-17 addition (owner
|
|
||||||
> permission 2026-08-23) — see the §4 SSE revision.
|
|
||||||
>
|
|
||||||
> The **Scroll** row is a phase-18 addition (owner choice 2026-08-23 —
|
|
||||||
> option 1: follow-the-bottom, no "↓ new content" pill).
|
|
||||||
|
|
||||||
### 7.5 Component inventory (ids used by tests)
|
|
||||||
`#messages` (stream), `#empty-state`, `#suggestions`, `.suggestion-chip`,
|
|
||||||
`#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
|
|
||||||
`.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
|
|
||||||
`#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
|
|
||||||
`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
|
|
||||||
(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
|
|
||||||
`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
|
|
||||||
path links); thinking (phase 17, owner permission 2026-08-23):
|
|
||||||
`.thinking`, `.thinking-text` (collapsible thinking block; plain
|
|
||||||
`<summary>`, no id); auth (phase 16, owner permission 2026-08-22):
|
|
||||||
`#sign-in-link`, `#sign-out-btn`, `#sources-gate`; shared header (phase
|
|
||||||
19, owner permission 2026-08-23): `#nav-sources` (Sources nav link,
|
|
||||||
hidden for anonymous), `#new-chat-btn` + `#sign-in-link` + `#sign-out-btn`
|
|
||||||
on the sources and viewer pages (ids shared with chat),
|
|
||||||
`.doc-header-actions` (viewer).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Debugging (debugpy protocol)
|
|
||||||
|
|
||||||
- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit test).
|
|
||||||
- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`, **non-blocking**,
|
|
||||||
app continues; IDE attaches on demand.
|
|
||||||
- Entry point: `app/core/debugging.py::configure_debugging()` called at the top
|
|
||||||
of `app/main.py` module import — so `uv run uvicorn app.main:app`,
|
|
||||||
`python -m scripts.…`, and tests all honor it.
|
|
||||||
- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Observability
|
|
||||||
|
|
||||||
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
|
|
||||||
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
|
|
||||||
- **Per-chat-turn log line (required):**
|
|
||||||
`question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N kb_chars=N threshold=… deflected=… sources=… thinking_chars=… tool_calls=N total_ms=…`
|
|
||||||
(`thinking_chars=` counts the turn's reasoning chars — phase 17, owner
|
|
||||||
permission 2026-08-23 — and is counted even when `BOR_STREAM_THINKING=0`
|
|
||||||
suppresses the frames.)
|
|
||||||
|
|
||||||
> **Log-line revision (phase 37, owner permission 2026-08-26):** the
|
|
||||||
> required per-turn line gains `tool_calls=N` after `thinking_chars=` —
|
|
||||||
> the count of agent tool executions that consumed budget on the turn
|
|
||||||
> (phase 37's `list_documents` / `read_document`; rejected calls do not
|
|
||||||
> count, and deflected turns run no tools). `summary_hits=` (phase 30)
|
|
||||||
> and `kb_chars=` (phase 31) are recorded here as well; `sources=` lists
|
|
||||||
> the retrieval docs plus any agent-read documents, deduped.
|
|
||||||
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
|
|
||||||
(counts, embedding batches, total time).
|
|
||||||
- **`query_log` table:** durable record of every question (score, deflection,
|
|
||||||
sources, latency) for tuning the threshold and finding gaps in the docs.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 10. Testing Strategy (LOCKED — A16)
|
|
||||||
|
|
||||||
| Layer | Tooling | Runs | Gate |
|
|
||||||
|-------|---------|------|------|
|
|
||||||
| Unit | pytest | `uv run pytest tests/unit` | pass |
|
|
||||||
| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
|
|
||||||
| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
|
|
||||||
| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
|
|
||||||
|
|
||||||
- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
|
|
||||||
OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap
|
|
||||||
vectors, so the cosine threshold behaves like production: on-topic
|
|
||||||
questions retrieve, off-topic questions deflect. `E2E_REAL_LLM=1` switches
|
|
||||||
the app fixture to live aipi (needs imported KB).
|
|
||||||
- **E2E prerequisites:** `podman compose up -d db`; Chromium installed via
|
|
||||||
`uv run playwright install chromium`.
|
|
||||||
- DB isolation: story E2E fixtures truncate `query_log` (and re-import
|
|
||||||
fixtures for import-dependent stories) per test module.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 11. Import & Update Workflow (documented in README)
|
|
||||||
|
|
||||||
```
|
|
||||||
# first import (and any future refresh):
|
|
||||||
uv run python -m scripts.import_docs # defaults: ~/Homelab ~/Deployments
|
|
||||||
uv run python -m scripts.import_docs --source ~/OtherProject # extra dirs
|
|
||||||
uv run python -m scripts.import_docs --prune # drop deleted / filtered-out files
|
|
||||||
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
|
|
||||||
uv run python -m scripts.llm_probe # sanity: models + dim
|
|
||||||
```
|
|
||||||
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
|
|
||||||
(no re-embedding); changed files are re-chunked + re-embedded (chunks
|
|
||||||
replaced atomically); `--prune` removes docs whose files disappeared or no
|
|
||||||
longer match the format filter. Formats per A9 (revised 2026-08-27):
|
|
||||||
`md, markdown, txt, yaml, yml, json, py`, the quadlet family
|
|
||||||
(`container, network, volume, image, pod, kube, swap, os, endpoint`), and
|
|
||||||
`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
|
|
||||||
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
|
|
||||||
ranks live hybrid results for a question (retrieval tuning).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 12. Roadmap (one story → one phase → one Playwright gate)
|
|
||||||
|
|
||||||
| Phase | File | Story | Playwright gate |
|
|
||||||
|-------|------|-------|-----------------|
|
|
||||||
| 01 | `01_infrastructure.md` | — (foundation) | `tests/e2e/test_smoke.py` |
|
|
||||||
| 02 | `02_story_import_documents.md` | `import-documents.md` | `tests/e2e/test_import_documents.py` |
|
|
||||||
| 03 | `03_story_chat_rag.md` | `chat-rag-answer.md` | `tests/e2e/test_chat_rag.py` |
|
|
||||||
| 04 | `04_story_honest_deflection.md` | `honest-deflection.md` | `tests/e2e/test_honest_deflection.py` |
|
|
||||||
| 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
|
|
||||||
| 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
|
|
||||||
| 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
|
|
||||||
| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
|
|
||||||
| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
|
|
||||||
| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
|
|
||||||
| 17 | `17_thinking_display.md` | `thinking-display.md` | `tests/e2e/test_thinking_display.py` |
|
|
||||||
| 18 | `18_follow_bottom_scroll.md` | `follow-bottom-scroll.md` | `tests/e2e/test_follow_bottom_scroll.py` |
|
|
||||||
| 19 | `19_shared_header.md` | `shared-header.md` | `tests/e2e/test_shared_header.py` |
|
|
||||||
| 24 | `24_whole_document_context.md` | `whole-document-context.md` | `tests/e2e/test_whole_document_context.py` |
|
|
||||||
|
|
||||||
> Row 17 (thinking display) added 2026-08-23 with owner permission — the
|
|
||||||
> A15 SSE extension recorded in §4.
|
|
||||||
>
|
|
||||||
> Row 18 (follow-the-bottom scroll) added 2026-08-23 with owner choice —
|
|
||||||
> option 1: follow-the-bottom, no "↓ new content" pill (UI-behavior-only
|
|
||||||
> change; no anchor revised).
|
|
||||||
>
|
|
||||||
> Row 19 (shared header) added 2026-08-23 with owner permission —
|
|
||||||
> Sign in/Sign out + New Chat on every page via one shared module, and
|
|
||||||
> the phase-16 "Sources" link UX choice revised: the nav link is hidden
|
|
||||||
> for anonymous (the soft-gate page and the A10 API split are unchanged).
|
|
||||||
>
|
|
||||||
> Row 24 (whole-document context) added 2026-08-24 with owner permission
|
|
||||||
> — A7's 24k context cap removed (documents are never truncated)
|
|
||||||
|
|
||||||
Completion = unit+integration green, coverage >90%, story E2E green in
|
|
||||||
isolation, UI verification passed, **one `--no-gpg-sign` commit**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 13. Future (post-v1 hooks, deliberately not built)
|
|
||||||
- Auth (stateless API makes this a drop-in: sessions → Valkey).
|
|
||||||
- HNSW index on `chunks.embedding` at scale.
|
|
||||||
- Conversation persistence (messages tables).
|
|
||||||
- Watchdog auto-re-import (inotify) — until then the script is the truth.
|
|
||||||
- More sources: any directory of A9-format files via `--source`.
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
# Task 02 — E2E story suite, story file, validation, commit
|
|
||||||
|
|
||||||
**Phase:** `21_thinking_no_scroll` · **Source:** `TODO.md` L4
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The story gate: `tests/e2e/test_thinking_no_scroll.py` proves the window
|
|
||||||
can't be user-scrolled but always tracks the live tail, plus regressions,
|
|
||||||
story file, final validation, and the single atomic commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_thinking_no_scroll.py` (new — reuse
|
|
||||||
`test_thinking_display.py`'s mock-LLM streaming scaffolding; the mock
|
|
||||||
must stream a **long** thinking body, in many chunks, so
|
|
||||||
`.thinking-text` overflow exceeds its 320px box). Tests:
|
|
||||||
1. `test_thinking_window_not_user_scrollable` — open the block, wait
|
|
||||||
until `scrollHeight > clientHeight`; focus `.thinking-text`
|
|
||||||
(`el.focus()`), dispatch mouse wheel over it
|
|
||||||
(`page.mouse.wheel(0, -200)` after moving the mouse over the
|
|
||||||
element) and press `Home`/`ArrowUp`: `scrollTop` must not decrease
|
|
||||||
(assert `scrollTop` unchanged within 1px between actions).
|
|
||||||
2. `test_thinking_window_tracks_live_tail` — while chunks stream,
|
|
||||||
after the 2nd-to-last and last chunk:
|
|
||||||
`scrollTop === scrollHeight` (within 1px) — the visible window is
|
|
||||||
the live tail; the **last** chunk's text is within the visible
|
|
||||||
rectangle (its offsetTop + scrollTop geometry check, or
|
|
||||||
`elementFromPoint` at the box's bottom).
|
|
||||||
3. `test_thinking_window_css_contract` — computed style of
|
|
||||||
`.thinking-text`: `overflow-y === "hidden"`,
|
|
||||||
`max-height === "320px"`.
|
|
||||||
4. `test_answer_bubble_still_scrollable` (regression, phase 11) — a
|
|
||||||
long answer (use the long-answer mock from
|
|
||||||
`test_long_answers.py`): the answer bubble is still
|
|
||||||
user-scrollable (scrollTop moves on wheel) and
|
|
||||||
`overflow-y` is not `hidden` there.
|
|
||||||
5. `test_restored_collapsed_thinking_unaffected` (regression,
|
|
||||||
phase 17) — a turn with stored `thinking`, reload: the collapsed
|
|
||||||
Thinking block renders with its text (existing pin from
|
|
||||||
`test_thinking_display.py` — replicate, don't duplicate the file).
|
|
||||||
2. `.agent/user_stories/thinking-no-scroll.md` (new) — story file per
|
|
||||||
the repo format: goal, the bug report verbatim from `TODO.md` L4, the
|
|
||||||
owner-confirmed A2 decisions from `00_phase.md`, E2E mapping table.
|
|
||||||
3. Run the suite **in isolation** (prereq `podman compose up -d db`):
|
|
||||||
`uv run pytest tests/e2e/test_thinking_no_scroll.py -v --no-cov`.
|
|
||||||
4. Regressions, in isolation, one command each:
|
|
||||||
- `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`
|
|
||||||
- `uv run pytest tests/e2e/test_long_answers.py -v --no-cov`
|
|
||||||
5. Final validation: `uv run pytest` green; `uv run pytest --cov=app
|
|
||||||
--cov-report=term-missing` ≥ today's number (>90% gate);
|
|
||||||
`uv run ruff check . && uv run pyright` clean.
|
|
||||||
6. **UI Structure Check** (AGENTS.md rule 5): no new surface; the block
|
|
||||||
keeps its summary chevron, focus-visible ring, aria-live/label
|
|
||||||
contract, and the reduced-motion stillness (styles.css ~line 686).
|
|
||||||
7. Write the phase report (`.agent/reports/21_thinking_no_scroll/`).
|
|
||||||
8. Commit (one atomic commit) and move the phase:
|
|
||||||
```bash
|
|
||||||
git add -A .agent/ frontend/ tests/
|
|
||||||
git commit --no-gpg-sign -m "fix(ui): thinking window no longer scrolls — live 320px view pinned to the stream tail"
|
|
||||||
mv .agent/phases/todo/21_thinking_no_scroll .agent/phases/complete/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Story suite green **in isolation**; both regression suites green in
|
|
||||||
isolation; full unit+integration suite green; `app/` coverage at or
|
|
||||||
above today's number (>90%); ruff + pyright clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `test_thinking_no_scroll.py` 5/5 in isolation.
|
|
||||||
- [ ] Regressions (thinking display, long answers) green in isolation.
|
|
||||||
- [ ] Story file + phase report exist.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
# Task 01 — Diagnose and fix the animated background (styles.css)
|
|
||||||
|
|
||||||
**Phase:** `22_background_animation` · **Source:** `TODO.md` L5 —
|
|
||||||
*"Fix background animation not working, just blinking"*
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Find why the phase-08 animated background reads as "just blinking" and
|
|
||||||
fix `styles.css` so the grid drift and the glow breathe are both visibly
|
|
||||||
and smoothly alive, per the phase-08 design comments (pure CSS, zero JS).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **Reproduce:** `uv run uvicorn app.main:app --reload` (db up), open `/`
|
|
||||||
in a visible Chromium window (Playwright or the interactive browser),
|
|
||||||
observe ≥15s. Note exactly what moves and what doesn't.
|
|
||||||
2. **Diagnose per the `00_phase.md` checklist** (per-layer visibility
|
|
||||||
toggles, `background-position` samples on `body::before`, mask
|
|
||||||
inspection, occlusion check against `html`/`body` rules, glow
|
|
||||||
opacity-swing perception). Record findings + before-screenshot in
|
|
||||||
`.agent/reports/22_background_animation/` and
|
|
||||||
`.agent/screenshots/22_background_animation/`.
|
|
||||||
**ASSUMPTION (to verify, not assume):** the likely culprits, in
|
|
||||||
order — (a) the masked grid drift is too faint/slow to perceive,
|
|
||||||
(b) only the glow opacity swing is visible and it reads as a blink,
|
|
||||||
(c) a later rule occludes the `z-index: -1` layers. Confirm which
|
|
||||||
one actually fires before touching CSS; the fix must match the found
|
|
||||||
cause.
|
|
||||||
3. **Fix in `frontend/assets/styles.css`** (smallest change that makes
|
|
||||||
the design read):
|
|
||||||
- grid: raise line alpha and/or the mask's visible radius and/or the
|
|
||||||
drift speed as needed for a clearly visible, seamless drift
|
|
||||||
(drift delta must still equal one 44px cell for a seamless loop —
|
|
||||||
if the speed changes, keep `background-position` 0→44px and only
|
|
||||||
move the duration);
|
|
||||||
- glow: if the breathe reads as a blink, narrow the opacity delta
|
|
||||||
(e.g. 0.8↔1) and/or lengthen the period — it must read as
|
|
||||||
breathing, not pulsing;
|
|
||||||
- keep: both layers `position: fixed; inset: 0; z-index: -1;
|
|
||||||
pointer-events: none`; no `filter: blur`; no JS; palette/contrast
|
|
||||||
untouched.
|
|
||||||
4. **After-screenshot** (same viewport, two frames a few seconds apart
|
|
||||||
showing motion) into the same screenshots dir.
|
|
||||||
5. `tests/unit/test_background_animation.py` (new — repo source-pin
|
|
||||||
pattern): pin the **final** `styles.css` values — both
|
|
||||||
`@keyframes` present, `body::before` → `bg-grid-drift linear
|
|
||||||
infinite`, `body::after` → `bg-glow-breathe`, both layers
|
|
||||||
`fixed`/`z-index: -1`/`pointer-events: none`, `html` keeps
|
|
||||||
`background: var(--bg)`, `body` keeps `background: transparent`.
|
|
||||||
6. Manual re-verify: the "just blinking" perception is gone — smooth
|
|
||||||
drift + gentle breathe, no jank, no static frame.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- `uv run pytest tests/unit/test_background_animation.py -v` green.
|
|
||||||
- `uv run ruff check . && uv run pyright` clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Root cause documented (with before/after screenshots) in the
|
|
||||||
phase report dir.
|
|
||||||
- [ ] Both layers visibly animate as the phase-08 design describes;
|
|
||||||
pure CSS, zero JS, no blur.
|
|
||||||
- [ ] Unit pins green against the final values; lint/types clean.
|
|
||||||
@@ -1,120 +0,0 @@
|
|||||||
# Phase 23 — Containerfile: Build the Whole App Image Again
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L6 — *"Fix Containerfile build not working"*
|
|
||||||
**Story:** `.agent/user_stories/containerfile-build.md` (created by task 02)
|
|
||||||
**Context:** `Containerfile` (3 stages: node:22-alpine + esbuild
|
|
||||||
0.25.5 frontend bundle → uv/python deps → slim runtime serving
|
|
||||||
`/app/static`); `frontend/` (4 pages: `index.html`, `sources.html`,
|
|
||||||
`document.html`, `login.html`; assets: `styles.css`, `markdown.js`
|
|
||||||
(classic script), `header.js`/`app.js`/`sources.js`/`document.js`/
|
|
||||||
`login.js` (ES modules)); `scripts/entrypoint.sh`.
|
|
||||||
|
|
||||||
## Verified diagnosis (2026-08-24, this conversion — not a guess)
|
|
||||||
1. **Root cause of the build failure:** phase 19 switched the page
|
|
||||||
scripts to `import … from "/assets/header.js"` (an absolute URL).
|
|
||||||
esbuild resolves that as the *filesystem* path `/assets/header.js`
|
|
||||||
and the stage-1 bundle dies:
|
|
||||||
`✘ [ERROR] Could not resolve "/assets/header.js"`
|
|
||||||
(reproduced with esbuild **0.25.5**, the exact pinned version, on a
|
|
||||||
copy of `frontend/`).
|
|
||||||
2. **Secondary gap (image would be broken even if it built):** stage 1
|
|
||||||
bundles only `app.js` + `sources.js` and copies only `index.html` +
|
|
||||||
`sources.html`. Missing from the image: `document.html` +
|
|
||||||
`login.html` (phases 10/16), `document.js` + `login.js`, and
|
|
||||||
`markdown.js` (classic script loaded by `index.html` +
|
|
||||||
`document.html`).
|
|
||||||
3. **Verified fix:** with relative imports (`from "./header.js"`) all
|
|
||||||
four page scripts bundle cleanly with esbuild 0.25.5.
|
|
||||||
4. **Latent double-evaluation trap:** all four HTML pages also load
|
|
||||||
`<script type="module" src="/assets/header.js">` directly while the
|
|
||||||
page script imports it. In dev the browser dedupes (same module
|
|
||||||
URL) — but in the image the bundled page script already contains the
|
|
||||||
header code, so shipping a raw `header.js` too would evaluate the
|
|
||||||
module **twice** (duplicate sign-out listener, double init). The
|
|
||||||
direct tags are redundant: the page script's `import` is hoisted and
|
|
||||||
guarantees `header.js` evaluates before the page script's body calls
|
|
||||||
`initSharedHeader()`, in dev and in the bundle alike.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
`podman build -f Containerfile .` succeeds, and the resulting image
|
|
||||||
serves the **whole app** — all four pages with their bundled, minified,
|
|
||||||
local-only assets (No CDN rule) — with `header.js` evaluated exactly
|
|
||||||
once per page.
|
|
||||||
|
|
||||||
## Owner-confirmed (2026-08-24, roadmap A4)
|
|
||||||
1. **Relative imports** (`./header.js`) over an esbuild alias — simpler,
|
|
||||||
verified working, dev-server behavior unchanged (files are
|
|
||||||
side-by-side).
|
|
||||||
2. **Remove the four redundant direct `header.js` script tags** (the
|
|
||||||
design above) rather than ship a raw `header.js` into the image —
|
|
||||||
single module evaluation, no duplicate listeners.
|
|
||||||
3. The image must cover **all four pages + all local assets** they
|
|
||||||
reference — the integration test (task 02) enforces this coverage so
|
|
||||||
the gap cannot silently reappear.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `19_shared_header` (complete) — introduced the absolute imports (root
|
|
||||||
cause) and the direct `header.js` tags.
|
|
||||||
- `10_story_document_viewer` / `16_admin_auth` (complete) — the pages
|
|
||||||
missing from the image.
|
|
||||||
- `08_story_dark_tech_theme` (complete) — No CDN rule the image must
|
|
||||||
honor.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_fix_containerfile_build.md` — relative imports, tag removal,
|
|
||||||
stage-1 asset coverage, green `podman build`, image smoke test.
|
|
||||||
2. `02_integration_test_commit.md` — `tests/integration/
|
|
||||||
test_containerfile_assets.py` (hermetic coverage pin), regression
|
|
||||||
suites, story file, final validation, the single atomic commit,
|
|
||||||
phase move to `complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A11 honored** — vanilla JS, no CDN, static serving from FastAPI.
|
|
||||||
**A16 honored** — integration test for the new build coverage; story
|
|
||||||
file + report; no Playwright suite required (this phase is
|
|
||||||
build/infrastructure — the phase gate is the hermetic integration
|
|
||||||
test + the real `podman build` + image smoke recorded in the report,
|
|
||||||
plus the dev-server E2E regressions). No anchor changed.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- **Integration (new `tests/integration/test_containerfile_assets.py`,
|
|
||||||
hermetic — no podman, no network):** every `frontend/*.html` is
|
|
||||||
copied into stage 1's `/out`; every local `src`/`href` asset
|
|
||||||
referenced by the four pages is produced by a stage-1 line (esbuild
|
|
||||||
`--outfile` or `cp`); the four page module scripts are the exact set
|
|
||||||
esbuild bundles; `markdown.js` is produced; no HTML references
|
|
||||||
`/assets/header.js` directly (single-evaluation design pin); the
|
|
||||||
esbuild version stays pinned.
|
|
||||||
- **Unit:** none (no `app/` changes).
|
|
||||||
- **Coverage:** the >90% `app/` gate is unaffected, re-run to prove it.
|
|
||||||
- **Build gate (manual, recorded in the report):** `podman build
|
|
||||||
-f Containerfile .` green; image smoke (task 01 step 6) results +
|
|
||||||
log excerpt in `.agent/reports/23_containerfile_build/`.
|
|
||||||
- **Dev regressions (E2E, isolated):** `test_smoke.py`,
|
|
||||||
`test_shared_header.py`, `test_chat_persistence.py` (the HTML tag
|
|
||||||
removal touches dev page load).
|
|
||||||
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Local esbuild 0.25.5 bundles all four page scripts cleanly.
|
|
||||||
- [ ] `podman build -f Containerfile .` green (log excerpt in the
|
|
||||||
report).
|
|
||||||
- [ ] Image smoke: container runs (throwaway Postgres 17 + pgvector);
|
|
||||||
`GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
|
|
||||||
`/assets/app.js` minified and contains the header code;
|
|
||||||
`/assets/markdown.js` 200; no `http(s)://` asset reference in any
|
|
||||||
served page (No CDN rule).
|
|
||||||
- [ ] Dev server unchanged in behavior: the three regression E2E suites
|
|
||||||
green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app
|
|
||||||
--cov-report=term-missing` ≥ today's number.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/containerfile-build.md` exists.
|
|
||||||
- [ ] One `--no-gpg-sign` commit (below);
|
|
||||||
`.agent/phases/todo/23_containerfile_build/` moved to
|
|
||||||
`.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Commit
|
|
||||||
```bash
|
|
||||||
git add -A .agent/ Containerfile frontend/ tests/ && git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
|
|
||||||
```
|
|
||||||
@@ -1,91 +0,0 @@
|
|||||||
# Task 01 — Fix the build: relative imports, tag removal, full stage-1 asset coverage
|
|
||||||
|
|
||||||
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6 —
|
|
||||||
*"Fix Containerfile build not working"*
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make `podman build -f Containerfile .` succeed and ship the **complete**
|
|
||||||
frontend in the image: all four pages, all four bundled page modules,
|
|
||||||
the classic `markdown.js`, and the minified `styles.css` — with
|
|
||||||
`header.js` evaluated exactly once per page.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **Reproduce the failure** and record it in
|
|
||||||
`.agent/reports/23_containerfile_build/` (log excerpt):
|
|
||||||
- fast: `npx -y esbuild@0.25.5` on a copy of `frontend/` → the
|
|
||||||
`Could not resolve "/assets/header.js"` error (root cause, already
|
|
||||||
reproduced during conversion);
|
|
||||||
- authoritative: `podman build -f Containerfile .` → stage 1 fails
|
|
||||||
at the same line.
|
|
||||||
2. **`frontend/assets/{app,sources,document,login}.js`** — change the
|
|
||||||
header import from absolute URL to relative (one line each; the
|
|
||||||
specifiers are currently `from "/assets/header.js"`):
|
|
||||||
```js
|
|
||||||
import { … } from "./header.js";
|
|
||||||
```
|
|
||||||
(owner-confirmed A4 — relative over esbuild alias; dev-server
|
|
||||||
behavior is unchanged since the files are side-by-side and the
|
|
||||||
module URL resolves to the same file.)
|
|
||||||
3. **Remove the four redundant direct `header.js` tags** (owner-confirmed
|
|
||||||
A4-2 — the single-evaluation design from `00_phase.md`):
|
|
||||||
- `frontend/index.html` (~line 119) —
|
|
||||||
`<script type="module" src="/assets/header.js"></script>`;
|
|
||||||
- `frontend/sources.html` (~line 125), `frontend/document.html`
|
|
||||||
(~line 80), `frontend/login.html` (~line 67) — same tag.
|
|
||||||
- Update the surrounding HTML comments that describe the
|
|
||||||
header-before-page-script load order (e.g. index.html ~lines
|
|
||||||
115–119): the order is now guaranteed by the page script's own
|
|
||||||
`import` (hoisted, evaluated before the page script body calls
|
|
||||||
`initSharedHeader()`).
|
|
||||||
4. **`Containerfile` stage 1** — cover the whole app (keep the pinned
|
|
||||||
`esbuild@0.25.5` and the existing flags):
|
|
||||||
```dockerfile
|
|
||||||
RUN mkdir -p /out/assets \
|
|
||||||
&& esbuild ./assets/app.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/app.js \
|
|
||||||
&& esbuild ./assets/sources.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/sources.js \
|
|
||||||
&& esbuild ./assets/document.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/document.js \
|
|
||||||
&& esbuild ./assets/login.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/login.js \
|
|
||||||
&& esbuild ./assets/markdown.js --minify --outfile=/out/assets/markdown.js \
|
|
||||||
&& esbuild ./assets/styles.css --minify --outfile=/out/assets/styles.css \
|
|
||||||
&& cp ./index.html ./sources.html ./document.html ./login.html /out/
|
|
||||||
```
|
|
||||||
(`markdown.js` is a classic script — minify only, **no** `--bundle`;
|
|
||||||
it exposes globals used by the pages.)
|
|
||||||
5. **Verify locally (no podman):** with esbuild 0.25.5, all four module
|
|
||||||
bundles + the markdown minify succeed on the real `frontend/` (not a
|
|
||||||
copy — the copy was only for the diagnosis).
|
|
||||||
6. **`podman build -f Containerfile .`** → green.
|
|
||||||
7. **Image smoke test** (results + log excerpt into the report dir):
|
|
||||||
- throwaway Postgres 17 + pgvector (`podman compose up -d db` and
|
|
||||||
point the container at it, or a one-off container with the same
|
|
||||||
env as `compose.yaml`);
|
|
||||||
- run the built image (migrations run via the entrypoint);
|
|
||||||
- `GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
|
|
||||||
- `GET /assets/app.js` → 200, minified (single-line-ish), and
|
|
||||||
contains the header code (e.g. the `clearChatStorage` function
|
|
||||||
body); `GET /assets/markdown.js`, `/styles.css`, the other three
|
|
||||||
page modules → 200;
|
|
||||||
- No CDN rule: none of the four served pages contain an `http(s)://`
|
|
||||||
`src`/`href` asset reference.
|
|
||||||
- Teardown the throwaway containers when done.
|
|
||||||
8. **Dev-server regression check** (the tag removal touches dev page
|
|
||||||
load — confirm boot order still holds): `uv run uvicorn
|
|
||||||
app.main:app --reload`, load all four pages, check the sign-out
|
|
||||||
binding exists exactly once (DevTools: no duplicate listener — one
|
|
||||||
`POST /api/logout` per click) and `initSharedHeader()` ran. (The
|
|
||||||
isolated E2E regressions run in task 02.)
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Steps 5–8 above; `uv run ruff check . && uv run pyright` clean
|
|
||||||
(no Python changes, but keep the gate green).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The recorded build failure is fixed at the root cause (relative
|
|
||||||
imports) — not masked by an alias/patch.
|
|
||||||
- [ ] All four direct `header.js` tags removed + comments updated; the
|
|
||||||
page scripts' `import "./header.js"` is the only header load.
|
|
||||||
- [ ] Stage 1 produces: 4 HTML pages, 4 bundled modules, minified
|
|
||||||
`markdown.js`, minified `styles.css`.
|
|
||||||
- [ ] `podman build` green; image smoke all-200 + No CDN + single
|
|
||||||
header evaluation; dev-server boot unchanged (step 8).
|
|
||||||
- [ ] Log/screenshot evidence in `.agent/reports/23_containerfile_build/`.
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
# Task 02 — Integration coverage test, story file, validation, commit
|
|
||||||
|
|
||||||
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Pin the stage-1 asset coverage so it can't silently rot again (a new
|
|
||||||
page/script/asset without a matching Containerfile line fails CI), plus
|
|
||||||
regressions, story file, final validation, and the single atomic commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/integration/test_containerfile_assets.py` (new — **hermetic**:
|
|
||||||
parses `Containerfile` + `frontend/` as text, no podman, no network).
|
|
||||||
Tests:
|
|
||||||
1. `test_every_html_page_is_copied_into_stage1` — for each
|
|
||||||
`frontend/*.html` in the repo, a stage-1 line copies it into
|
|
||||||
`/out` (regex over the `cp` line; the set must be exactly the
|
|
||||||
four current pages — a new page added to `frontend/` fails this).
|
|
||||||
2. `test_every_local_asset_reference_is_produced` — collect every
|
|
||||||
local `src=`/`href=` under `assets/` or `/assets/` from the four
|
|
||||||
HTML files; each basename must be produced by a stage-1 line
|
|
||||||
(an `esbuild … --outfile=/out/assets/<name>` or a `cp` of it).
|
|
||||||
(This is what catches a missing `markdown.js`-style gap.)
|
|
||||||
3. `test_page_module_scripts_are_bundled` — the set of `type="module"`
|
|
||||||
page scripts referenced by the HTML (basenames) equals the set of
|
|
||||||
scripts esbuild bundles in stage 1 (`app.js`, `sources.js`,
|
|
||||||
`document.js`, `login.js`).
|
|
||||||
4. `test_header_module_is_imported_not_directly_loaded` — no HTML
|
|
||||||
file contains a `<script … src="/assets/header.js">` (or
|
|
||||||
`assets/header.js`) tag (the single-evaluation design pin,
|
|
||||||
owner-confirmed A4-2); and each of the four page scripts imports
|
|
||||||
it relatively (`from "./header.js"`).
|
|
||||||
5. `test_markdown_js_is_a_produced_classic_script` — `markdown.js`
|
|
||||||
has a stage-1 minify line **without** `--bundle` (it is a classic
|
|
||||||
global script) and no `import`/`export` statements at its top
|
|
||||||
level (source pin of that assumption).
|
|
||||||
6. `test_esbuild_stays_pinned` — the frontend stage pins a concrete
|
|
||||||
`esbuild@X.Y.Z` version (no floating version).
|
|
||||||
2. `.agent/user_stories/containerfile-build.md` (new) — story file per
|
|
||||||
the repo format: goal, the bug report verbatim from `TODO.md` L6, the
|
|
||||||
verified diagnosis (root cause + missing-asset gap + double-eval
|
|
||||||
trap), the owner-confirmed A4 decisions, and the test mapping table.
|
|
||||||
3. Regressions, in isolation, one command each (prereq
|
|
||||||
`podman compose up -d db`):
|
|
||||||
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
|
|
||||||
- `uv run pytest tests/e2e/test_shared_header.py -v --no-cov`
|
|
||||||
- `uv run pytest tests/e2e/test_chat_persistence.py -v --no-cov`
|
|
||||||
4. Final validation: `uv run pytest` green (includes the new
|
|
||||||
integration test); `uv run pytest --cov=app --cov-report=term-missing`
|
|
||||||
≥ today's number (>90% gate); `uv run ruff check . && uv run pyright`
|
|
||||||
clean.
|
|
||||||
5. Finish the phase report (`.agent/reports/23_containerfile_build/` —
|
|
||||||
build log excerpt, smoke results, regression results).
|
|
||||||
6. Commit (one atomic commit) and move the phase:
|
|
||||||
```bash
|
|
||||||
git add -A .agent/ Containerfile frontend/ tests/
|
|
||||||
git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
|
|
||||||
mv .agent/phases/todo/23_containerfile_build .agent/phases/complete/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- New integration suite green within `uv run pytest`; the three
|
|
||||||
regression E2E suites green in isolation; full suite green; `app/`
|
|
||||||
coverage at or above today's number (>90%); ruff + pyright clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `test_containerfile_assets.py` 6/6 within the full suite.
|
|
||||||
- [ ] Regressions (smoke, shared header, chat persistence) green in
|
|
||||||
isolation.
|
|
||||||
- [ ] Story file + phase report (build log + smoke evidence) exist.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
|
|
||||||
@@ -1,138 +0,0 @@
|
|||||||
# Phase 24 — Whole-Document Context: a matched document is never truncated
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L3–L4 — *"Documents are truncated for some reason?
|
|
||||||
This should never happen"* + *"When the LLM matches a chunk it should get
|
|
||||||
the entire document placed in its context so it can see the whole thing
|
|
||||||
before answering the question"*
|
|
||||||
**Story:** `.agent/user_stories/whole-document-context.md` (created by
|
|
||||||
task 03)
|
|
||||||
**Context:** `app/rag/retriever.py::select_documents` (the one and only
|
|
||||||
place document content is cut — the 24k budget), `app/api/chat.py::plan_turn`
|
|
||||||
(passes the budget on both HIGH and LOW paths), `app/config.py`
|
|
||||||
(`max_context_chars`), `app/rag/prompts.py` (the shared `[…truncated…]`
|
|
||||||
marker — still owned by the steering section), `tests/unit/test_retriever.py`
|
|
||||||
(pins the current cap), `tests/e2e/mock_llm.py` + `tests/e2e/test_chat_rag.py`
|
|
||||||
(E2E patterns), `.env.example` + `README.md` (document the knob).
|
|
||||||
|
|
||||||
## Verified diagnosis (2026-08-24, this conversion — not a guess)
|
|
||||||
1. **The importer stores the whole file** — `app/rag/importer.py:199–228`
|
|
||||||
reads each file into `documents.content` in full (sha256 over the whole
|
|
||||||
content). No truncation at import time.
|
|
||||||
2. **Chunking never touches the LLM context** — the 2000-char target /
|
|
||||||
1200-char hard cap only shapes `chunks` rows (retrieval + embeddings);
|
|
||||||
the chat prompt is built from `documents.content`.
|
|
||||||
3. **The viewer serves raw content** — `GET /api/documents/content`
|
|
||||||
(`app/api/docs.py`) returns `doc.content` unchanged; no truncation there
|
|
||||||
either. So the owner's "truncated for some reason" is *not* a separate
|
|
||||||
import/viewer bug.
|
|
||||||
4. **The one and only truncation point is `select_documents()`**
|
|
||||||
(`app/rag/retriever.py:252–289`): the top-2 documents' combined text is
|
|
||||||
capped at `BOR_MAX_CONTEXT_CHARS` (default **24 000**) and the
|
|
||||||
lowest-ranked overflowing document is truncated in place with
|
|
||||||
`[…truncated…]`. `plan_turn()` (`app/api/chat.py`) passes the budget on
|
|
||||||
both the HIGH (grounded) and LOW (deflected) paths. The marker the owner
|
|
||||||
saw in answers comes from here.
|
|
||||||
5. **The cap is pinned + documented** — `tests/unit/test_retriever.py`
|
|
||||||
(`test_combined_content_capped_with_truncation_marker`,
|
|
||||||
`test_single_doc_over_budget_is_truncated_to_budget`,
|
|
||||||
`test_under_budget_no_truncation`); the knob is in `.env.example`
|
|
||||||
(`BOR_MAX_CONTEXT_CHARS=24000`) and `README.md:377`.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
When hybrid retrieval matches a chunk, the LLM sees the **entire** parent
|
|
||||||
document — the 24k context budget (and `BOR_MAX_CONTEXT_CHARS`) is removed
|
|
||||||
from the document path, and a dedicated story E2E proves deterministically
|
|
||||||
that a >24k document — and the *second* document of a >24k pair (the exact
|
|
||||||
case the old budget cut) — reaches the model whole.
|
|
||||||
|
|
||||||
## Owner-confirmed (2026-08-24, roadmap D1–D5)
|
|
||||||
1. **D1 — no cap at all (revises LOCKED A7):** `select_documents` returns
|
|
||||||
the full top-N document texts, always. The `max_context_chars` setting
|
|
||||||
and `BOR_MAX_CONTEXT_CHARS` env var are removed. If a future KB ever
|
|
||||||
makes the prompt too large for the model, the existing `LLMError` → SSE
|
|
||||||
`error` path surfaces it loudly — no silent partial context. The
|
|
||||||
emergency-valve variant (raised cap + warning log) was **explicitly
|
|
||||||
rejected**.
|
|
||||||
2. **D2 — `top_n_docs = 2` unchanged** (the TODO is about truncation, not
|
|
||||||
about how many documents).
|
|
||||||
3. **D3 — no viewer/import changes** — both already serve full content
|
|
||||||
(verified diagnosis above).
|
|
||||||
4. **D4 — E2E evidence via a deterministic mock tail-echo** (repo pattern,
|
|
||||||
cf. the phase-15 tuning-note echo); the big documents are seeded
|
|
||||||
directly in the DB inside the E2E test — `tests/fixtures/docs/` must
|
|
||||||
not grow, because other suites pin `summary.added == 8`.
|
|
||||||
5. **D5 — no `query_log` schema change** (no new columns, no migration).
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `03_story_chat_rag` (complete) — the RAG turn + `plan_turn` this phase
|
|
||||||
modifies.
|
|
||||||
- `09_story_retrieval_quality` (complete) — hybrid retrieval +
|
|
||||||
`select_documents` (A7) whose cap this phase revises.
|
|
||||||
- `15_steering_notes` (complete) — still owns `[…truncated…]` +
|
|
||||||
`BOR_STEERING_MAX_CHARS` (shared marker; unchanged).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_remove_context_cap.md` — remove the 24k budget from
|
|
||||||
`select_documents`, `plan_turn`, `config`, `.env.example`, `README`;
|
|
||||||
rewrite the unit tests to pin *no* truncation.
|
|
||||||
2. `02_whole_doc_e2e_suite.md` — mock tail-echo trigger +
|
|
||||||
`tests/e2e/test_whole_document_context.py` (whole >24k doc, whole second
|
|
||||||
doc of a >24k pair, small-doc regression).
|
|
||||||
3. `03_story_docs_plan_commit.md` — story file, PLAN.md A7/§6/§12 revision
|
|
||||||
(owner permission 2026-08-24), full validation, one `--no-gpg-sign`
|
|
||||||
commit, phase move.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A7 revision (owner permission 2026-08-24, D1):** A7's context clause
|
|
||||||
becomes *"feed the **full text of top-N=2 documents** (deduped)"* — the
|
|
||||||
"capped at 24k chars" clause is **removed**; matched parent documents are
|
|
||||||
**never truncated**. The revision note lands in PLAN.md via task 03
|
|
||||||
(phase-17/19 precedent — recorded, not silently deviated).
|
|
||||||
- **Steering unchanged** — `BOR_STEERING_MAX_CHARS` (8 000) still caps the
|
|
||||||
`<tuning>` section with the same `[…truncated…]` marker (phase-15
|
|
||||||
behavior byte-identical).
|
|
||||||
- **A16 honored** — dedicated Playwright story suite run in isolation;
|
|
||||||
unit + integration green; `app/` coverage >90%.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- **Unit:** `tests/unit/test_retriever.py` — `select_documents` returns
|
|
||||||
byte-identical full content well past the old 24k budget, no marker;
|
|
||||||
ranking / dedup / n-cap tests unchanged.
|
|
||||||
- **Integration:** `uv run pytest tests/integration` green — existing
|
|
||||||
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
|
|
||||||
signature (no integration test pins the cap — verified).
|
|
||||||
- **Coverage:** >90% on `app/` held
|
|
||||||
(`uv run pytest --cov=app --cov-report=term-missing`).
|
|
||||||
- **E2E (new, isolated):** `tests/e2e/test_whole_document_context.py` —
|
|
||||||
the tail sentinel of a 30k-char document (and of the *second* document of
|
|
||||||
a >24k pair) appears in the rendered answer; `[…truncated…]` never
|
|
||||||
appears; the small-document grounded path is unchanged.
|
|
||||||
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `select_documents` has no budget parameter and never truncates;
|
|
||||||
`TRUNCATION_MARKER` remains for the steering section only.
|
|
||||||
- [ ] `BOR_MAX_CONTEXT_CHARS` gone from `app/config.py`, `.env.example`,
|
|
||||||
and `README.md`.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app
|
|
||||||
--cov-report=term-missing` ≥ today's number.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
|
|
||||||
green in isolation.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/whole-document-context.md` exists; PLAN.md
|
|
||||||
carries the A7 revision + §6 bullet + §12 row 24 (owner permission
|
|
||||||
2026-08-24).
|
|
||||||
- [ ] One `--no-gpg-sign` commit (below);
|
|
||||||
`.agent/phases/todo/24_whole_document_context/` moved to
|
|
||||||
`.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Commit (task 03 — after the phase dir has moved to `complete/`)
|
|
||||||
```bash
|
|
||||||
git add -f .agent/phases/complete/24_whole_document_context/ .agent/user_stories/whole-document-context.md .agent/PLAN.md
|
|
||||||
git add -A .agent/phases/todo/24_whole_document_context/ app/ README.md .env.example tests/
|
|
||||||
git commit --no-gpg-sign -m "feat(rag): feed whole matched documents to the LLM — no context truncation (A7 revised)"
|
|
||||||
```
|
|
||||||
|
|
||||||
(The *conversion* commit — this phase dir force-added under `todo/` plus
|
|
||||||
the cleared `TODO.md` — lands separately when the roadmap is written, per
|
|
||||||
the phase-20–23 precedent, commit `824914c`.)
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# Task 01 — Remove the 24k document-context budget
|
|
||||||
|
|
||||||
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
|
|
||||||
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
`select_documents` returns the full text of the top-N parent documents,
|
|
||||||
always — the budget parameter, the `max_context_chars` setting, the env var,
|
|
||||||
and the documentation lines are removed, and the unit tests pin the new
|
|
||||||
no-truncation contract (owner-confirmed D1: no cap at all — the
|
|
||||||
emergency-valve variant was explicitly rejected).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/retriever.py` — `select_documents(chunks, n=None)`: drop the
|
|
||||||
`max_chars` parameter and the whole `remaining`/truncation loop; return
|
|
||||||
the deduped top-N documents with their `content` byte-identical. Update
|
|
||||||
the module docstring (remove the "capped at ``BOR_MAX_CONTEXT_CHARS``"
|
|
||||||
sentence; state the A7 revision — whole documents, never truncated,
|
|
||||||
owner permission 2026-08-24) and the `select_documents` docstring.
|
|
||||||
`TRUNCATION_MARKER` **stays** exported — `app/rag/prompts.py`
|
|
||||||
(steering section, phase 15) still imports it.
|
|
||||||
2. `app/api/chat.py` — `plan_turn`: both `select_documents(…)` call sites
|
|
||||||
(HIGH and LOW paths) drop the `max_chars=settings.max_context_chars`
|
|
||||||
argument. No other turn-flow change (deflection, steering, `query_log`,
|
|
||||||
SSE all unchanged).
|
|
||||||
3. `app/config.py` — delete `max_context_chars: int = 24_000` and its
|
|
||||||
comment. (Any leftover `BOR_MAX_CONTEXT_CHARS=…` in an operator's
|
|
||||||
gitignored `.env` is silently ignored via `extra="ignore"` — no
|
|
||||||
migration, no startup check needed.)
|
|
||||||
4. `.env.example` — delete the `BOR_MAX_CONTEXT_CHARS=24000` line.
|
|
||||||
5. `README.md` — delete the settings-table row
|
|
||||||
`| BOR_MAX_CONTEXT_CHARS | 24000 | cap on total document text sent to the LLM |`
|
|
||||||
(line ~377). Leave the steering-notes `[…truncated…]` mention (~line 178)
|
|
||||||
alone — that budget still exists.
|
|
||||||
6. `tests/unit/test_retriever.py` — drop the `max_chars=…` argument from
|
|
||||||
every `select_documents` call; replace the three cap tests
|
|
||||||
(`test_combined_content_capped_with_truncation_marker`,
|
|
||||||
`test_single_doc_over_budget_is_truncated_to_budget`,
|
|
||||||
`test_under_budget_no_truncation`) with
|
|
||||||
`test_content_never_truncated_even_past_old_budget`: two documents of
|
|
||||||
20 000 + 15 000 chars (35 000 combined — past the old 24 000 cap) come
|
|
||||||
back with content **byte-identical** (assert equality against the
|
|
||||||
originals) and `TRUNCATION_MARKER` absent from both. Keep the module
|
|
||||||
docstring accurate ("ordering and dedup — no context cap, A7 revised").
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `uv run pytest tests/unit/test_retriever.py -v` green — ranking,
|
|
||||||
dedup, n-cap, and the new no-truncation test.
|
|
||||||
- Integration: `uv run pytest tests/integration` green — the existing
|
|
||||||
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
|
|
||||||
signature (no integration test pins the cap — verified during conversion).
|
|
||||||
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` ≥ today's
|
|
||||||
number (the >90% gate).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `rg "max_context" app/ tests/ .env.example README.md` → **no matches**
|
|
||||||
(`steering_max_chars` is a different setting and must remain).
|
|
||||||
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app
|
|
||||||
--cov-report=term-missing` ≥ today's number.
|
|
||||||
- [ ] No behavior change outside the context budget — deflection, steering,
|
|
||||||
viewer, and import behave byte-identically to before.
|
|
||||||
@@ -1,84 +0,0 @@
|
|||||||
# Task 02 — Story E2E: whole documents reach the LLM (mock tail-echo)
|
|
||||||
|
|
||||||
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"* (this task makes the no-truncation contract provable end-to-end)
|
|
||||||
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
`tests/e2e/test_whole_document_context.py` proves, with the deterministic
|
|
||||||
mock, that a matched document **larger than the old 24k cap** — and the
|
|
||||||
**second** document of a pair whose combined size exceeds it (the exact case
|
|
||||||
the old budget cut) — reach the LLM whole.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/mock_llm.py` — add one user-message trigger, same pattern as
|
|
||||||
`LONG_ANSWER_TRIGGER` and the phase-15 tuning-note echo:
|
|
||||||
- `END_OF_NOTES_TRIGGER = "show the end of your notes"` — verified
|
|
||||||
2026-08-24: no existing E2E question or fixture file contains the
|
|
||||||
phrase, so every other suite is unaffected.
|
|
||||||
- In `compose_answer`, after the `DEFLECT_MODE` check and before the
|
|
||||||
generic branch: when the trigger is present in the user message, the
|
|
||||||
answer quotes the **tail of the context** — e.g.
|
|
||||||
`f"…and the very end of my notes reads: “{_context(body)[-160:]}” (Deterministic mock answer for E2E.)"`.
|
|
||||||
Byte-stable across runs, so a sentinel placed at the *end* of a
|
|
||||||
document appears in the rendered answer **iff the whole document was
|
|
||||||
in the prompt**. (The tail includes the closing `</documents>` —
|
|
||||||
harmless for `to_contain_text` sentinel assertions.)
|
|
||||||
- Keep the mock docstring's trigger list updated.
|
|
||||||
2. `tests/e2e/test_whole_document_context.py` — new story suite. Reuse the
|
|
||||||
`page`, `app_url`, `mock_llm`, `db_ready` fixtures and the
|
|
||||||
TRUNCATE-then-seed pattern from `tests/e2e/test_chat_rag.py` /
|
|
||||||
`tests/integration/test_document_content.py`. **Seed the big documents
|
|
||||||
directly via SQLAlchemy** (a `documents` row + 2–3 `chunks` rows, each
|
|
||||||
chunk with `embedding = embed_text(chunk_text)` imported from
|
|
||||||
`tests.e2e.mock_llm` — the same deterministic bag-of-words vector the
|
|
||||||
mock computes, so the question's live mock embedding genuinely overlaps).
|
|
||||||
**Do not add fixture files** — `tests/fixtures/docs/` must stay at its
|
|
||||||
current 8 files; other suites pin `summary.added == 8`.
|
|
||||||
- Content helper: a ~30 000-char document = repeated
|
|
||||||
"gitlab install playbook" body (tokens shared with the question →
|
|
||||||
hybrid hit) whose **last line is a unique sentinel**, e.g.
|
|
||||||
`WHOLE-DOC-TAIL-GITLAB-<test-unique>`.
|
|
||||||
- `test_whole_document_over_old_cap_reaches_llm` — seed one 30 000-char
|
|
||||||
document (past the old 24 000 cap); ask
|
|
||||||
`"Show the end of your notes about the gitlab install playbook, please."`;
|
|
||||||
assert the rendered brain bubble contains the tail sentinel **and**
|
|
||||||
`"Deterministic mock answer for E2E"`, contains **no**
|
|
||||||
`[…truncated…]`, the document's `.source-chip` renders, and the
|
|
||||||
`query_log` row has `deflected == False`.
|
|
||||||
- `test_second_document_of_over_cap_pair_reaches_llm` — seed two
|
|
||||||
documents of ~16 000 chars each (32 000 combined — under the old
|
|
||||||
budget the lower-ranked one was truncated in place). Doc 1's first
|
|
||||||
chunk carries the question's key tokens ("gitlab install playbook");
|
|
||||||
doc 2's first chunk carries a weaker overlap ("playbook", "notes") so
|
|
||||||
doc 1 ranks first and doc 2 is the **last block inside
|
|
||||||
`<documents>`** — i.e. the tail-echo surfaces doc 2's sentinel. Assert
|
|
||||||
doc 2's tail sentinel in the answer. If the ranking ever flips locally,
|
|
||||||
strengthen doc 1's token overlap (repeat the question phrase in its
|
|
||||||
first chunk) until two consecutive isolated runs are stable — do not
|
|
||||||
weaken the assertion.
|
|
||||||
- `test_small_document_path_unchanged` — regression: re-import the
|
|
||||||
standard fixtures via the real importer (the `test_chat_rag.py`
|
|
||||||
`_import_fixtures` pattern), ask the usual on-topic question
|
|
||||||
(`"How is my Kubernetes cluster set up?"`) → grounded answer +
|
|
||||||
`kubernetes.md` chip, no `[…truncated…]` (the under-24k path is
|
|
||||||
byte-identical to before).
|
|
||||||
- Header comment: story path + run-in-isolation command
|
|
||||||
(`uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
|
|
||||||
green **in isolation** (DB up: `podman compose up -d db`).
|
|
||||||
- E2E regressions: `uv run pytest tests/e2e/test_chat_rag.py
|
|
||||||
tests/e2e/test_honest_deflection.py -v --no-cov` still green (the mock
|
|
||||||
change is additive — the trigger phrase appears in no existing question).
|
|
||||||
- Coverage: the >90% `app/` gate is unaffected (tests + mock only) — re-run
|
|
||||||
`uv run pytest --cov=app --cov-report=term-missing` to prove it.
|
|
||||||
- Lint/types: `uv run ruff check . && uv run pyright` clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
|
|
||||||
— all three tests green in isolation.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_chat_rag.py
|
|
||||||
tests/e2e/test_honest_deflection.py -v --no-cov` green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `tests/fixtures/docs/` unchanged (still 8 files).
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
# Task 03 — Story file, PLAN.md revision, validation, commit
|
|
||||||
|
|
||||||
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
|
|
||||||
**Story:** `.agent/user_stories/whole-document-context.md` (created by this task)
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Record the user story and the A7 revision (owner permission 2026-08-24),
|
|
||||||
run the full validation gate, land one atomic commit, and move the phase
|
|
||||||
to `complete/`.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `.agent/user_stories/whole-document-context.md` — story file in house
|
|
||||||
style (cf. `.agent/user_stories/sources-midstream.md`):
|
|
||||||
- Header: `**Phase:** 24_whole_document_context · **E2E:**
|
|
||||||
tests/e2e/test_whole_document_context.py`.
|
|
||||||
- Bug report section: the two TODO.md items (L3–L4), verbatim.
|
|
||||||
- Narrative: *Given* a question whose chunk matches a document,
|
|
||||||
*when* the turn assembles context, *then* the LLM receives the
|
|
||||||
**entire** parent document — never a `[…truncated…]`-cut version —
|
|
||||||
so the answer is grounded in the whole note.
|
|
||||||
- Owner-confirmed (2026-08-24, roadmap D1–D5) — the five decisions from
|
|
||||||
the phase overview, verbatim.
|
|
||||||
- Acceptance criteria: no budget parameter in `select_documents`;
|
|
||||||
`BOR_MAX_CONTEXT_CHARS` gone from settings/env/README; the story E2E's
|
|
||||||
three tests green in isolation; steering-note truncation (phase 15)
|
|
||||||
unchanged; unit+integration green, coverage >90%, one
|
|
||||||
`--no-gpg-sign` commit.
|
|
||||||
- Playwright Mapping Rule: the three tests of task 02, one line each.
|
|
||||||
2. `.agent/PLAN.md` revisions (phase-17/19 precedent — record the owner
|
|
||||||
permission, never deviate silently):
|
|
||||||
- §2, A7 row: replace *"feed the **full text of top-N=2 documents**
|
|
||||||
(deduped, capped at 24k chars)"* with *"feed the **full text of
|
|
||||||
top-N=2 documents** (deduped)"* and append the rationale addendum:
|
|
||||||
"A7 revised 2026-08-24 — matched documents never truncated (owner:
|
|
||||||
'this should never happen'; emergency-valve variant rejected)".
|
|
||||||
- §2, revision-note block under the anchors table: add
|
|
||||||
**A7 revision (phase 24, owner permission 2026-08-24):** the
|
|
||||||
`[…truncated…]` cap on document context is removed —
|
|
||||||
`select_documents` always returns the full top-N texts;
|
|
||||||
`BOR_MAX_CONTEXT_CHARS` is gone. The steering section
|
|
||||||
(`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
|
|
||||||
marker.
|
|
||||||
- §6, retrieval bullet: *"top 2 → full content, concatenated, truncated
|
|
||||||
to `BOR_MAX_CONTEXT_CHARS` (24k) with a `[…truncated…]` marker"* →
|
|
||||||
*"top 2 → full content, concatenated — **never truncated** (A7
|
|
||||||
revised, phase 24, owner permission 2026-08-24)"*.
|
|
||||||
- §12 roadmap table: add row 24 —
|
|
||||||
`| 24 | 24_whole_document_context.md | whole-document-context.md | tests/e2e/test_whole_document_context.py |`
|
|
||||||
plus a footnote: "Row 24 added 2026-08-24 with owner permission —
|
|
||||||
A7's 24k context cap removed (documents are never truncated)".
|
|
||||||
3. Final validation (AGENTS.md §9 — all must be green before the commit):
|
|
||||||
- `uv run pytest` (unit + integration)
|
|
||||||
- `uv run pytest --cov=app --cov-report=term-missing` (>90% gate)
|
|
||||||
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
|
|
||||||
(in isolation)
|
|
||||||
- `uv run ruff check . && uv run pyright`
|
|
||||||
4. Commit + phase move (AGENTS.md §8; use the commit block in
|
|
||||||
`00_phase.md` — it assumes the move happens first):
|
|
||||||
`mv .agent/phases/todo/24_whole_document_context
|
|
||||||
.agent/phases/complete/`, then the two `git add` lines + one
|
|
||||||
`--no-gpg-sign` commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- All AGENTS.md §9 gates: unit + integration green, `app/` coverage >90%,
|
|
||||||
story E2E green in isolation, lint + types clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `.agent/user_stories/whole-document-context.md` exists (story,
|
|
||||||
D1–D5, acceptance criteria, Playwright mapping).
|
|
||||||
- [ ] `.agent/PLAN.md` carries the A7 row + revision note (§2), the §6
|
|
||||||
bullet, and the §12 row 24 + footnote — every one marked
|
|
||||||
"owner permission 2026-08-24".
|
|
||||||
- [ ] All validation commands green (work step 3).
|
|
||||||
- [ ] One `--no-gpg-sign` commit;
|
|
||||||
`.agent/phases/complete/24_whole_document_context/` exists and
|
|
||||||
`.agent/phases/todo/24_whole_document_context/` is gone.
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
# Task 01 — Still Background: Static Grid + Three Opacity-Only Glow Fades
|
|
||||||
|
|
||||||
**Phase:** `25_background_no_motion` · **Story:** `.agent/user_stories/background-no-motion.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Redesign the background block of `frontend/assets/styles.css` to the
|
|
||||||
owner's spec (no movement; different bright spots slowly fading in and
|
|
||||||
out), and pin the new contract at source level.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
0. **Capture the "before" evidence FIRST** (pre-change): with the DB up
|
|
||||||
(`podman compose up -d db`), boot the app the same way
|
|
||||||
`tests/e2e/conftest.py`'s `app_server` fixture does (uvicorn
|
|
||||||
`app.main:app` on a free port, wait for it to answer), and with a small
|
|
||||||
throwaway Playwright script screenshot the `/` page at 1280×800 →
|
|
||||||
`.agent/screenshots/25_background_no_motion/before.png`, then again
|
|
||||||
~4s later → `before_4s.png` (the pair shows the down-right jitter +
|
|
||||||
the uniform brightness pulse). Kill the server when done.
|
|
||||||
1. `frontend/assets/styles.css` — **grid layer `body::before`:** remove
|
|
||||||
the `animation: bg-grid-drift 60s linear infinite;` declaration and
|
|
||||||
delete the whole `@keyframes bg-grid-drift { … }` block. Keep the 44px
|
|
||||||
cells, the 60% `--line` alpha 1px lines, and the widened radial mask —
|
|
||||||
the grid remains as a **static texture**. Update the block's comment:
|
|
||||||
drift removed (owner 2026-08-25) — the 0.73px/s sub-pixel drift
|
|
||||||
rasterizes as a once-per-second down-right jitter; the owner wants no
|
|
||||||
movement.
|
|
||||||
2. `frontend/assets/styles.css` — **glow layers.** Replace the
|
|
||||||
whole-layer breathe with three independent spot layers:
|
|
||||||
- `body::after` — keep the phase-08 indigo spot exactly:
|
|
||||||
`background-image: radial-gradient(circle 56rem at 12% 8%,
|
|
||||||
rgb(109 120 242 / 0.14), transparent 62%);` and
|
|
||||||
`animation: bg-glow-a 26s ease-in-out infinite;`
|
|
||||||
- **new `html::before`** — the phase-08 cyan spot:
|
|
||||||
`background-image: radial-gradient(circle 60rem at 88% 92%,
|
|
||||||
rgb(34 211 238 / 0.10), transparent 62%);` and
|
|
||||||
`animation: bg-glow-b 34s ease-in-out -12s infinite;`
|
|
||||||
- **new `html::after`** — a third indigo spot:
|
|
||||||
`background-image: radial-gradient(circle 52rem at 14% 86%,
|
|
||||||
rgb(109 120 242 / 0.09), transparent 62%);` and
|
|
||||||
`animation: bg-glow-c 42s ease-in-out -23s infinite;`
|
|
||||||
- All three layers (and the grid layer) must declare:
|
|
||||||
`content: ""; position: fixed; inset: 0; z-index: -1;
|
|
||||||
pointer-events: none;` — no `transform`, no `background-position`,
|
|
||||||
no `filter` on any of them.
|
|
||||||
- Delete `@keyframes bg-glow-breathe { … }` and add the three
|
|
||||||
**opacity-only** keyframe blocks (nothing else may appear in any
|
|
||||||
`bg-*` keyframe):
|
|
||||||
```css
|
|
||||||
@keyframes bg-glow-a { 0%, 100% { opacity: 0.25; } 50% { opacity: 1; } }
|
|
||||||
@keyframes bg-glow-b { 0%, 100% { opacity: 0.20; } 50% { opacity: 1; } }
|
|
||||||
@keyframes bg-glow-c { 0%, 100% { opacity: 0.15; } 50% { opacity: 1; } }
|
|
||||||
```
|
|
||||||
3. `frontend/assets/styles.css` — **reduced motion:** in the existing
|
|
||||||
`@media (prefers-reduced-motion: reduce)` block that stills the
|
|
||||||
background (currently `body::before, body::after { animation: none; }`,
|
|
||||||
right after the spinner block), extend the selector list to all four
|
|
||||||
layers: `body::before, body::after, html::before, html::after
|
|
||||||
{ animation: none; }`. Do not touch the typing/spinner/thinking
|
|
||||||
reduced-motion blocks.
|
|
||||||
4. `frontend/assets/styles.css` — **section comment:** rewrite the
|
|
||||||
"Animated background" comment to document the new spec: no movement
|
|
||||||
(owner 2026-08-25); three independent soft spots, opacity-only fades
|
|
||||||
at 26/34/42s with negative delays → out of phase, so the total light
|
|
||||||
fluxuates smoothly and irregularly; `html::before`/`html::after` are
|
|
||||||
background layers (root stacking context: they paint above the
|
|
||||||
`var(--bg)` canvas and below the transparent, non-stacking `<body>`'s
|
|
||||||
content — the no-occlusion contract is unchanged).
|
|
||||||
5. **New `tests/unit/test_background_no_motion.py`** (repo source-pin
|
|
||||||
pattern — reuse the helpers from `tests/unit/test_background_animation.py`
|
|
||||||
(`_css`, `_css_no_comments`, `_rule_block`; note `_rule_block` matches
|
|
||||||
top-level `selector { … }`, which works for `html::before`/`html::after`
|
|
||||||
as written in step 2). Pin, at minimum:
|
|
||||||
- `body::before` carries **no `animation`** declaration;
|
|
||||||
`@keyframes bg-grid-drift` is absent from the file; the grid keeps
|
|
||||||
its static texture (`background-size: 44px 44px`, the two 60%-alpha
|
|
||||||
1px line gradients, the widened mask, both mask properties).
|
|
||||||
- `body::after` runs `bg-glow-a 26s ease-in-out infinite`;
|
|
||||||
`html::before` runs `bg-glow-b 34s ease-in-out -12s infinite`;
|
|
||||||
`html::after` runs `bg-glow-c 42s ease-in-out -23s infinite`;
|
|
||||||
each glow layer's `background-image` is exactly the single radial
|
|
||||||
gradient from step 2 (color, radius, position, 62% transparent stop).
|
|
||||||
- **No movement:** parse every `@keyframes bg-*` rule in the file —
|
|
||||||
the set of declared property names across all keyframe frames is
|
|
||||||
exactly `{opacity}` (no `transform`, `scale`, `background-position`,
|
|
||||||
…); none of the three glow layers declares `transform` or
|
|
||||||
`background-position`.
|
|
||||||
- The three glow durations are distinct and each ≥ 20s (slow).
|
|
||||||
- All four pseudo-layers: `position: fixed`, `inset: 0`,
|
|
||||||
`z-index: -1`, `pointer-events: none`, `content: ""`.
|
|
||||||
- Plumbing: `html` keeps `background: var(--bg)`; `body` keeps
|
|
||||||
`background: transparent` and declares none of `z-index`,
|
|
||||||
`transform`, `opacity`, `filter`.
|
|
||||||
- The reduced-motion block stills all four layers (all four selectors
|
|
||||||
present together with `animation: none`).
|
|
||||||
- No `filter` in any background layer block; no `blur` anywhere in
|
|
||||||
the file (comments stripped).
|
|
||||||
6. **Adapt `tests/unit/test_background_animation.py`** (the phase-22
|
|
||||||
source pins) to the new contract so the whole unit suite is green:
|
|
||||||
replace the grid-drift and breathe tests with their new-contract
|
|
||||||
equivalents (where a check is already covered by the new file, keep the
|
|
||||||
file self-contained rather than importing from it — the repo pattern is
|
|
||||||
local pins); keep the generic layer-plumbing tests
|
|
||||||
(`test_both_layers_are_fixed_zminus1_noninteractive` — extend it to
|
|
||||||
cover `html::before`/`html::after` — and
|
|
||||||
`test_html_owns_bg_and_body_stays_transparent`) and the no-blur/no-JS
|
|
||||||
anchor test; update the module docstring to describe the phase-25
|
|
||||||
design and point at `.agent/user_stories/background-no-motion.md`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- `uv run pytest tests/unit -v` green (new + adapted pins).
|
|
||||||
- `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ the
|
|
||||||
pre-change number (`app/` is untouched — the >90% gate holds).
|
|
||||||
- `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- Coverage **>90%** on new/modified code: the functional change is CSS;
|
|
||||||
the new/modified Python is pytest source pins, exercised in full.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `styles.css`: no `bg-grid-drift`, no `bg-glow-breathe`, no
|
|
||||||
`animation` on `body::before`; exactly three opacity-only
|
|
||||||
`bg-glow-a/b/c` cycles on `body::after`, `html::before`,
|
|
||||||
`html::after` (26s/34s/42s, delays 0/−12s/−23s); reduced-motion
|
|
||||||
stills all four layers; comments document the owner's no-movement
|
|
||||||
spec.
|
|
||||||
- [ ] `uv run pytest tests/unit` green with
|
|
||||||
`tests/unit/test_background_no_motion.py` present and passing.
|
|
||||||
- [ ] `uv run pytest --cov=app` TOTAL ≥ pre-change; ruff + pyright clean.
|
|
||||||
- [ ] `.agent/screenshots/25_background_no_motion/before.png` and
|
|
||||||
`before_4s.png` captured **before** the CSS change.
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
# Phase 26 — Document Modal Viewer
|
|
||||||
|
|
||||||
**Source:** `TODO.md L4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
|
|
||||||
**Story:** `.agent/user_stories/document-modal.md`
|
|
||||||
**Context:** Phase 10 added the separate `/document.html` viewer page; phase 19 added the shared header bar that now lives on every page. The document content is served by the stateless `GET /api/documents/content` endpoint (PLAN §4).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Stop opening cited documents in a new page/tab. Clicking a source chip or a Sources-table path link now opens the document in an **almost-fullscreen modal overlay** on the current page, fed by the same `/api/documents/content` endpoint. The existing `/document.html` page stays as the no-JS / direct-link fallback and its behaviour is unchanged.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `10_story_document_viewer` (complete) — the `/document.html` page, the `document.js` renderer, the `renderMarkdown` escape-first renderer in `markdown.js`, and the `#doc-content` / `.doc-md` / `.doc-raw` markup this phase reuses inside the modal.
|
|
||||||
- `19_shared_header` (complete) — the shared header bar the modal sits under; the modal must not disturb the header.
|
|
||||||
- `08_story_dark_tech_theme` (complete) — the Phase-08 tokens and the ≥4.5:1 contrast / `prefers-reduced-motion` contract the modal must honour.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_modal_css_and_html.md` — the modal CSS (overlay, backdrop, close button, scrollable content area) + inject the modal skeleton into `index.html`
|
|
||||||
2. `02_app_js_modal_intercept.md` — intercept document links in `app.js` + `sources.js`, fetch content via `/api/documents/content`, render inside the modal
|
|
||||||
3. `03_document_js_modal_mode.md` — adapt `document.js` to optionally render in modal mode (reuse the same API call) for the direct-link fallback path
|
|
||||||
4. `04_e2e_regression_suite.md` — update `test_document_viewer.py` to verify modal behaviour; the story gate, run in isolation
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: none required for the modal itself (frontend-only); the `/api/documents/content` endpoint is unchanged (no `app/` change → no coverage delta).
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
- E2E: `tests/e2e/test_document_viewer.py` rewritten for the modal contract (task 4), green **in isolation** (prereq `podman compose up -d db`).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Clicking a source chip (chat) or a Sources-table path link opens the document in an almost-fullscreen modal on the **same page** (no new tab, no navigation).
|
|
||||||
- [ ] The modal renders the same content the `/document.html` page renders: md/markdown via the shared renderer (`.doc-md`), other formats in `<pre class="doc-raw">`, source/format/path/indexed/chunks meta.
|
|
||||||
- [ ] The modal has a visible close control, closes on Escape, closes on backdrop click, and keeps the dark theme + a11y frame (skip-link, focus trap, `:focus-visible`, aria-label).
|
|
||||||
- [ ] The existing `/document.html` page still works unchanged (direct link, back button, XSS-safe rendering, not-found state).
|
|
||||||
- [ ] No CDN tags on any touched page; every asset reference is same-origin or `data:`.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (gate >90%).
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): modal content uses the standard centered column width for md; backdrop behind content; no 360px overflow.
|
|
||||||
- [ ] `.agent/user_stories/document-modal.md` exists.
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/26_document_modal_viewer/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **No backend change** — the modal reuses `GET /api/documents/content` unchanged (A10 untouched: the API stays stateless).
|
|
||||||
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, zero new packages, no new assets, system font stack; the modal is pure CSS + JS.
|
|
||||||
- **No anchor revised** — this is a UI-behaviour change (PLAN §7.5 gains `#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-close`, `#doc-modal-content`); the `/document.html` page and its story are unchanged.
|
|
||||||
- **A16 honoured** — one story E2E suite (rewritten) + adapted regressions.
|
|
||||||
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# Task 01 — Modal CSS + HTML skeleton
|
|
||||||
|
|
||||||
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
|
|
||||||
**Story:** `.agent/user_stories/document-modal.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the modal markup to `index.html` and the CSS that styles an almost-fullscreen overlay (backdrop + panel + close button + scrollable content) using the Phase-08 tokens.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/index.html` — insert the modal skeleton just before the closing `</body>` (after the existing script tags, or before them — order doesn't matter for a static skeleton). The skeleton:
|
|
||||||
```html
|
|
||||||
<div class="doc-modal" id="doc-modal" hidden>
|
|
||||||
<div class="doc-modal-backdrop" id="doc-modal-backdrop" aria-hidden="true"></div>
|
|
||||||
<div class="doc-modal-panel" role="dialog" aria-modal="true" aria-labelledby="doc-modal-title" aria-describedby="doc-modal-desc">
|
|
||||||
<header class="doc-modal-header">
|
|
||||||
<h2 class="doc-modal-title" id="doc-modal-title">Loading…</h2>
|
|
||||||
<div class="doc-modal-actions">
|
|
||||||
<a class="doc-modal-open" id="doc-modal-open" target="_blank" rel="noopener" hidden aria-label="Open in full page">
|
|
||||||
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/><path d="M15 3h6v6"/><path d="M10 14 21 3"/></svg>
|
|
||||||
<span>Full page</span>
|
|
||||||
</a>
|
|
||||||
<button type="button" class="doc-modal-close" id="doc-modal-close" aria-label="Close document">
|
|
||||||
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 6 6 18M6 6l12 12"/></svg>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</header>
|
|
||||||
<div class="doc-modal-meta" id="doc-modal-meta" aria-live="polite"></div>
|
|
||||||
<p class="visually-hidden" id="doc-modal-desc" role="status">Document content is loading.</p>
|
|
||||||
<main class="doc-modal-content" id="doc-modal-content" tabindex="-1">
|
|
||||||
<p class="doc-modal-loading" role="status">Loading document…</p>
|
|
||||||
</main>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
```
|
|
||||||
The `hidden` attribute keeps the modal off until JS opens it. The `#doc-modal-open` "Full page" link points at the same `/document.html?source=…&path=…&modal=…` URL the modal will build so a user can still open the dedicated page if JS is off.
|
|
||||||
2. `frontend/assets/styles.css` — add a `--doc-modal-*` block (Phase-08 tokens). Styling contract:
|
|
||||||
- `.doc-modal` — `position: fixed; inset: 0; z-index: 1000;` (above the shared header and every page layer, below the phase-25 background which is `z-index: -1`); the panel is flex, column; the backdrop + panel fill the viewport.
|
|
||||||
- `.doc-modal-backdrop` — `position: fixed; inset: 0; background: rgba(10,14,23,0.82);` backdrop blur is **not** used (phase-08 no-blur perf anchor); `opacity` transition 120ms.
|
|
||||||
- `.doc-modal-panel` — `display: flex; flex-direction: column; width: min(1100px, 96vw); height: 92vh; margin: auto; background: var(--surface, #121a2e); border: 1px solid var(--line, #232b52); border-radius: 12px; box-shadow: 0 24px 80px rgba(0,0,0,.55);` — "almost-fullscreen" = 96vw × 92vh, centered.
|
|
||||||
- `.doc-modal-header` — sticky top, same height/spacing as the doc header (64px / 58px pins from phase 12); title uses `--ink`; close button ≥44px target, focus-visible ring.
|
|
||||||
- `.doc-modal-content` — `flex: 1; overflow: auto;` (vertical scroll inside the panel, not the viewport); padding; the md content reuses `.doc-md` (≤46rem centered column) — the modal just provides the scroll container. For wide raw formats the `.doc-raw` pre already has `overflow-x: auto`.
|
|
||||||
- `.doc-modal-meta` — reuses the `.doc-meta` styling already defined for the viewer page (source/format/path/indexed/chunks badges); keep it compact (single row, wrap).
|
|
||||||
- `.doc-modal-close` — icon-only button, `aria-label` kept, `:focus-visible` 3px ring.
|
|
||||||
- Transitions respect `prefers-reduced-motion: reduce` (no opacity/transform animation, or `animation: none` under the reduced-motion media query — same pattern as the phase-25 background layers).
|
|
||||||
- `.doc-modal[hidden]` — `display: none` (the `hidden` IDL attribute default already hides it; add the rule to be explicit and testable).
|
|
||||||
- Ensure the modal panel does not add horizontal width at 360px (no `box-sizing` surprises; the panel is `96vw` ≤ viewport).
|
|
||||||
3. Verify the new CSS classes do not collide with any existing selector in `styles.css` (grep for `.doc-modal`, `.doc-modal-`).
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The modal panel is `96vw × 92vh` ("almost-fullscreen"). If the owner wants a different fraction, that's a follow-up.
|
|
||||||
- The "Full page" link is admin-agnostic (it just opens `/document.html`); it is shown for everyone since the viewer is public.
|
|
||||||
- The modal uses the existing `.doc-meta` badge classes already defined for the viewer page (no duplicate styling).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No unit/integration test for static CSS/HTML.
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `index.html` contains the `.doc-modal` skeleton with the documented ids (`#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-panel`, `#doc-modal-close`, `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`, `#doc-modal-open`).
|
|
||||||
- [ ] The modal CSS block is present, uses Phase-08 tokens, has no `filter: blur`/`backdrop-filter`, and the panel is `96vw × 92vh` centered.
|
|
||||||
- [ ] No selector collision (grep clean).
|
|
||||||
- [ ] `prefers-reduced-motion` stills any modal transition.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Task 02 — Intercept document links → modal
|
|
||||||
|
|
||||||
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
|
|
||||||
**Story:** `.agent/user_stories/document-modal.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Intercept document links on the chat page (`app.js` source chips) and the Sources page (`sources.js` table links): instead of navigating to `/document.html` in a new tab, fetch the document via `GET /api/documents/content` and render it inside the modal from task 01.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/document.js` — extract the rendering logic into a reusable, importable function so both the standalone page (task 03) and the modal share the exact same renderer. Specifically:
|
|
||||||
- Export `renderDocument(doc, { containerEl, metaEl, titleEl })` that populates a title element, a meta element (the `.doc-meta` badges), and a content element (`.doc-md` for markdown via `renderMarkdown`, `<pre class="doc-raw">` otherwise). Keep the escape-first XSS-safety contract (`innerHTML` only through `renderMarkdown`; `textContent` for raw + badges).
|
|
||||||
- The standalone `document.html` page keeps its own `load()` that calls `renderDocument` with its page elements (task 03 wires this).
|
|
||||||
2. `frontend/assets/app.js` — add a `openDocumentModal(source, path)` helper:
|
|
||||||
- Build the modal URL: `/api/documents/content?source=…&path=…` (same encoding the chips already use).
|
|
||||||
- Show the modal: set `#doc-modal.hidden = false`, set the loading state, move focus into `#doc-modal-content` (a11y — the panel is `tabindex="-1"`).
|
|
||||||
- `fetch(contentUrl)` → on `!r.ok` render a short "document not found" line in the content area; on success call `renderDocument` with `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`.
|
|
||||||
- The "Full page" link (`#doc-modal-open`) is set to the `/document.html?source=…&path=…` URL on open.
|
|
||||||
- Keep the existing `documentUrl()` builder for the "Full page" link (unchanged output).
|
|
||||||
- Add modal close behaviour: `#doc-modal-close` click → `closeDocumentModal()`; backdrop click → close; `Escape` key → close; closing restores focus to the link that opened the modal (best-effort — store the triggering element).
|
|
||||||
3. `frontend/assets/app.js` — wire the source chips: replace `chip.target = "_blank"` navigation with `chip.addEventListener("click", e => { e.preventDefault(); e.stopPropagation(); openDocumentModal(s.source, s.path, chip); })`. Keep the `title`/aria-label truncation logic the chips already have. The chip keeps its `href` too (no-JS fallback would navigate to `/document.html`).
|
|
||||||
4. `frontend/assets/sources.js` — wire the table links the same way: the `.doc-link` click is intercepted, `preventDefault`, and `openDocumentModal(d.source, d.path, link)` is called. Since `openDocumentModal` lives in `app.js` (the chat page module) and `sources.js` is a separate module, **export** `openDocumentModal` from `app.js` and import it in `sources.js` — but `app.js` is loaded as a module on the chat page only. To avoid a second module instance, move the shared modal logic into a small new module `frontend/assets/document-modal.js` (task 02 step 1 refined below) and have both `app.js` and `sources.js` import it.
|
|
||||||
- **Refined split:** create `frontend/assets/document-modal.js` exporting `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()`. This module owns the modal DOM wiring (close on ESC / backdrop / button, focus management) and the `fetch` + `renderDocument` call. `app.js` and `sources.js` just call `openDocumentModal(...)` from their click handlers. This is the cleanest single-implementation approach (mirrors how `header.js` is the single owner of the shared header).
|
|
||||||
- `document.js` (standalone page) also imports `renderDocument` from itself (or a shared `document-render.js`) — keep the standalone page self-contained; it doesn't need the modal module.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The modal module (`document-modal.js`) is a classic or module script loaded on both `index.html` and `sources.html`. It's a module (imports `renderDocument` from `document.js`), so both pages must load it via `<script type="module">`. `document.js` will export `renderDocument`.
|
|
||||||
- Close-on-`Escape` and close-on-backdrop are modal UX standards; the owner's item says "modal, not a new page", which implies standard modal affordances.
|
|
||||||
- The "Full page" link remains for users who want the dedicated viewer; it is optional and doesn't interfere with the modal.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No unit/integration test (frontend-only).
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Clicking a source chip or a Sources-table path link opens the modal and renders the document (md via `.doc-md`, other formats via `.doc-raw`).
|
|
||||||
- [ ] The modal closes on button click, on backdrop click, and on `Escape`; focus returns to the triggering control.
|
|
||||||
- [ ] No new tab opens from either link type.
|
|
||||||
- [ ] The "Full page" link still navigates to `/document.html` (unchanged).
|
|
||||||
- [ ] XSS-safe rendering preserved (markdown escaped, raw set via `textContent`).
|
|
||||||
- [ ] Both pages load the modal module without a duplicate-module error.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Task 03 — Standalone viewer page reuses the shared renderer
|
|
||||||
|
|
||||||
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
|
|
||||||
**Story:** `.agent/user_stories/document-modal.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Keep `/document.html` working exactly as before (it is the no-JS / direct-link fallback) but refactor its `document.js` so the markdown/raw rendering lives in a shared function the modal module can reuse. No behavioural change to the standalone page.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/document.js` — split the current inline renderer into an exported `renderDocument(doc, { titleEl, metaEl, contentEl })` function (the escape-first contract: markdown → `renderMarkdown` into a `.doc-md` div; other formats → `<pre class="doc-raw">` via `textContent`; badges via `textContent`). The page's existing `load()` IIFE now calls `renderDocument` with the page's `#doc-title`, `#doc-meta`, `#doc-content` elements. Everything else in `document.js` (query-param parsing, `back` target, not-found card, shared header wiring, New Chat button, `mainEl.focus()`) is **unchanged**.
|
|
||||||
2. `frontend/assets/document-modal.js` (new) — imports `renderDocument` from `./document.js`. Owns `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()` (see task 02). On open it fetches `/api/documents/content` and calls `renderDocument(doc, { titleEl: #doc-modal-title, metaEl: #doc-modal-meta, contentEl: #doc-modal-content })`. It also sets `#doc-modal-open.href` to the `/document.html?source=…&path=…` URL.
|
|
||||||
3. `frontend/index.html` — load `document-modal.js` as a module (add `<script type="module" src="/assets/document-modal.js"></script>` alongside the existing `app.js` module script). `index.html` already loads `markdown.js` as a classic script (needed by `renderDocument`).
|
|
||||||
4. `frontend/sources.html` — load `document-modal.js` as a module (it needs `document.js`'s `renderDocument`, so both `document.js` and `document-modal.js` must be module scripts; `markdown.js` classic script stays). The Sources page currently loads `sources.js` as a module; add the modal module script next to it.
|
|
||||||
5. Verify the no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes — the new module scripts are same-origin `<script src>`, so they satisfy the "local asset" rule. If the test counts script tags, update the expected count.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- `renderDocument` depends on `renderMarkdown` (from `markdown.js`), which is a classic script — so `document.js` (module) importing nothing but using the global `renderMarkdown` is fine, and `document-modal.js` (module) importing `renderDocument` from `document.js` also relies on the global `renderMarkdown` being present. Both pages load `markdown.js` before the module scripts (hoisting guarantees module scripts run after classic scripts already on the page).
|
|
||||||
- The standalone page's `document.js` no longer needs to be a module for its own rendering — but it stays a module because it imports `header.js` (shared header). Keep it a module.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No unit/integration test for the refactor itself.
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `/document.html?source=…&path=…` still renders title/meta/content exactly as before (md in `.doc-md`, raw in `<pre.doc-raw>`).
|
|
||||||
- [ ] The not-found state, `back` target, shared header, and New Chat button on `/document.html` are unchanged.
|
|
||||||
- [ ] `document-modal.js` is loaded on `index.html` and `sources.html`; `document.js` exports `renderDocument`.
|
|
||||||
- [ ] No-CDN test still passes (new scripts are same-origin).
|
|
||||||
- [ ] No console errors on any of the three pages.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Task 04 — E2E story suite (modal) + regressions
|
|
||||||
|
|
||||||
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
|
|
||||||
**Story:** `.agent/user_stories/document-modal.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Rewrite the phase-10 E2E suite to assert the **modal** contract (open in a modal on the same page, no new tab; close on button/Escape/backdrop; dark theme; no CDN; a11y frame), and confirm the standalone `/document.html` page still works.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_document_viewer.py` — rewrite for the modal contract (the seeding harness from the phase-10 file — `_import_fixtures` / `_reset_db` / `_run_in_thread` — stays identical; only the assertions change):
|
|
||||||
- `test_source_chip_opens_modal` — from the chat page, ask the QUESTION, wait for the `kubernetes.md` source chip, click it (no `target=_blank` click → `expect_popup`); assert the modal `.doc-modal` is visible, NOT hidden; `#doc-modal-title` = "Kubernetes Homelab Cluster"; `#doc-content`/`.doc-md` present; content text "Talos Linux on three nodes". Assert the page URL is unchanged (still `/`).
|
|
||||||
- `test_sources_row_opens_modal` — log in, find the `gitlab-compose.yaml` row link, click it; assert the modal is open with the yaml rendered in `<pre.doc-raw>` containing "gitlab/gitlab-ce:17.2.1-ce.0", mono font.
|
|
||||||
- `test_modal_closes_on_button_escape_and_backdrop` — open the modal, click `#doc-modal-close` → hidden; re-open, click backdrop → hidden; re-open, press Escape → hidden.
|
|
||||||
- `test_modal_focus_and_a11y` — on open, focus is inside `#doc-modal-content`; the panel has `role="dialog"` + `aria-modal="true"`; the close button has `aria-label`.
|
|
||||||
- `test_modal_xss_safe` — seed an XSS fixture doc, open via modal, assert the `<script>` shows as escaped text and no dialog fires (same as the phase-10 test but inside the modal).
|
|
||||||
- `test_standalone_page_still_works` — the phase-10 assertions for `/document.html` (title/content/format badge, not-found state, dark theme, no-CDN, a11y frame, `#doc-content .doc-md` ≤ 736px) are **kept** — the dedicated page must not regress.
|
|
||||||
- `test_modal_theme_and_no_cdn` — dark theme (document background `rgb(10,14,23)`), and the modal panel uses Phase-08 surface colour.
|
|
||||||
2. `tests/integration/test_api.py` — if the no-CDN test counts `<script>` tags on `index.html` / `sources.html`, bump the expected count to include `document-modal.js` (and confirm `document.html` count is unchanged).
|
|
||||||
3. Regressions to run green in isolation after the change: `test_document_back_navigation.py` (source chips now open a modal; verify the back-navigation story doesn't assert a new tab — if it does, adapt), `test_header_consistency.py` (new module scripts don't disturb the header), `test_smoke.py`.
|
|
||||||
4. `.agent/user_stories/document-modal.md` — write the story file mapping the modal behaviour to the E2E scenarios above.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The phase-10 `expect_popup` calls are removed (no new tab); the modal opens in-page.
|
|
||||||
- The standalone page test is kept to guard the no-JS / direct-link fallback.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `tests/e2e/test_document_viewer.py` rewritten — the story gate, green **in isolation** (prereq `podman compose up -d db`).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `test_document_back_navigation.py`, `test_header_consistency.py`, `test_smoke.py` green in isolation (adapted if they asserted a new tab).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# Task 01 — Steering update endpoint
|
|
||||||
|
|
||||||
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
|
|
||||||
**Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add a `PUT /api/steering/{note_id}` endpoint so an existing steering note can be updated in place (the chat-page + header UI currently support create + delete only).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/schemas.py` — add `SteeringNoteUpdate(BaseModel)`:
|
|
||||||
```python
|
|
||||||
class SteeringNoteUpdate(BaseModel):
|
|
||||||
"""``PUT /api/steering/{id}`` body: a new tuning instruction."""
|
|
||||||
note: str = Field(min_length=1, max_length=2000)
|
|
||||||
@field_validator("note", mode="before")
|
|
||||||
@classmethod
|
|
||||||
def _trim_note(cls, v): return v.strip() if isinstance(v, str) else v
|
|
||||||
```
|
|
||||||
Mirror `SteeringNoteIn`'s trim-before-length-constraint behaviour so an empty/whitespace body is a 422.
|
|
||||||
2. `app/api/steering.py` — add the route:
|
|
||||||
```python
|
|
||||||
@router.put("/{note_id}", response_model=SteeringNoteOut)
|
|
||||||
def update_steering_note(note_id, payload: SteeringNoteUpdate, db):
|
|
||||||
row = db.get(SteeringNote, note_id)
|
|
||||||
if row is None:
|
|
||||||
raise HTTPException(status_code=404, detail="steering note not found")
|
|
||||||
row.note = payload.note.strip()
|
|
||||||
db.commit(); db.refresh(row)
|
|
||||||
return SteeringNoteOut(id=row.id, note=row.note, created_at=row.created_at)
|
|
||||||
```
|
|
||||||
The router already carries `dependencies=[Depends(require_admin)]` (phase 16), so anonymous callers get 403 automatically — no extra guard. Keep the existing `list`/`create`/`delete` routes unchanged.
|
|
||||||
3. Update the module docstring to name the new `PUT` route and that it reuses the router-level admin dependency (no new auth surface).
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The update reuses the same 1–2000-char, trimmed contract as create (no separate validation policy).
|
|
||||||
- `created_at` is preserved on update (editing a note doesn't redate it; the list order by newest-first is stable for edits).
|
|
||||||
- The endpoint is `PUT` (idempotent, full replacement of `note`), matching the "edit" semantics.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit (`tests/unit/test_steering.py`): `update_steering_note` via the router — 200 with updated `note`; 404 unknown id; 422 empty/whitespace/over-2000; and the router-level 403 for anonymous (via `require_admin`).
|
|
||||||
- Integration: `PUT /api/steering/{id}` → 200 returns the new note; the updated note is then read back by `load_steering_notes` (oldest-first order preserved) and appears in `build_steering_section`; anonymous → 403.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `PUT /api/steering/{id} {note}` → 200 with the updated note; `GET /api/steering` reflects the new text and order.
|
|
||||||
- [ ] Unknown id → 404; invalid body → 422; anonymous → 403.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
# Task 02 — Tuning page HTML + CSS
|
|
||||||
|
|
||||||
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
|
|
||||||
**Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create the `/tuning.html` page: a title, a "Add a global tuning note" form, and a list of existing notes each with an inline edit control and a delete control. Follow the Phase-08 tokens, WCAG 2.1 AA, and the standard app frame (landmarks, skip-link, sticky header reuse via `header.js`).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/tuning.html` (new) — standard app frame:
|
|
||||||
- `<header class="app-header">` reusing the same markup as `index.html`'s header (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The page gains an extra admin-only link:
|
|
||||||
```html
|
|
||||||
<a href="/tuning.html" class="nav-link is-active" id="nav-tuning" aria-current="page">Tuning</a>
|
|
||||||
```
|
|
||||||
(placed after the Sources link; it is always present but only visible to admins because the whole nav is admin-scoped on non-chat pages — the Sources link is already admin-only, so this is consistent).
|
|
||||||
- `<main id="main" class="app-main" tabindex="-1">` with a centered `container` column:
|
|
||||||
- `<h1>Global Tuning</h1>` + a sub-heading explaining every note steers all future answers.
|
|
||||||
- A create form `#tune-form`: a visually-hidden `<label for="tune-note">` + `<textarea id="tune-note" maxlength="2000" rows="3" placeholder="e.g. be more concise — or: assume I'm on NixOS">` + a submit button `#tune-save` ("Add note").
|
|
||||||
- A status/announcer `<p class="visually-hidden" id="tune-announcer" role="status" aria-live="polite"></p>`.
|
|
||||||
- A list `<ul id="tune-list">` (role=list) for existing notes; each `<li class="tuning-note">` holds:
|
|
||||||
- a `.tuning-note-text` span (textContent, XSS-safe),
|
|
||||||
- an `#edit` button (`.tuning-edit`, icon + "Edit"),
|
|
||||||
- a `.tuning-delete` button (icon + "Delete").
|
|
||||||
- An empty-state `<p id="tune-empty">No tuning notes yet — add one above.</p>` toggled by `tuning.js`.
|
|
||||||
- `<footer class="app-footer">` as on other pages.
|
|
||||||
- Load `assets/markdown.js` (classic) and `assets/tuning.js` (module) + the shared header (module).
|
|
||||||
2. `frontend/assets/styles.css` — add a `--tuning-*` block mirroring the `.steering-*` / `.tune-*` tokens:
|
|
||||||
- `.tuning-note` — flex row, align-items center, gap; text flexes, actions shrink-0; border-bottom divider.
|
|
||||||
- `.tuning-note-text` — `--ink`; ellipsis overflow if very long.
|
|
||||||
- `.tuning-edit`, `.tuning-delete` — icon + label buttons, ≥44px targets, `:focus-visible` ring; edit uses brand colour, delete uses the error colour (`--err-ink`/`--err-line`) consistent with `.steering-delete`.
|
|
||||||
- `.tuning-note.is-editing .tuning-note-text` — hidden while editing (replaced by the inline form).
|
|
||||||
- Inline edit form (`.tuning-edit-form`) — a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled + Save/Cancel, styled like the existing `.tune-form` / `.tune-save` / `.tune-cancel`.
|
|
||||||
- `#tune-form` + `#tune-note` — styled like the existing composer/`#message-input`; `#tune-save` styled like `.tune-save`.
|
|
||||||
- `#tune-empty` — `--ink-soft`, centered, italic.
|
|
||||||
- Ensure the centered column matches the chat/sources width discipline (≥80–90% of container; not a skinny list).
|
|
||||||
- No `filter: blur`, no CDN, system font stack.
|
|
||||||
3. Verify no selector collision (grep `.tuning-note`, `#tune-form`, `.tuning-edit-*`).
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The page title is "Global Tuning"; the nav link label is "Tuning" (consistent with the chat header's "Tuning" panel).
|
|
||||||
- Editing is inline (swap the text for a textarea + Save/Cancel in the same list row) — the owner's "previous tunes should be editable" is satisfied without a separate editor page.
|
|
||||||
- The list is newest-first (same as the header panel) for consistency.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No unit/integration test for static CSS/HTML.
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `/tuning.html` renders the title, the create form, the note list, and the empty state.
|
|
||||||
- [ ] All controls are labeled, ≥44px, `:focus-visible`, contrast ≥4.5:1; landmarks + skip-link present.
|
|
||||||
- [ ] No selector collision; no CDN tags; the page loads `tuning.js` + `markdown.js`.
|
|
||||||
- [ ] The admin-only "Tuning" nav link is present in the header.
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# Task 03 — Tuning page CRUD logic
|
|
||||||
|
|
||||||
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
|
|
||||||
**Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create `frontend/assets/tuning.js` — the single owner of the tuning page's behaviour: load notes, create, edit (inline), cancel, and delete, all announced through a polite live region.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/tuning.js` (new, module) — mirrors the structure of `app.js`'s steering section but for the standalone page:
|
|
||||||
- Cache the elements: `#tune-form`, `#tune-note`, `#tune-save`, `#tune-list`, `#tune-empty`, `#tune-announcer`.
|
|
||||||
- `announce(msg)` — sets `#tune-announcer.textContent` (role=status, aria-live=polite).
|
|
||||||
- `loadNotes()` — `GET /api/steering`; on `r.ok` render the list, else keep the last rendered list (progressive enhancement). Populates `#tune-list` (role=list) with `<li>` rows; toggles `#tune-empty` (`hidden = notes.length > 0`); each row:
|
|
||||||
- `.tuning-note-text` span (`textContent`, XSS-safe),
|
|
||||||
- `.tuning-edit` button (icon + "Edit"),
|
|
||||||
- `.tuning-delete` button (icon + "Delete", aria-label "Delete tuning note: <note>").
|
|
||||||
- Create handler (`#tune-form` submit): `POST /api/steering {note}`; 201 → clear the textarea, announce "Tuning note added. Future answers will follow it.", reload; non-2xx → inline error under the form (kept, form not cleared) with the API detail; network error → friendly message. Disable `#tune-save` during the request.
|
|
||||||
- Edit flow (`tuning-edit` click): swap the row's `.tuning-note-text` span for an inline `.tuning-edit-form` containing a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled with the note + Save/Cancel; focus the textarea. Track the note id on the form (data attribute).
|
|
||||||
- Save-edit handler: `PUT /api/steering/{id} {note}`; 200 → replace the form with a `.tuning-saved` status (role=status) + announce "Tuning note updated."; non-2xx → keep the form, show inline error; cancel → revert to the text span.
|
|
||||||
- Delete handler (`tuning-delete` click): disable the button; `DELETE /api/steering/{id}`; 204/404 → remove the `<li>` from the DOM immediately (optimistic), announce, and if 404 reload; non-2xx → re-enable the button + announce retry.
|
|
||||||
- Keep the 1–2000-char contract on the client (maxlength on the textareas); the server re-validates.
|
|
||||||
2. Wire the shared header + whoami gate: `tuning.js` imports `{ initSharedHeader, fetchIsAdmin }` from `./header.js` and at boot awaits `initSharedHeader()` then `loadNotes()` only if `fetchIsAdmin()` is true (anonymous users see the page frame but the list stays empty / the create form 403s gracefully — consistent with the Sources page gate pattern). Actually simpler: the header already hides the "Tuning" nav link for anonymous (it's a nav link like Sources); but a direct anonymous URL should still be safe — `loadNotes()` swallows non-2xx, and the create form 403s. So the page is anonymous-safe without a hard gate.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The edit is inline in the same list row (no separate editor page) — the owner's "editable" is satisfied.
|
|
||||||
- Optimistic delete (remove the `<li>` before the server confirms) matches the chat page's `deleteSteeringNote` UX.
|
|
||||||
- The page is anonymous-safe: the list won't render for anonymous (the create/delete calls 403), and the nav link is hidden for anonymous.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No unit/integration test (frontend-only).
|
|
||||||
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Create a note on `/tuning.html` → it appears in the list; the announcer announces the change.
|
|
||||||
- [ ] Edit a note inline → the saved text is updated in the list; the `<tuning>` prompt reflects it (verified via the integration test on the endpoint).
|
|
||||||
- [ ] Delete a note → the row is removed and the list updates.
|
|
||||||
- [ ] XSS-safe: note text rendered via `textContent`, never `innerHTML`.
|
|
||||||
- [ ] Empty state shows when there are no notes.
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# Task 04 — Header "Tuning" button + E2E suite
|
|
||||||
|
|
||||||
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
|
|
||||||
**Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Expose the tuning page via an admin-only header link and add the E2E story suite.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/tuning.html` header — add the admin-only "Tuning" nav link (is-active) in the `<nav class="app-nav">`, alongside the existing Sources link (also admin-only). The header markup mirrors `index.html`/`sources.html` (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The "Tuning" link ships visible on the tuning page (it's the current page); on other pages the admin-only nav links are revealed by `header.js` whoami, consistent with Sources.
|
|
||||||
2. No change to `app.js`/`sources.html` nav needed — the tuning page is reached from its own header link. (The chat-page "Tune" button and header panel from phase 15 remain unchanged.)
|
|
||||||
3. `tests/e2e/test_global_tuning.py` (new — the story gate). Reuse the seeding harness pattern from `test_steering.py` / `test_document_viewer.py` (`_import_fixtures` / `_reset_db` / `_run_in_thread`) so the endpoint-under-test is exercised against a seeded KB. Test → mapping (Playwright Mapping Rule):
|
|
||||||
1. `test_create_note_without_chat` — log in, go to `/tuning.html`, type a note, submit; assert it appears in `#tune-list` and the announcer announced it. No chat turn was made.
|
|
||||||
2. `test_edit_note_inline` — create a note, click "Edit", change the text, Save; assert the list shows the new text and the edit form is gone.
|
|
||||||
3. `test_delete_note` — create a note, delete it; the row is removed and `#tune-empty` shows again.
|
|
||||||
4. `test_edit_note_steers_answer` — create a note via `/tuning.html`, then go to the chat, ask the QUESTION, and assert the note's marker leaks into the answer (same echo trick `test_steering.py` uses: `(tuning: <first note line>)`), proving the edited note is read into the system prompt.
|
|
||||||
5. `test_tuning_page_a11y_and_no_cdn` — landmarks, skip-link, labeled controls, ≥44px targets, `:focus-visible`; every `script[src]`/`link[href]` is same-origin or `data:`; dark theme.
|
|
||||||
6. `test_anonymous_cannot_manage` — anonymous `/tuning.html`: the list is empty, and a scripted `PUT /api/steering/{id}` returns 403 (or the create form 403s).
|
|
||||||
4. Regressions to run green in isolation: `test_steering.py` (the chat-page Tune button + panel still work — create + delete), `test_header_consistency.py` (the tuning page's header is consistent; the new nav link doesn't break the height/consistency assertions on the other pages), `test_document_viewer.py` (unrelated — modal still works), `test_smoke.py`.
|
|
||||||
5. `.agent/user_stories/global-tuning.md` — write the story file.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The tuning page's "Tuning" nav link is admin-only (consistent with the Sources link — the catalog and tuning are admin-only).
|
|
||||||
- The chat-page "Tune" button + header panel are **not** removed (nothing regresses); the tuning page is the primary global manager, the button is a quick-add affordance.
|
|
||||||
- The edit-steers-answer test reuses the mock-LLM echo behaviour already established in `test_steering.py`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `tests/e2e/test_global_tuning.py` — the story gate, green **in isolation** (prereq `podman compose up -d db`).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_global_tuning.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `test_steering.py`, `test_header_consistency.py`, `test_document_viewer.py`, `test_smoke.py` green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Task 01 — Settings: `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`
|
|
||||||
|
|
||||||
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified."`
|
|
||||||
**Story:** `.agent/user_stories/git-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the two new settings that drive the git-based source flow: the comma-separated list of git URLs and the dedicated local clone location.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — in the "Import scope" section (near `import_extensions`) add:
|
|
||||||
- `git_sources: str = ""` — comma-separated git repository URLs (no default; when empty, `import_docs` falls back to `--source` / the old `DEFAULT_SOURCES`). Document: "list of git repo URLs to clone/pull into `sources_dir` before indexing (phase 28); empty means no git sources".
|
|
||||||
- `sources_dir: str = "~/bor-sources"` — the dedicated local directory the repos are cloned/pulled into (phase 28). Document: "where `import_docs` clones/pulls `git_sources` repos (expanded ~ via `Path.expanduser`)".
|
|
||||||
- Add a property `git_source_list` returning the non-empty, stripped URLs (list[str]) — used by the import script.
|
|
||||||
2. `tests/unit/test_config.py` — add tests:
|
|
||||||
- `git_sources` default is `""`; `git_source_list` returns `[]` when empty.
|
|
||||||
- `git_sources` env override parses a comma-separated list (whitespace trimmed, empties dropped).
|
|
||||||
- `sources_dir` default is `~/bor-sources` (raw string, not expanded in the setting — expansion happens in the script).
|
|
||||||
3. `.env.example` — document both under an "Import scope / git sources" comment block:
|
|
||||||
```
|
|
||||||
# --- Import sources (git; phase 28) ---
|
|
||||||
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
|
|
||||||
# BOR_SOURCES_DIR=~/bor-sources
|
|
||||||
```
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- `git_sources` is empty by default (backwards-compatible: the `import_docs` script keeps its `--source` default until the owner sets `BOR_GIT_SOURCES`).
|
|
||||||
- URLs are stored raw (no parsing of scheme/auth in the setting) — parsing happens in `git_sync.py`.
|
|
||||||
- `sources_dir` is stored as a raw string; `Path.expanduser()` is applied in the script (so the setting stays env-agnostic and testable).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `tests/unit/test_config.py` additions.
|
|
||||||
- Coverage: the new setting is exercised by the unit test; the >90% `app/` gate is maintained.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `Settings().git_source_list` returns `[]` by default and the parsed list when `BOR_GIT_SOURCES` is set.
|
|
||||||
- [ ] `sources_dir` defaults to `~/bor-sources`.
|
|
||||||
- [ ] `.env.example` documents both vars.
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
# Task 02 — Git clone/pull utility
|
|
||||||
|
|
||||||
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
|
|
||||||
**Story:** `.agent/user_stories/git-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create `scripts/git_sync.py` with a single `clone_or_pull(url, dest)` function that clones a repo (shallow, first run) or fast-forwards it (subsequent runs), returning the destination path. This is the only place `git` is invoked.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `scripts/git_sync.py` (new) — stdlib `subprocess` only (A11: no new packages):
|
|
||||||
```python
|
|
||||||
"""Git source sync for import_docs (phase 28).
|
|
||||||
|
|
||||||
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
|
|
||||||
the first time, or fast-forwards an existing checkout with ``git pull
|
|
||||||
--ff-only`` on subsequent runs. Auth is whatever the URL/SSH config
|
|
||||||
supplies — no credentials are stored here.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
class GitSyncError(RuntimeError): ...
|
|
||||||
|
|
||||||
def clone_or_pull(url: str, dest: Path | str) -> Path:
|
|
||||||
dest = Path(dest)
|
|
||||||
if not dest.exists() or not (dest / ".git").exists():
|
|
||||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
|
|
||||||
else:
|
|
||||||
_run(["git", "pull", "--ff-only"], cwd=dest)
|
|
||||||
return dest
|
|
||||||
|
|
||||||
def _run(argv, cwd):
|
|
||||||
try:
|
|
||||||
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
|
|
||||||
except FileNotFoundError:
|
|
||||||
raise GitSyncError("git was not found on PATH — install git and retry")
|
|
||||||
if proc.returncode != 0:
|
|
||||||
raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): {proc.stderr.strip()}")
|
|
||||||
return proc.stdout
|
|
||||||
```
|
|
||||||
- `--depth 1` clone (fast, and the KB is re-imported incrementally anyway).
|
|
||||||
- `--ff-only` pull (refuses to merge unrelated histories — a broken checkout fails loudly rather than producing a dirty index).
|
|
||||||
- `GitSyncError` carries the `git` stderr so the caller can name the failing repo + reason.
|
|
||||||
- Auth: nothing special — a `https://…` URL uses the OS credential helper / prompts; an `git@host:repo.git` URL uses the machine's SSH key. Document this in the module docstring.
|
|
||||||
2. `tests/unit/test_git_sync.py` (new):
|
|
||||||
- `clone_or_pull` **clones** when the dest has no `.git` — verify by monkeypatching `subprocess.run` to a fake that records the argv and returns `returncode=0`; assert `["git","clone","--depth","1",url,str(dest)]` was called and `dest` returned.
|
|
||||||
- `clone_or_pull` **pulls** when `.git` exists — monkeypatch; assert `["git","pull","--ff-only"]` called.
|
|
||||||
- `clone_or_pull` raises `GitSyncError` on non-zero exit, carrying the stderr text.
|
|
||||||
- `clone_or_pull` raises `GitSyncError("git was not found…")` when `subprocess.run` raises `FileNotFoundError`.
|
|
||||||
- `dest.parent` is created before clone (assert the fake saw a parent that `mkdir` would create — or test the `mkdir` call path directly).
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- `--depth 1` shallow clone is sufficient (the KB is re-imported incrementally; no need for full history).
|
|
||||||
- `--ff-only` pull is the right policy (refuse merges — a dirty/broken checkout fails loudly).
|
|
||||||
- Auth is delegated to the machine (SSH key / credential helper); no secrets are stored in code or `.env`.
|
|
||||||
- `git` CLI is assumed present (standard on homelab machines; a clear error is raised otherwise).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `tests/unit/test_git_sync.py` (clone vs pull dispatch, error propagation, missing-git error).
|
|
||||||
- Coverage: the new `scripts/git_sync.py` is fully covered.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `clone_or_pull` clones a fresh repo and pulls an existing one (verified via monkeypatched `subprocess`).
|
|
||||||
- [ ] A failing `git` call raises `GitSyncError` with the stderr; a missing `git` raises `GitSyncError` naming git.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# Task 03 — `import_docs` resolves git sources → local dirs
|
|
||||||
|
|
||||||
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
|
|
||||||
**Story:** `.agent/user_stories/git-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Rewire `scripts/import_docs.py` so that, when `BOR_GIT_SOURCES` is set, it clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and indexes the resulting directories — while keeping `--source <path>` overriding for manual local directories.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `scripts/import_docs.py`:
|
|
||||||
- Import `clone_or_pull` from the sibling module: `from scripts.git_sync import clone_or_pull` (or, since `main()` runs as `python -m scripts.import_docs`, a top-level `from git_sync import clone_or_pull` works — `scripts/` is on `sys.path`).
|
|
||||||
- Keep `DEFAULT_SOURCES` as the fallback for the no-`--source` + no-`BOR_GIT_SOURCES` case (backwards-compatible).
|
|
||||||
- New resolution logic in `main()`:
|
|
||||||
```python
|
|
||||||
settings = get_settings()
|
|
||||||
sources = _resolve_sources(args.source, settings)
|
|
||||||
```
|
|
||||||
where `_resolve_sources`:
|
|
||||||
- If `args.source` is given → expanduser each and return (unchanged manual behaviour; `--source` wins).
|
|
||||||
- Else if `settings.git_source_list` is non-empty → for each URL, `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))`; collect the dest dirs; return them. Any `GitSyncError` propagates (the script exits non-zero naming the failing repo — see below).
|
|
||||||
- Else → return the old `DEFAULT_SOURCES`.
|
|
||||||
- `repo_name(url)` — derive a directory name from the URL: strip a trailing `.git`, take the basename after the last `/` (or `:` for scp-style `git@host:repo.git`). Fall back to a slug of the URL if no basename.
|
|
||||||
- Before importing, log which dirs are being imported (so the operator sees the cloned paths). The existing "source dir not found" warning still applies if a clone left an empty dir.
|
|
||||||
2. Keep the existing `--prune` / `--limit` flags and the summary print unchanged.
|
|
||||||
3. Exit code: if any `GitSyncError` is raised, let it propagate to a top-level `except` that prints `import_docs: git sync failed: <reason>` to stderr and returns 1 **before** importing anything (so a bad repo doesn't silently import partial junk). Structure:
|
|
||||||
```python
|
|
||||||
try:
|
|
||||||
sources = _resolve_sources(...)
|
|
||||||
except GitSyncError as e:
|
|
||||||
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
```
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- `--source` always wins over `BOR_GIT_SOURCES` (explicit CLI flag beats env).
|
|
||||||
- When `BOR_GIT_SOURCES` is set, `--source` is ignored (only one source mode at a time) — document this.
|
|
||||||
- `BOR_SOURCES_DIR` defaults to `~/bor-sources`; each repo is a subdirectory named after the repo.
|
|
||||||
- A repo that fails to clone/pull aborts the whole run (no partial import) — the operator fixes the URL and re-runs; the already-cloned repos are left on disk and will be pulled on the next run.
|
|
||||||
- The `documents.source` column will be the repo directory name (e.g. `homelab`), matching the current `source=root.name` behaviour in the importer.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: `tests/integration/test_import_docs_git.py` (new) — monkeypatch `clone_or_pull` to a fake that creates a temp dir with a fixture `.md` file and returns it; assert `import_docs.main(["--source"])`-equivalent resolution picks the cloned dir and that `import_sources` is called with it. Also assert a `GitSyncError` from the fake → exit code 1 and no import attempt.
|
|
||||||
- Coverage: the new `main()` resolution branch is covered.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `BOR_GIT_SOURCES` set → `import_docs` clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and imports them.
|
|
||||||
- [ ] `--source <path>` still imports that manual directory (unchanged).
|
|
||||||
- [ ] A failing git sync → non-zero exit, message naming the repo, no partial import.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Task 04 — Integration test + docs
|
|
||||||
|
|
||||||
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
|
|
||||||
**Story:** `.agent/user_stories/git-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the integration test that exercises the full `import_docs` git-resolution path with a mocked `clone_or_pull`, and finalise the docs.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/integration/test_import_docs_git.py` (new) — drive `scripts.import_docs.main(argv)` end to end with the git flow mocked:
|
|
||||||
- Monkeypatch `scripts.import_docs.clone_or_pull` (import the name into the module's namespace after importing it) so it **creates** a temp directory containing a fixture `.md` file and returns that path (simulating a clone/pull landing real content).
|
|
||||||
- Set `settings.sources_dir` to a `tmp_path`-based dir (via `monkeypatch.setattr` on the settings or by patching `get_settings`).
|
|
||||||
- Call `main([])` with `BOR_GIT_SOURCES` patched to a single URL.
|
|
||||||
- Assert: `clone_or_pull` was called once with the URL; the returned dir was passed to `import_sources`; the summary printed includes the fixture file (added > 0).
|
|
||||||
- Negative case: monkeypatch `clone_or_pull` to raise `GitSyncError`; assert `main([])` returns `1` and prints a message naming the repo, and that `import_sources` was **not** called.
|
|
||||||
- Manual override case: `main(["--source", str(tmp_path)])` imports the manual dir and does **not** call `clone_or_pull`.
|
|
||||||
- Use the same `FakeEmbedder`-style approach the importer tests use so no real LLM is needed (the importer's two-phase embed is duck-typed; pass a fake `llm` if `main` allows injection, or let `import_docs` build the real `LLMClient` but patch `import_sources` to capture its `sources` arg and short-circuit). Simpler: monkeypatch `scripts.import_docs.import_sources` to a fake that records the `sources` list and returns a trivial `ImportSummary` — this isolates the git-resolution logic from the whole embed pipeline.
|
|
||||||
2. `README.md`:
|
|
||||||
- Section 5 (Import your knowledge base): document the two source modes — (a) git sources via `BOR_GIT_SOURCES` (clone/pull into `BOR_SOURCES_DIR`), (b) `--source <path>` for manual directories. Keep the `~/Homelab + ~/Deployments` note as the *previous* default, now replaced by `BOR_GIT_SOURCES`.
|
|
||||||
- Add a short "Git-based sources" subsection: set `BOR_GIT_SOURCES` (comma-separated URLs) + `BOR_SOURCES_DIR`; `import_docs` clones (first run) or pulls (subsequent runs) each repo and indexes them; `--source` overrides; a failed sync aborts the run.
|
|
||||||
- The "Clicking a chip opens that document in a new tab" bullet (line ~69) is now stale — update to "opens in an almost-fullscreen modal" (phase 26).
|
|
||||||
3. `.env.example` — already updated in task 01; double-check the comment block is present and correct.
|
|
||||||
4. `.agent/user_stories/git-sources.md` — write the story file.
|
|
||||||
|
|
||||||
## ASSUMPTIONS
|
|
||||||
- The integration test mocks `clone_or_pull` + `import_sources` so it runs without network, without `git`, and without a real embed endpoint — it isolates the resolution logic.
|
|
||||||
- The README keeps a migration note for owners currently relying on the hardcoded `~/Homelab`/`~/Deployments` default (set `BOR_GIT_SOURCES` to the same two repos).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: `tests/integration/test_import_docs_git.py`.
|
|
||||||
- Coverage: the new `main()` branch covered.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/integration/test_import_docs_git.py -v --no-cov` green.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] README documents both source modes; the "opens in a new tab" bullet updated to "modal".
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Phase 29 — Global Tuning nav link
|
|
||||||
|
|
||||||
**Source:** `.agent/phases/complete/27_global_tuning` — the `/tuning.html` page and its create/edit/delete machinery shipped, but the "Tuning" header link that was supposed to *expose* the page (`27` task 04: *"Expose the tuning page via an admin-only header link"`) only exists on `tuning.html` itself. It is not present on the Chat or Sources pages, so there is no way to reach `/tuning.html` from anywhere except typing the URL.
|
|
||||||
**Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
**Context:** The shared header (phase 19) renders `[Chat, Sources — admin only]` in `<nav class="app-nav">` on the Chat and Sources pages. `frontend/assets/header.js` already contains logic to reveal an admin-only `#nav-tuning` link for admins (lines 69–70) — but that element is never in `index.html`/`sources.html` markup, so the reveal is a no-op. This phase adds the missing markup so the link the JS already handles actually exists.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the Global Tuning page reachable by adding the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages — hidden by default, revealed for admins by the existing `header.js` whoami gate (the exact same pattern as the Sources link). No backend, schema, or API change.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `19_shared_header` (complete) — the `<nav class="app-nav">` contract, the shared header markup/ids, and the 64px header-height rule on the Chat and Sources pages.
|
|
||||||
- `16_admin_auth` (complete) — the `/api/whoami` gate + `header.js` `fetchIsAdmin()` that reveals admin-only nav links.
|
|
||||||
- `27_global_tuning` (complete) — the `/tuning.html` page and `PUT/POST/DELETE /api/steering` endpoints that this link points to.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_tuning_links_markup.md` — add the admin-only `#nav-tuning` link to the Chat (`index.html`) and Sources (`sources.html`) headers, hidden by default.
|
|
||||||
2. `02_tuning_nav_e2e_and_regression.md` — add the story E2E suite (`test_tuning_nav_link.py`) + verify no regressions in the header/shared-header/no-CDN suites and commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: no new backend logic (frontend-only markup). The existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) must still pass — the new link is same-origin `<a>`, no new tags.
|
|
||||||
- Coverage: frontend-only; the `app/` >90% gate is unaffected (unchanged).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` — the story gate, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] On the **Chat** page (`/`), a signed-in admin sees a **"Tuning"** link (next to Chat/Sources) that navigates to `/tuning.html`; an anonymous visitor does **not** see it (ships `hidden`, `header.js` reveals only for admin).
|
|
||||||
- [ ] Same on the **Sources** page (`/sources.html`).
|
|
||||||
- [ ] The link is absent from the anonymous DOM-reveal (verified via `test_anonymous` behavior); header markup stays valid and semantic.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (no backend change).
|
|
||||||
- [ ] Existing header suites stay green in isolation: `test_header_consistency.py` (64px height + nav structure unchanged), `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py`.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean (no Python change, but run the gate).
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): the link is a labeled, focus-visible `<a>`; header landmarks/contrast unchanged; no CDN.
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/29_tuning_nav_link/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; the link is a same-origin `<a href="/tuning.html">`.
|
|
||||||
- **A10 untouched** — no new endpoint or auth surface; the link reuses the existing `#nav-sources` reveal path (`header.js` already gates `#nav-tuning` on `fetchIsAdmin()`).
|
|
||||||
- **No schema / migration** — purely a markup addition.
|
|
||||||
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
- **Scope boundary** — only the Chat and Sources pages (the two that share the standard `app-nav` with the Sources link). The viewer page (`document.html`) uses a different `.doc-header-actions` header variant and the login page is the auth gate; both stay out of scope for this fix.
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
# Task 01 — Tuning nav link on the Chat and Sources pages
|
|
||||||
|
|
||||||
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages, so `/tuning.html` is reachable from both.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
Add one link to each page, inside the existing `<nav class="app-nav" aria-label="Primary">`, **immediately after** the Sources link, mirroring the Sources pattern exactly.
|
|
||||||
|
|
||||||
1. `frontend/index.html` (Chat page) — after the `#nav-sources` link, add:
|
|
||||||
```html
|
|
||||||
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
|
|
||||||
2026-08-25) — hidden by default, header.js reveals it once
|
|
||||||
whoami says admin, exactly like the Sources link above. Points at
|
|
||||||
the standalone /tuning.html manager (phase 27). -->
|
|
||||||
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
|
|
||||||
```
|
|
||||||
2. `frontend/sources.html` (Sources page) — same link, placed after its `#nav-sources` link:
|
|
||||||
```html
|
|
||||||
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
|
|
||||||
2026-08-25) — hidden by default, header.js reveals it once
|
|
||||||
whoami says admin, exactly like the Sources link above. -->
|
|
||||||
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
|
|
||||||
```
|
|
||||||
|
|
||||||
Rules for both edits:
|
|
||||||
- Keep `hidden` by default (anonymous-safe ship-hidden, identical to `#nav-sources`).
|
|
||||||
- Do **not** add `is-active` / `aria-current="page"` — those are "current page" visual states; this link is always hidden on the Chat/Sources pages, and `tuning.html` already sets `is-active` on its own link.
|
|
||||||
- Preserve surrounding markup and indentation so the header layout and the pinned 64px height (phases 12/19) are unchanged. The link is inline in the nav, so height is unaffected.
|
|
||||||
- **Do not modify `frontend/assets/header.js`.** It already handles this element (lines 69–70: `const navTuning = document.querySelector("#nav-tuning"); if (navTuning) navTuning.hidden = !admin;`). The link now exists for that existing reveal code to act on. If, against expectation, that reveal code were absent, add it next to the `navSources` reveal — but it is present, so leave `header.js` untouched.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No new unit/integration logic (frontend markup only).
|
|
||||||
- Coverage: frontend-only; `app/` coverage unaffected.
|
|
||||||
- This is a same-origin `<a>` — the existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `index.html` and `sources.html` each contain an admin-only `#nav-tuning` `<a href="/tuning.html">` placed after `#nav-sources`, `hidden` by default, without `is-active`.
|
|
||||||
- [ ] Header structure/height unchanged (confirmed in task 02 via `test_header_consistency.py` / `test_shared_header.py`).
|
|
||||||
- [ ] Full test suite green; no behavior change in completed phases.
|
|
||||||
@@ -1,53 +0,0 @@
|
|||||||
# Task 02 — Tuning nav-link E2E suite, regression checks, and commit
|
|
||||||
|
|
||||||
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the new admin-only "Tuning" link works (and is hidden for anonymous users) on the Chat and Sources pages, confirm no header/no-CDN regressions, and commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **New story E2E suite** — create `tests/e2e/test_tuning_nav_link.py`, following the harness already used in `tests/e2e/test_global_tuning.py`:
|
|
||||||
- Reuse `from e2e.auth_helpers import login` (real form login on `/login.html`; `login(page, app_url, next=...)` redirects to `next`) and the `db_ready` fixture. No KB seeding is needed for the link-visibility tests (the link is pure header markup gated by `/api/whoami`), so a light module fixture that just brings up the app + mock LLM is enough — mirror `test_global_tuning.py`'s `mock_llm`/`app_url`/`db_ready` params and its `_reset_db` if you want a clean DB.
|
|
||||||
- The visible `#nav-tuning` on an admin page is the sync point (exactly the Sources-link contract): `header.js` reveals it only after whoami says admin.
|
|
||||||
- Write these tests (Playwright `expect`, sync API):
|
|
||||||
1. `test_admin_sees_tuning_link_on_chat` — `login(page, app_url, next="/")`; on the Chat page assert `#nav-tuning` is visible and its `href` is `/tuning.html`; click it and assert the URL navigates to `/tuning.html` (and `#nav-tuning` becomes `is-active`, matching the page's own link).
|
|
||||||
2. `test_admin_sees_tuning_link_on_sources` — `login(page, app_url, next="/sources.html")`; assert `#nav-tuning` is visible on the Sources page and clicking it navigates to `/tuning.html`.
|
|
||||||
3. `test_anonymous_hides_tuning_link_on_chat_and_sources` — visit `/` and `/sources.html` **without** logging in; assert `#nav-tuning` is `hidden` on both pages, `#sign-in-link` visible, `#nav-sources` also hidden (consistent with the existing Sources gate).
|
|
||||||
4. `test_tuning_link_a11y_and_no_cdn` — on the admin Chat page, assert the link is a labeled, focus-visible `<a>` (tab through the header, focus lands on it), and assert every `script[src]`/`link[href]` on the page is same-origin or `data:` (no CDN) — reuse the same `evaluate` scan `test_global_tuning.py::test_tuning_page_a11y_and_no_cdn` uses.
|
|
||||||
2. **Regression verification** — after the new suite is green, run the header- and page-sensitive suites **in isolation** to prove no regression from the added markup/height:
|
|
||||||
```bash
|
|
||||||
uv run pytest tests/e2e/test_header_consistency.py -v --no-cov
|
|
||||||
uv run pytest tests/e2e/test_shared_header.py -v --no-cov
|
|
||||||
uv run pytest tests/e2e/test_smoke.py -v --no-cov
|
|
||||||
uv run pytest tests/e2e/test_global_tuning.py -v --no-cov
|
|
||||||
uv run pytest tests/integration/test_api.py::test_index_html_served_locally -v --no-cov
|
|
||||||
```
|
|
||||||
Fix anything the new link breaks (it should not — the link is inline in the existing nav and does not change header height).
|
|
||||||
3. **Full gate** — run the whole suite with coverage and confirm `app/` coverage TOTAL is **unchanged** (this phase adds no backend logic):
|
|
||||||
```bash
|
|
||||||
uv run pytest
|
|
||||||
uv run pytest --cov=app --cov-report=term-missing # TOTAL must not drop
|
|
||||||
```
|
|
||||||
4. **Lint / types** — `uv run ruff check . && uv run pyright` (no Python change expected, but run the gate).
|
|
||||||
5. **Commit** — one atomic `--no-gpg-sign` commit staging only this phase's files (the new E2E file + any committed edits), then move the phase directory to `.agent/phases/complete/`:
|
|
||||||
```bash
|
|
||||||
git add -f .agent/phases/todo/29_tuning_nav_link tests/e2e/test_tuning_nav_link.py
|
|
||||||
git commit --no-gpg-sign -m "feat(ui): expose Global Tuning from Chat + Sources headers (admin-only Tuning nav link)"
|
|
||||||
git mv .agent/phases/todo/29_tuning_nav_link .agent/phases/complete/29_tuning_nav_link
|
|
||||||
```
|
|
||||||
(`.agent/` is gitignored — use `git add -f` / `git mv -f` as needed.)
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` green **in isolation**:
|
|
||||||
```bash
|
|
||||||
uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov
|
|
||||||
```
|
|
||||||
- No new unit/integration logic (frontend markup only); `app/` coverage unchanged.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Admin sees a visible **"Tuning"** link on the Chat and Sources pages → clicking it opens `/tuning.html`.
|
|
||||||
- [ ] Anonymous visitors see **no** Tuning link on either page (ships `hidden`, revealed only for admin).
|
|
||||||
- [ ] Link is labeled, focus-visible; page stays CDN-free.
|
|
||||||
- [ ] `test_tuning_nav_link.py` green in isolation; `test_header_consistency.py`, `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py` green in isolation; `test_index_html_served_locally` green.
|
|
||||||
- [ ] `uv run pytest` green; `app/` coverage TOTAL unchanged; `ruff` + `pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# Phase 30 — Document Summaries (lite-model summaries for non-markdown documents)
|
|
||||||
|
|
||||||
**Source:** `TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'."`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
**Context:** The importer (`app/rag/importer.py::_index_file` — chunk → embed → upsert per file, A9 scope), hybrid retrieval (`app/rag/retriever.py` — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (`app/rag/prompts.py`), the aipi client (`app/rag/llm.py` — A5: `turbo` chat streaming + `embed` embeddings), and the E2E mock LLM (`tests/e2e/mock_llm.py` — deterministic, keys on system-prompt markers like `DEFLECT_MODE` / `<tuning>`).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Give every **non-markdown** A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi **`lite`** model. The summary is stored on the document (`documents.summary`) **and indexed as one extra embedded chunk** (`chunks.is_summary`), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the **full source document** to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `29_tuning_nav_link` (complete) — the latest finished phase (sequencing only).
|
|
||||||
- Substantively builds on: `02_story_import_documents` / `24_whole_document_context` (import pipeline + full-document context contract), `09_story_retrieval_quality` (A7 hybrid retrieval the summary chunk flows through unchanged), `01_infrastructure` (models/alembic, LLM client, E2E mock).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_lite_model_client.md` — `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming `LLMClient.chat()` for the lite model.
|
|
||||||
2. `02_migration_summary_columns.md` — Alembic 0004: `documents.summary TEXT NULL` + `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
|
|
||||||
3. `03_summarizer_module.md` — `app/rag/summarizer.py`: `SUMMARY_MODE` prompt (capped input), lite call, output validation + deterministic `Source: <source>/<path>` pointer line.
|
|
||||||
4. `04_importer_summary_integration.md` — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters.
|
|
||||||
5. `05_pipeline_summary_resolution.md` — `is_summary` through the retriever, `summary_hits` in `TurnPlan` + the per-turn log line; full source document on summary hit (existing mapping, asserted).
|
|
||||||
6. `06_mock_and_e2e.md` — deterministic `lite` in `mock_llm.py`, sentinel fixture, `tests/e2e/test_document_summaries.py`, story file, README, commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (`is_summary` through both candidate lists + `fuse`), chat gate (`TurnPlan.summary_hits`), LLM client (`chat()`).
|
|
||||||
- Integration: migration 0004 up/down.
|
|
||||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`, TOTAL ≥ pre-change number).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_document_summaries.py` — one story, run **in isolation** (`uv run pytest tests/e2e/test_document_summaries.py -v --no-cov`); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary).
|
|
||||||
- All existing E2E suites stay green (new columns are defaulted; all existing chunks have `is_summary=false`).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] After `uv run python -m scripts.import_docs`, every non-markdown fixture/doc has `documents.summary` set and exactly one `is_summary` chunk (position −1, embedded); markdown docs have neither.
|
|
||||||
- [ ] A question whose best match is a summary chunk yields an answer grounded in the **full source document** (E2E sentinel) and the per-turn log line shows `summary_hits>=1`.
|
|
||||||
- [ ] A lite-model failure during import does **not** drop the document — it is indexed without a summary, logged, and counted (`summary_errors`).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green in isolation; existing suites (`test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py`) stay green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/document-summaries.md` exists.
|
|
||||||
- [ ] `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS` and the summary behavior.
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc`); `.agent/phases/todo/30_document_summaries/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A5 extended, not revised** — the `lite` model is served by the same OpenAI-compatible endpoint (`https://aipi.reeseapps.com/v1`) via a new `BOR_LLM_SUMMARY_MODEL` setting (default `lite`); no new model management, no new package.
|
|
||||||
- **A7 untouched** — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated.
|
|
||||||
- **A9 untouched** — "non-markdown" means every *already-imported* A9 document except `md`/`markdown`. The TODO's quadlet-file example is **out of scope**: `.quadlet` is not an A9 format and `BOR_IMPORT_EXTENSIONS` may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision).
|
|
||||||
- **A13** — migration 0004 adds two columns (`documents.summary`, `chunks.is_summary`); no table rework, both reversible.
|
|
||||||
- **Summary generation is best-effort** — a lite failure logs + counts (`summary_errors`) and the file is still indexed without a summary (same fail-soft spirit as the per-file `EmbeddingError` handling, but weaker: the doc is already committed).
|
|
||||||
- **Pointer is code-deterministic** — the `Source: <source>/<path>` line is appended by `summarizer.py`, never trusted to the model.
|
|
||||||
- **A16 honoured** — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's `SUMMARY_MODE` marker.
|
|
||||||
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# Task 01 — lite model setting + non-streaming chat()
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "The small model available on aipi.reeseapps.com is 'lite'." (enabler for "we need a small model to analyze non markdown documents")`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the aipi **`lite`** model callable from the app: a new model setting plus a non-streaming `LLMClient.chat()` one-shot completion method that the summarizer (task 03) and the phase-31 overview generator will use.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — add `llm_summary_model: str = "lite"` (env `BOR_LLM_SUMMARY_MODEL`), documented like the other LLM settings.
|
|
||||||
2. `app/rag/llm.py` — add to `LLMClient`:
|
|
||||||
- `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str` — `chat.completions.create(model=model or self.settings.llm_summary_model, messages=…, temperature=0.2, max_tokens=2048, stream=False)`; returns the first choice's `message.content` stripped.
|
|
||||||
- Raise `LLMError` (wrapped, with the base URL in the message — same style as `chat_stream`) on any transport/HTTP/malformed failure, and on an empty/missing content field (a silent empty summary must never be stored).
|
|
||||||
3. `.env.example` — document `BOR_LLM_SUMMARY_MODEL` (default `lite`).
|
|
||||||
4. `README.md` — models section: add `lite` (document summaries — this phase; KB overview in phase 31) next to `turbo`/`embed`.
|
|
||||||
|
|
||||||
- ASSUMPTION: the model name is `lite` per the TODO item; single (non-streaming) completion with `temperature=0.2`, `max_tokens=2048` — summaries/outlines are short, so a fixed budget is enough (no new setting).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `tests/unit/test_llm_client.py` — extend the existing mock-transport pattern: `chat()` returns content (trimmed); HTTP ≥400 → `LLMError`; empty content → `LLMError`; explicit `model=` overrides the default (`llm_summary_model`).
|
|
||||||
- Coverage: **>90%** on this task's new/modified code (`app/` TOTAL ≥ pre-change).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `Settings().llm_summary_model == "lite"` by default; `BOR_LLM_SUMMARY_MODEL` env override works (config test).
|
|
||||||
- [ ] `chat()` unit tests green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] No change to `embed`/`chat_stream` behavior (existing suite green).
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
# Task 02 — Migration 0004: summary columns
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "provide a textual summary of those documents with a pointer back to the source" (storage)`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the schema for storing a document's summary and for marking the extra summary chunk: `documents.summary TEXT NULL` and `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `alembic/versions/0004_summary_columns.py` — new revision (down_revision = the 0003 steering-notes revision, whatever `alembic/versions/` currently heads to):
|
|
||||||
- upgrade: `op.add_column("documents", sa.Column("summary", sa.Text(), nullable=True))`; `op.add_column("chunks", sa.Column("is_summary", sa.Boolean(), nullable=False, server_default=sa.text("false")))`.
|
|
||||||
- downgrade: drop both columns.
|
|
||||||
2. `app/models.py` — `Document.summary: Mapped[str | None] = mapped_column(Text, default=None)` (comment: lite-model summary, phase 30); `Chunk.is_summary: Mapped[bool] = mapped_column(Boolean, default=False)` (comment: summary chunk, position −1, phase 30).
|
|
||||||
3. `tests/integration/test_migration_0004.py` — mirror `tests/integration/test_migration_0002.py`'s style: upgrade to head → both columns exist, `is_summary` default `false`; downgrade to 0003 → both gone; upgrade again → back (round-trip).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the migration test above (real Postgres, as `test_migration_0002.py` does).
|
|
||||||
- Coverage: models are exercised by the existing model tests; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB (and `alembic downgrade -1` + `upgrade head` round-trips).
|
|
||||||
- [ ] `uv run pytest` green (including all pre-existing migration/importer tests — `is_summary` default keeps old rows valid).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
# Task 03 — app/rag/summarizer.py (prompt + lite call + pointer)
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source"`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create the summarizer module: build the `lite` prompt for one document, call the model (task 01), validate the output, and return the summary text with a **code-deterministic** pointer line back to the source.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — add `summary_max_chars: int = 12_000` (env `BOR_SUMMARY_MAX_CHARS`): the cap on document content sent to the lite model in one call.
|
|
||||||
2. `app/rag/summarizer.py` (new) —
|
|
||||||
- `SUMMARY_MODE = "SUMMARY_MODE"` — marker constant the E2E mock keys on in the system prompt (same convention as `DEFLECT_MODE`).
|
|
||||||
- `build_summary_prompt(source: str, path: str, content: str, max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`:
|
|
||||||
- system: `SUMMARY_MODE` + instruction — "Write a 3–6 sentence plain-text summary of this document in natural language. Cover what it configures/defines and its most important values. Do not use markdown. Do not invent anything that is not in the document."
|
|
||||||
- user: the document content, capped at *max_chars* (default `get_settings().summary_max_chars`); on overflow cut at the cap and append the shared `TRUNCATION_MARKER` (imported from `app.rag.retriever`).
|
|
||||||
- `async def generate_summary(llm, *, source: str, path: str, content: str) -> str` — calls `llm.chat([{"role":"system",…},{"role":"user",…}], model=llm.settings.llm_summary_model)`; validates non-empty after trim (else raise `LLMError` — the client already does this, but re-assert defensively); appends the deterministic pointer line: `f"\nSource: {source}/{path}"` (the pointer is **never** model-generated).
|
|
||||||
3. `tests/unit/test_summarizer.py` (new) — fake LLM object (duck-typed `chat` + `settings`):
|
|
||||||
- prompt: system contains `SUMMARY_MODE`; user == full content when under cap; user truncated + `TRUNCATION_MARKER` when over cap (custom and default cap).
|
|
||||||
- generation: returned text = model text + pointer line `Source: <source>/<path>`; whitespace model text → `LLMError`; `LLMError` from the client propagates.
|
|
||||||
|
|
||||||
- ASSUMPTION: the pointer is the literal line `Source: <source>/<path>` appended by code (the TODO's "pointer back to the source"); the model is told what to summarize but not to write the pointer.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the tests in Work step 3.
|
|
||||||
- Coverage: **>90%** on `app/rag/summarizer.py`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `generate_summary` returns a non-empty summary ending in the deterministic pointer line; all unit tests green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] No app endpoint change yet (pipeline integration is task 05).
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Task 04 — Importer: generate, store, index summaries (best-effort)
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "a small model to analyze non markdown documents and provide a textual summary… The similarity search will have a higher chance of hitting those summaries than the original document"`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Hook summarization into the import pipeline: every **non-markdown** file gets a lite summary stored on `documents.summary` and indexed as one extra embedded chunk (`is_summary`, position −1) — best-effort, so a lite failure never loses the document.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/importer.py`:
|
|
||||||
- `Embedder` protocol — add `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str: ...` (task 01's `LLMClient.chat` already satisfies it; the protocol is what tests duck-type).
|
|
||||||
- `ImportSummary` — new counters `summaries: int = 0` and `summary_errors: int = 0`; include both in the `log()` summary line (`… summaries=%d summary_errors=%d`).
|
|
||||||
- `_index_file` — **after** the existing doc+chunks commit (so the document is safe):
|
|
||||||
- If `full_path.suffix.lower()` is **not** in `(".md", ".markdown")`: try
|
|
||||||
`summary = await generate_summary(llm, source=source, path=rel, content=content)`;
|
|
||||||
delete any existing `is_summary` chunk of this document (re-import replacement);
|
|
||||||
add `Chunk(document_id=doc.id, position=-1, content=summary, is_summary=True)`;
|
|
||||||
`vector = (await llm.embed([summary]))[0]`; set `chunk.embedding = vector`, `doc.summary = summary`; `session.commit()`; `summary.summaries += 1`; log `import: summary source=%s path=%s chars=%d`.
|
|
||||||
- On `LLMError | EmbeddingError`: `session.rollback()`, `summary.summary_errors += 1`, log `import: summary failed source=%s path=%s — %s`, and **continue** (the document row + content chunks stay committed; `doc.summary` remains NULL).
|
|
||||||
- Markdown files: no summary, `doc.summary` stays NULL, no `is_summary` chunk.
|
|
||||||
2. `scripts/import_docs.py` — the final `print` gains `summaries=%d summary_errors=%d` from the `ImportSummary`.
|
|
||||||
3. `tests/unit/test_importer.py` — extend the existing fake embeder with a `chat` method (deterministic: returns `"Summary of " + first token of content`; raises `LLMError` when the content contains the sentinel word `SUMMARY-BLOWUP`):
|
|
||||||
- non-md file (e.g. `.yaml`) → after import: `doc.summary` set, exactly one `is_summary` chunk at position −1 with a non-NULL embedding; `summary.summaries == 1`.
|
|
||||||
- `.md` file → `doc.summary is None`, no `is_summary` chunk, `summaries == 0`.
|
|
||||||
- fail-soft: content with `SUMMARY-BLOWUP` → document fully indexed (chunks present, embedding set), `doc.summary is None`, `summary_errors == 1`, no exception.
|
|
||||||
- replacement: re-import the same file with changed content → still exactly **one** `is_summary` chunk (old one deleted), new text.
|
|
||||||
- `log()` line includes the new counters (existing log-format test updated accordingly).
|
|
||||||
|
|
||||||
- ASSUMPTION: the summary chunk sits at `position = -1` (content chunks stay 0-based in order) so chunk ordering and the viewer are undisturbed; only one summary chunk per document at a time.
|
|
||||||
- ASSUMPTION: "non-markdown" = suffix not in `(.md, .markdown)` — every other A9 format (txt, yaml, yml, json, py) gets a summary.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the tests in Work step 3 (reuse the existing test file's session/fake-embedder fixtures).
|
|
||||||
- Coverage: **>90%** on the modified `app/rag/importer.py`; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All new unit tests green; existing importer tests green (protocol change is additive).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] A manual `uv run python -m scripts.import_docs` run against the dev KB logs `summaries=N` for the non-md docs (observable in the importer log line, PLAN §9).
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# Task 05 — Pipeline: summary hits resolve to the full source document
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "the chat LLM can retrieve the source pointed to by the summary document (so as part of the pipeline: if retrieval == summary, fetch documents referenced by summary)"`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the retrieval + chat pipeline summary-aware: carry `is_summary` through both candidate lists into the fused result, and record in the per-turn log how many of the selected documents were hit via their summary chunk. The context assembly itself is **unchanged** — a summary chunk's parent *is* the source document, and `select_documents` already feeds the full document (A7 revised / phase 24); this task makes that resolution explicit, asserted, and observable.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/retriever.py`:
|
|
||||||
- `RetrievedChunk` — add field `is_summary: bool = False` (documented: True for the lite-model summary chunk, position −1).
|
|
||||||
- `_vector_candidates` — select `Chunk.is_summary` and pass it into the constructed `RetrievedChunk`s.
|
|
||||||
- `_LEXICAL_SQL` — add `c.is_summary AS is_summary`; `_lexical_candidates` passes `row.is_summary`.
|
|
||||||
- `fuse` needs no change (dataclass passthrough) — but assert in tests that the flag survives fusion.
|
|
||||||
2. `app/api/chat.py`:
|
|
||||||
- `TurnPlan` — add `summary_hits: int = 0` (count of selected-document hit chunks with `is_summary`).
|
|
||||||
- `plan_turn` — after the selected-docs decision is known for each branch, compute `summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)` and store it on the `TurnPlan` (both HIGH and LOW branches).
|
|
||||||
- Per-turn log line — add `summary_hits=%d` after `fts_hits=%d` (PLAN §9 line extension; record it in the phase's locked decisions). Update any existing test that asserts the log line format verbatim.
|
|
||||||
- **No change** to `build_high_prompt`/`build_deflect_prompt` inputs or to `select_documents` — the full source document of a summary hit already lands in `<documents>`; task 06's E2E proves it end-to-end.
|
|
||||||
3. `tests/unit/test_retriever.py` — `is_summary` survives: vector candidates (flag set), lexical candidates (flag set), `fuse` (both a double-hit and a summary-only lexical hit keep the flag; default stays `False` for legacy chunks).
|
|
||||||
4. `tests/unit/test_chat_gate.py` — `plan_turn`: a summary chunk on a selected top document → `summary_hits == 1`; a summary chunk on a document **outside** the top-N selection → not counted; no summaries → `0` (existing cases unchanged).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the tests in Work steps 3–4; existing chat-gate and retriever tests stay green (new field is defaulted).
|
|
||||||
- Coverage: **>90%** on modified `app/rag/retriever.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] A summary chunk retrieved via vector **or** lexical carries `is_summary=True` through `retrieve()` (unit).
|
|
||||||
- [ ] The per-turn log line (PLAN §9) now reads `… fts_hits=… summary_hits=… …` and existing log-format tests are updated + green.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] No prompt change: HIGH/LOW prompts byte-identical for summary-less KBs (covered by existing prompt tests).
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
# Task 06 — Deterministic lite mock + story E2E + docs + commit
|
|
||||||
|
|
||||||
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — (whole item: bad embedder context for non-markdown docs → lite summaries → summary hits fetch the referenced source; the aipi 'lite' model)`
|
|
||||||
**Story:** `.agent/user_stories/document-summaries.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Close the loop: a deterministic `lite` behavior in the E2E mock, a sentinel fixture proving that **a summary hit still delivers the full source document to the LLM**, the story E2E suite, the story file, README docs, and the phase commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/mock_llm.py` — in the chat-completions handler (non-stream and stream paths), **before** the `DEFLECT_MODE` check: if the system prompt contains `SUMMARY_MODE`, return the deterministic digest
|
|
||||||
`f"This document covers {' '.join(TOKEN_RE.findall(_user(body).lower())[:24])}."`
|
|
||||||
— the first 24 tokens of the document content (the summarizer puts the content in the *user* message). Byte-stable for a given fixture.
|
|
||||||
2. E2E fixture — a new non-markdown fixture doc, e.g. `quadlet/qwen-llamacpp.yaml` under the existing E2E fixture KB (follow the import-dependent fixtures' pattern in `tests/e2e/conftest.py` / the fixture dir used by `test_whole_document_context.py`):
|
|
||||||
- The document **opens** with a header comment line dense in the question tokens (e.g. `# qwen 3.8 llama.cpp optimal parameters deployment notes`) so the mock's 24-token summary digest contains the question's words, followed by ~4–5 k of other yaml content (so the raw chunks dilute their overlap and the summary chunk ranks first — the mock's embeddings are a pure function of tokens, so the ranking is fully deterministic for a fixed fixture; iterate the fixture text until the E2E assertions hold).
|
|
||||||
- A unique sentinel `RESE-SUMMARY-SENTINEL-7f3a` on the **last line** of the document (outside the 24-token digest, unreachable from the summary).
|
|
||||||
3. `tests/e2e/test_document_summaries.py` (new, the story gate) — reuse the E2E conftest app/DB fixtures:
|
|
||||||
- Import the fixture KB (re-import pattern used by import-dependent stories).
|
|
||||||
- Ask the question (e.g. "What are the optimal parameters for qwen 3.8 on llama.cpp? show the end of your notes" — the phase-24 tail-echo trigger makes the answer quote the **last 160 chars of the document context**).
|
|
||||||
- Assert the rendered brain answer contains `RESE-SUMMARY-SENTINEL-7f3a` → the full **source** document was in the LLM context (only possible via the summary→parent-document resolution, since the summary digest cannot contain the sentinel).
|
|
||||||
- Assert the source chip shows the fixture doc's path and `deflected` is false.
|
|
||||||
- Control: a markdown fixture doc in the same KB gets **no** summary chunk — assert via the Sources table (admin, `#docs-table`) or a direct DB check in the test: markdown doc's chunk count == raw chunks only; the yaml doc has exactly one `is_summary` row.
|
|
||||||
4. `.agent/user_stories/document-summaries.md` (new) — narrative + acceptance criteria + Playwright mapping rule (story → `tests/e2e/test_document_summaries.py`), matching the style of `.agent/user_stories/git-sources.md`.
|
|
||||||
5. `README.md` — new "Document summaries" section: what gets summarized (non-markdown A9 docs), the `Source:` pointer, `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS`, fail-soft behavior, and how summary hits appear in the per-turn log.
|
|
||||||
6. Commit: `git add` the phase's app/script/test/README files; `git commit --no-gpg-sign -m "feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc"`; move `.agent/phases/todo/30_document_summaries/` → `.agent/phases/complete/` (`.agent/` is gitignored by design — force-add only if the commit must record the plan change, otherwise leave the move out of git).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green **in isolation** (Chromium installed; `podman compose up -d db` up; mock LLM — no live aipi needed).
|
|
||||||
- Regression: run `test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py` in isolation — all stay green.
|
|
||||||
- Full gate: `uv run pytest` + `uv run pytest --cov=app --cov-report=term-missing` (TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `tests/e2e/test_document_summaries.py` green in isolation; the sentinel assertion proves summary hit → full source document.
|
|
||||||
- [ ] All regression suites listed above green in isolation.
|
|
||||||
- [ ] Full test gate + lint/type gate green (per this phase's 00_phase.md).
|
|
||||||
- [ ] Story file + README + `.env.example` complete; one `--no-gpg-sign` commit made.
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# Phase 31 — KB Overview in the System Prompt (lite-generated knowledge-base outline)
|
|
||||||
|
|
||||||
**Source:** `TODO.md L4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents. This part of the system prompt should be generated by the lite model and should be stored somewhere so it can be updated whenever we import new documents."`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
**Context:** Phase 15 steering notes (the `<tuning>` prompt section, its char budget, and the **byte-identical-when-absent** convention — `app/rag/prompts.py::build_steering_section`), phase 30 (the `lite` client method `LLMClient.chat`, and per-document summaries that make a much better overview input than raw titles), `scripts/import_docs.py` (the place "whenever we import new documents" happens), and the E2E mock's answer-echo convention (the `(tuning: …)` suffix — `tests/e2e/mock_llm.py`).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Store a lite-generated, plain-text outline of the knowledge base's basic **categories** in a single-row `kb_overview` table, inject it into **both** chat prompts (HIGH and LOW) as a `<knowledge_base>` section so the agent knows roughly what the KB contains before retrieval, and regenerate it automatically whenever an import changes the KB.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `30_document_summaries` (todo) — `LLMClient.chat` + `BOR_LLM_SUMMARY_MODEL` (task 01) and the stored per-document summaries (task 04) that feed the overview input.
|
|
||||||
- `15_steering_notes` (complete) — the prompt-section pattern this phase mirrors (budget, marker, byte-identical-when-absent, per-turn load in `app/api/chat.py`).
|
|
||||||
- `11_long_answers` / README import workflow (complete) — `scripts/import_docs.py`'s `main()` structure, which this phase extends with the post-import regeneration.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_migration_kb_overview.md` — Alembic 0005: single-row `kb_overview` table + `KbOverview` model.
|
|
||||||
2. `02_overview_generator.md` — `app/rag/overview.py`: `KB_OVERVIEW_MODE` prompt builder, `load_kb_overview`, `regenerate_overview` (best-effort upsert).
|
|
||||||
3. `03_prompt_injection.md` — `<knowledge_base>` section in HIGH + LOW prompts (budgeted, byte-identical when absent); `plan_turn`/chat wire it in; `kb_chars` in the per-turn log.
|
|
||||||
4. `04_import_trigger.md` — `import_docs` regenerates the overview after a KB-changing import (shared with phase 32's sync).
|
|
||||||
5. `05_mock_and_e2e.md` — deterministic `KB_OVERVIEW_MODE` mock + `(kb: …)` echo, `tests/e2e/test_kb_overview.py`, story file, commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: overview generator (prompt build/cap, load, regenerate upsert/fail-soft/zero-docs), prompts (section present/budgeted/absent → byte-identical, ordering vs `<tuning>`), chat gate (`kb_chars`, prompt carries the section).
|
|
||||||
- Integration: migration 0005 up/down; `import_docs` regeneration trigger (changed vs unchanged imports, failure isolation).
|
|
||||||
- Coverage: **>90%** on `app/` (`app/` TOTAL ≥ pre-change).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_kb_overview.py` — one story, run **in isolation**; the injected section is observable in the mock answer via the `(kb: …)` echo (steering precedent).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] After a KB-changing import, `kb_overview` holds a fresh outline (log line `overview: regenerated docs=… chars=…`); an unchanged re-import does **not** call the lite model.
|
|
||||||
- [ ] Every chat turn's system prompt (HIGH and LOW) contains the `<knowledge_base>` section when a row exists; with no row, both prompts are **byte-identical** to the pre-phase text (unit-asserted).
|
|
||||||
- [ ] The per-turn log line records `kb_chars=<n>`; section overflow beyond `BOR_KB_OVERVIEW_MAX_CHARS` is capped with the shared `[…truncated…]` marker.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; existing prompt/steering/chat suites stay green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/kb-overview-prompt.md` exists; `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` / `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior.
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts`); `.agent/phases/todo/31_kb_overview_prompt/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A13** — migration 0005 adds one single-row table `kb_overview` (`id INTEGER PK DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`, `updated_at TIMESTAMPTZ`); no other schema change.
|
|
||||||
- **A5 extended** — the overview is generated by the same `lite` model via the same `BOR_LLM_SUMMARY_MODEL` setting and `LLMClient.chat` (phase 30); no new model or package.
|
|
||||||
- **Prompt-section convention (phase 15 precedent)** — the section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default **4000**) with the shared `TRUNCATION_MARKER` overflow; **zero/empty row → prompts byte-identical** to pre-phase text. Section order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body.
|
|
||||||
- **Regeneration is best-effort and change-gated** — runs only when an import added/updated at least one document (or no row exists yet); a lite failure logs and leaves the previous overview intact (an old outline is better than none).
|
|
||||||
- **Overview input is capped** — `BOR_OVERVIEW_INPUT_MAX_CHARS` (default **40 000**) on the document list (source/path/title/first summary line) sent to the model.
|
|
||||||
- **No per-turn LLM call** — chat turns only *read* the stored row (one indexed PK lookup); generation happens at import/sync time (phase 32's button triggers the same `regenerate_overview`).
|
|
||||||
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
# Task 01 — Migration 0005: kb_overview table
|
|
||||||
|
|
||||||
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be stored somewhere so it can be updated whenever we import new documents"`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create the storage for the knowledge-base outline: a single-row `kb_overview` table and its SQLAlchemy model.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `alembic/versions/0005_kb_overview.py` — new revision (down_revision = phase 30's 0004):
|
|
||||||
- upgrade: create table `kb_overview` (`id INTEGER` PK `server_default sa.text("1")`, `content TEXT NOT NULL server_default sa.text("''")`, `updated_at TIMESTAMPTZ NOT NULL server_default=sa.func.now()`).
|
|
||||||
- downgrade: drop the table.
|
|
||||||
2. `app/models.py` — `class KbOverview(Base)`: `id: Mapped[int] = mapped_column(Integer, primary_key=True, server_default="1")`, `content: Mapped[str] = mapped_column(Text, server_default="")`, `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())`. Docstring: single row, lite-generated KB outline, phase 31.
|
|
||||||
3. `tests/integration/test_migration_0005.py` — same style as `test_migration_0002.py` / `test_migration_0004.py`: upgrade → table exists with the three columns and defaults; downgrade → gone; upgrade again → back.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the migration test above (real Postgres).
|
|
||||||
- Coverage: model exercised by existing model-test patterns; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run alembic upgrade head` clean on the dev DB; round-trip with `alembic downgrade -1` + `upgrade head`.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Task 02 — app/rag/overview.py (generator + loader)
|
|
||||||
|
|
||||||
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "basic categories of everything that's been read… generated by the lite model… updated whenever we import new documents"`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Create the overview generator module: build the `lite` prompt from the document catalogue, generate the outline, store it in the single row, and expose a cheap loader for the chat path.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — add:
|
|
||||||
- `kb_overview_max_chars: int = 4_000` (`BOR_KB_OVERVIEW_MAX_CHARS`) — prompt-section budget (task 03).
|
|
||||||
- `overview_input_max_chars: int = 40_000` (`BOR_OVERVIEW_INPUT_MAX_CHARS`) — cap on the document list sent to the model.
|
|
||||||
2. `app/rag/overview.py` (new):
|
|
||||||
- `KB_OVERVIEW_MODE = "KB_OVERVIEW_MODE"` — marker the E2E mock keys on (same convention as `SUMMARY_MODE` / `DEFLECT_MODE`).
|
|
||||||
- `build_overview_prompt(rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`. Each row is `(source, path, title, summary)`; system = `KB_OVERVIEW_MODE` + instruction ("From the document list below, write a compact plain-text outline of the basic categories and topics this knowledge base covers. Group by source where useful, use `-` bullet lines, at most ~1500 characters, no markdown headings, and no topics not present in the list."); user = one line per doc `source — path — title — {first line of summary or ''}` joined by newlines, capped at *max_chars* (default `overview_input_max_chars`, overflow → shared `TRUNCATION_MARKER`).
|
|
||||||
- `load_kb_overview(db: Session) -> str` — the single row's `content` (trimmed) or `""` when the row is missing/empty.
|
|
||||||
- `async def regenerate_overview(llm, session: Session | None = None) -> bool` — load all documents (`source, path, title, summary` ordered by source, path); **zero documents → leave the existing row untouched, return False**; build the prompt; `text = await llm.chat([system, user], model=llm.settings.llm_summary_model)`; upsert the single row (`id=1`, `content=text`, `updated_at=now(UTC)`); commit; log `overview: regenerated docs=%d chars=%d`; return True. On `LLMError`: log `overview: regeneration failed — %s` and return False (previous row stays — see phase locked decisions).
|
|
||||||
3. `tests/unit/test_overview.py` (new) — fake LLM (duck-typed `chat` + `settings`), in-memory/SQLite session where the existing test infra allows (else a real-DB integration test in the same style as `test_steering.py`):
|
|
||||||
- prompt: system contains `KB_OVERVIEW_MODE`; user lines carry source/path/title/first summary line; cap truncates + marker.
|
|
||||||
- `load_kb_overview`: no row → `""`; row present → content.
|
|
||||||
- `regenerate_overview`: happy path upserts (content + fresh `updated_at`, returns True); zero docs → no DB write, returns False; `LLMError` → previous row unchanged, returns False.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: the tests in Work step 3.
|
|
||||||
- Coverage: **>90%** on `app/rag/overview.py`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `regenerate_overview` is idempotent (single row, always id=1) and fail-soft; all tests green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
# Task 03 — `<knowledge_base>` section in both prompts + chat wiring
|
|
||||||
|
|
||||||
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents"`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Inject the stored overview into **every** chat turn's system prompt (HIGH and LOW modes) as a budgeted `<knowledge_base>` section — absent row → byte-identical prompts — and record `kb_chars` in the per-turn log line.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/prompts.py`:
|
|
||||||
- `build_kb_section(overview: str, max_chars: int | None = None) -> str` — empty/whitespace → `""`; otherwise `<knowledge_base>\n` + intro line ("The basic categories of everything in this knowledge base (generated at import time):") + the overview content, budgeted at *max_chars* (default `get_settings().kb_overview_max_chars`) with the shared `TRUNCATION_MARKER` for overflow (exact pattern of `build_steering_section`, including its pathological-budget handling).
|
|
||||||
- `build_high_prompt(documents, notes=None, kb_overview: str | None = None)` and `build_deflect_prompt(titles, notes=None, kb_overview: str | None = None)` — insert the section **between `<relevance>` and the `<tuning>` section** (i.e. order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body); with an empty overview the output is byte-identical to today's text in both modes.
|
|
||||||
2. `app/api/chat.py`:
|
|
||||||
- Load per turn: `kb_overview = load_kb_overview(db)` next to the steering-notes load (one PK lookup — no LLM call).
|
|
||||||
- `plan_turn(chunks, settings, notes=None, kb_overview: str | None = None)` — pass it to both prompt builders; `TurnPlan` gains `kb_chars: int = 0` (length of the stored overview text when a non-empty row exists, else 0).
|
|
||||||
- Per-turn log line (PLAN §9): add `kb_chars=%d` after `tuning=%d`. Update any existing test asserting the log line format verbatim.
|
|
||||||
3. `tests/unit/test_prompts.py` —
|
|
||||||
- HIGH: no overview → byte-identical to the pre-phase builder output (build the expected string with `notes=None, kb_overview=None`); with overview → section present, ordered before `<tuning>` when both exist.
|
|
||||||
- LOW: same pair of assertions (deflection prompt).
|
|
||||||
- budget: overview longer than `kb_overview_max_chars` → capped + `TRUNCATION_MARKER`.
|
|
||||||
4. `tests/unit/test_chat_gate.py` — `plan_turn` with an overview: both branches' `system_prompt` contains the section; `TurnPlan.kb_chars` == len(overview); empty overview → `kb_chars == 0` and prompt unchanged.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: Work steps 3–4; existing steering/prompt/gate tests stay green (new param is defaulted).
|
|
||||||
- Coverage: **>90%** on modified `app/rag/prompts.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] HIGH and LOW prompts are byte-identical to pre-phase text when no overview row exists (unit-asserted against the exact strings).
|
|
||||||
- [ ] With a row, both prompts carry the budgeted `<knowledge_base>` section in the locked order; the per-turn log line shows `kb_chars=…`.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Task 04 — import_docs regenerates the overview after a KB-changing import
|
|
||||||
|
|
||||||
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be updated whenever we import new documents"`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Wire the "update whenever we import new documents" trigger into the import script: after an import that changed the KB, regenerate the stored overview (best-effort). Phase 32's admin sync button reuses the exact same `regenerate_overview` call.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `scripts/import_docs.py`:
|
|
||||||
- Restructure `main()`'s single `asyncio.run(import_sources(…))` into one `async def _run()` that (a) runs `import_sources(sources, llm, prune=args.prune, limit=args.limit)` and (b) — when the summary has `added + updated > 0` **or** no overview row exists yet — awaits `regenerate_overview(llm)` (import it from `app.rag.overview`). One event loop, same `LLMClient` instance.
|
|
||||||
- `--limit` debug runs skip the regeneration (an incomplete walk must not rewrite the outline — mirrors the existing `--prune`-with-`--limit` guard).
|
|
||||||
- The final `print` gains `overview=updated|skipped|failed` (failed = `regenerate_overview` returned False via LLMError; the import's own exit code is **unchanged** — a failed outline must not fail the import).
|
|
||||||
- `Limit` guard: when `limit` is set, `added + updated > 0` does *not* trigger regeneration (log `overview: skipped (--limit)`).
|
|
||||||
2. `tests/integration/test_import_docs_overview.py` (new) — with the git sync mocked out (reuse `test_import_docs_git.py`'s mocking style) and a fake LLM whose `chat` records calls:
|
|
||||||
- import with changed files → `kb_overview` row written; `chat` called once; print shows `overview=updated`.
|
|
||||||
- unchanged re-import (same hashes) → `chat` **not** called; print shows `overview=skipped`.
|
|
||||||
- `chat` raising `LLMError` → exit code still `0` (no import errors), print shows `overview=failed`, previous row untouched.
|
|
||||||
- `--limit` run with changes → `overview=skipped`.
|
|
||||||
- first-ever import (no row) with zero *changed* docs is not possible (new docs are "added") — but an empty-source run with no row → no row created, `overview=skipped`.
|
|
||||||
3. `.env.example` — `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000), `BOR_OVERVIEW_INPUT_MAX_CHARS` (default 40000).
|
|
||||||
4. `README.md` — import workflow section: the import now refreshes the KB overview after a KB-changing run (fail-soft, `overview=` token in the summary line).
|
|
||||||
|
|
||||||
- ASSUMPTION: "whenever we import new documents" = whenever an import **added or updated** at least one document (or the row doesn't exist yet); unchanged re-imports and `--limit` debug runs do not burn a lite call.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: Work step 2 (real Postgres, mocked git + fake LLM — no live aipi).
|
|
||||||
- Coverage: `app/` TOTAL ≥ pre-change (the script change is covered by the integration tests; the script itself is outside the `app/` gate).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All new integration tests green; `tests/integration/test_import_docs_git.py` stays green.
|
|
||||||
- [ ] A manual run (`uv run python -m scripts.import_docs`) against the dev KB logs `overview: regenerated docs=… chars=…` after a KB-changing import and `overview=skipped` otherwise.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# Task 05 — Deterministic KB_OVERVIEW_MODE mock + story E2E + commit
|
|
||||||
|
|
||||||
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — (whole item: system prompt injects basic categories of everything read, lite-generated, stored, updated on import)`
|
|
||||||
**Story:** `.agent/user_stories/kb-overview-prompt.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the overview observable end-to-end in a deterministic E2E: the mock generates a `KB_OVERVIEW_MODE` outline and echoes the injected section into its answer (the `(tuning: …)` precedent), plus the story suite, story file, and the phase commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/mock_llm.py`:
|
|
||||||
- Generation: in the chat-completions handler, if the system prompt contains `KB_OVERVIEW_MODE` → return the deterministic outline `f"Knowledge base outline:\n- {first 8 tokens of _user(body), space-joined}"` (the user message carries the document list).
|
|
||||||
- Echo: in `compose_answer` (all answer paths), if the system prompt contains a `<knowledge_base>` section, append `(kb: <first bullet line of the section>)` — parse with a regex in the `first_tuning_note` style (skip the intro line, take the first `-` line, strip the dash). This mirrors the steering echo exactly.
|
|
||||||
2. `tests/e2e/test_kb_overview.py` (new, the story gate):
|
|
||||||
- Seed the `kb_overview` row directly in the DB (the E2E test has DB access via the conftest fixtures — content with a recognizable first bullet, e.g. `- Kubernetes cluster and node maintenance notes`), so the test exercises the **injection** path deterministically (the CLI trigger path is covered by task 04's integration tests).
|
|
||||||
- Ask a normal on-topic question (the fixture KB is already imported by the conftest pattern) → assert the rendered brain answer ends with `(kb: Kubernetes cluster and node maintenance notes)`.
|
|
||||||
- Deflection control: ask an off-topic question (deflection path) → the answer still carries the `(kb: …)` echo (the section is in the LOW prompt too).
|
|
||||||
- Absence control: delete the row → a fresh question's answer has **no** `(kb: …)` suffix (byte-identical prompt behavior is unit-asserted in task 03; this proves it end-to-end).
|
|
||||||
3. Integration test (same task): extend `tests/integration/test_chat_api.py` (or a new `test_kb_overview_api.py`) with the capturing-fake-LLM pattern — no row: the system prompt sent to the model equals the pre-phase construction (assert the exact string via the existing `build_high_prompt`/`build_deflect_prompt` with `kb_overview=None`); row present: it contains the section in both HIGH and LOW turns.
|
|
||||||
4. `.agent/user_stories/kb-overview-prompt.md` (new) — narrative + acceptance criteria + Playwright mapping rule, styled like the other story files.
|
|
||||||
5. Commit: `git commit --no-gpg-sign -m "feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts"`; move `.agent/phases/todo/31_kb_overview_prompt/` → `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green **in isolation**.
|
|
||||||
- Regression: `test_steering.py`, `test_chat_rag.py`, `test_honest_deflection.py` stay green in isolation (prompt change is additive and defaulted).
|
|
||||||
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `tests/e2e/test_kb_overview.py` green in isolation (injection, deflection, and absence all asserted).
|
|
||||||
- [ ] Integration prompt-capture tests green; existing steering/chat suites green.
|
|
||||||
- [ ] Full test + lint/type gates green (per this phase's 00_phase.md).
|
|
||||||
- [ ] Story file + `.env.example` + README complete; one `--no-gpg-sign` commit made.
|
|
||||||
@@ -1,45 +0,0 @@
|
|||||||
# Phase 32 — Admin Sync Button (one-click doc import sync)
|
|
||||||
|
|
||||||
**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"`
|
|
||||||
**Story:** `.agent/user_stories/admin-sync-button.md`
|
|
||||||
**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it.
|
|
||||||
- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git).
|
|
||||||
- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button.
|
|
||||||
- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin).
|
|
||||||
2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions.
|
|
||||||
3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does).
|
|
||||||
- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic.
|
|
||||||
- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings.
|
|
||||||
- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403.
|
|
||||||
- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure.
|
|
||||||
- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop.
|
|
||||||
- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites).
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`).
|
|
||||||
- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently).
|
|
||||||
- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed.
|
|
||||||
- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running).
|
|
||||||
- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…".
|
|
||||||
- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Task 01 — Sync API: in-process runner + status
|
|
||||||
|
|
||||||
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"`
|
|
||||||
**Story:** `.agent/user_stories/admin-sync-button.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/api/sync.py` (new):
|
|
||||||
- `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`.
|
|
||||||
- `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null).
|
|
||||||
- `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`.
|
|
||||||
- `async def _run_sync()`:
|
|
||||||
1. `_status.state = "running"`, `started_at = now(UTC)`.
|
|
||||||
2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`.
|
|
||||||
3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr).
|
|
||||||
4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision).
|
|
||||||
5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`.
|
|
||||||
6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`.
|
|
||||||
7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`.
|
|
||||||
2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers).
|
|
||||||
3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern):
|
|
||||||
- anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403.
|
|
||||||
- admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`).
|
|
||||||
- 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409.
|
|
||||||
- failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called.
|
|
||||||
- empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message.
|
|
||||||
- prune: assert the monkeypatched `import_sources` received `prune=True`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB).
|
|
||||||
- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change).
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Task 02 — The admin-only Sync button on Sources (§7.4 feedback)
|
|
||||||
|
|
||||||
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"`
|
|
||||||
**Story:** `.agent/user_stories/admin-sync-button.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page):
|
|
||||||
- `<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">` — a small refresh-cycle `<svg aria-hidden="true">` icon (spin it via CSS in the running state) + `<span class="sync-label" id="sync-label">Sync sources</span>`.
|
|
||||||
- `<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>` right after the button (announces last-result / counts to screen readers).
|
|
||||||
- Update the `.page-sub` copy: it still says "Re-run the import to refresh" — extend it to mention the button (e.g. "…or hit **Sync sources** in the header to clone the repos and re-import.").
|
|
||||||
2. `frontend/assets/header.js` — add `#sync-btn` to the **admin reveal** that already handles `#nav-sources` / `#nav-tuning` after `fetchIsAdmin()` (one fetch, no extra whoami call); anonymous users never see it (stays `hidden`).
|
|
||||||
3. `frontend/assets/sources.js` — the sync state machine (new, isolated section):
|
|
||||||
- On load (admin only — `header.js` exposes the whoami result or a shared `isAdmin` flag; reuse whatever mechanism it already provides for the nav reveals): `GET /api/sync/status` →
|
|
||||||
- `running` → enter the running state and start polling (the user may have reloaded mid-sync).
|
|
||||||
- `success` / `failed` → render the last result (below) but keep the button ready for a fresh sync.
|
|
||||||
- Click → `POST /api/sync` → `202` → running state: button `disabled` + `aria-busy="true"`, icon spinning, label **"Syncing…"**, start polling `GET /api/sync/status` every **2000 ms**.
|
|
||||||
- Terminal state (stop polling):
|
|
||||||
- `success` → enabled, icon reset, label **"Synced HH:MM"** (local time of `finished_at`), `#sync-result` = `"{added} added · {updated} updated · {pruned} pruned"` (omit zero terms) — announced via `aria-live`.
|
|
||||||
- `failed` → enabled, label **"Sync sources"** (retry-ready), and show the page error banner (the existing `role="alert"` pattern used elsewhere in `sources.js`, or the chat error-banner markup style) with the `error` text; `#sync-result` cleared.
|
|
||||||
- `409` on POST (a run started elsewhere) → just enter running state + polling (adopt the in-flight run); `403` → treat as not-admin (hide the button — defense in depth).
|
|
||||||
- **No client-side hard timeout** (phase locked decision — the poll is the feedback loop; the server state is authoritative).
|
|
||||||
4. `frontend/assets/styles.css` — `.sync-btn` styled like `.new-chat-btn`/`.auth-link` (dark tech theme tokens; text contrast ≥4.5:1 — use the dark-ink-on-brand pairing per PLAN §7.2 if the button is filled, else soft-ink on surface), `.sync-btn[disabled]` state, `.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite }` with the existing `prefers-reduced-motion` opt-out, `:focus-visible` 3px outline, ≥44 px touch target on mobile.
|
|
||||||
5. `tests/unit/test_sync_button.py` (new, frontend-assertion style of `test_shared_header.py` / `test_frontend_feedback.py`):
|
|
||||||
- `sources.html` contains `#sync-btn` with `hidden`, `aria-label="Sync sources"`, and `#sync-result` with `role="status"` + `aria-live="polite"`.
|
|
||||||
- `header.js` reveals `#sync-btn` in the admin branch (assert the element id appears in the reveal logic, same as `#nav-sources`).
|
|
||||||
- `sources.js` references `/api/sync` (POST + status GET), the 2000 ms poll, `409` adoption, `403` hide, and the terminal labels (`Syncing…` / `Synced` / failure banner).
|
|
||||||
- no-CDN integration test stays green (no new external references).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: Work step 5 (frontend-assertion tests); no-CDN integration test green.
|
|
||||||
- Coverage: `app/` TOTAL unchanged (frontend-only task); the UI is gated end-to-end by task 03's E2E.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Anonymous: `#sync-btn` never leaves `hidden` in the DOM; admin: it is revealed without a page reload round-trip beyond the existing whoami fetch.
|
|
||||||
- [ ] Full lifecycle works against the dev server (manual check): click → "Syncing…" (disabled) → "Synced HH:MM" + counts, or error banner with retry; reload mid-sync re-enters the running state.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): labeled + focus-visible + contrast ≥4.5:1 + `aria-live` result; reduced-motion respected; no CDN.
|
|
||||||
- [ ] `uv run pytest` green (including the new frontend-assertion tests); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# Task 03 — Story E2E (real file:// git fixture) + README + commit
|
|
||||||
|
|
||||||
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — (whole item: admin-only button → clone the relevant repos → run the import script)"`
|
|
||||||
**Story:** `.agent/user_stories/admin-sync-button.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The story gate: a deterministic E2E that runs the **real** sync path end-to-end (real `git clone` of a local `file://` fixture repo, real import against the mock LLM, real overview regeneration) and verifies both the admin-only visibility and the full button lifecycle, plus README docs, story file, and the phase commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_sync_button.py` (new) — module fixtures:
|
|
||||||
- Build a temp git repo in a `tmp_path` via `subprocess` (`git init -q`, write `notes/sync-fixture.md` containing a unique sentinel `RESE-SYNC-SENTINEL-9b2c`, `git add -A && git -c user.email=e@x -c user.name=t commit -qm one`); set the app's env for this module: `BOR_GIT_SOURCES=file://<repo>`, `BOR_SOURCES_DIR=<tmp_path>/checkouts` (follow the E2E conftest pattern for per-module app env; git is a documented environment prerequisite — phase 28).
|
|
||||||
- Truncate `query_log` (and the KB tables the fixture needs) per the existing E2E isolation pattern so the run starts clean.
|
|
||||||
2. Tests (in isolation):
|
|
||||||
- **`test_anonymous_sees_no_button`** — load `/sources.html` logged out: `#sync-btn` is `hidden` (or absent from the revealed DOM); `POST /api/sync` via `page.request` → 403.
|
|
||||||
- **`test_admin_sync_lifecycle`** — sign in (reuse `tests/e2e/auth_helpers.py`):
|
|
||||||
- `#sync-btn` visible with label "Sync sources".
|
|
||||||
- Click → button disabled, label "Syncing…".
|
|
||||||
- Wait (poll with Playwright, generous timeout ~60 s — real git + embed against the mock LLM): label becomes `Synced …`, `#sync-result` shows `1 added` (the fixture doc).
|
|
||||||
- The Sources table (`#docs-tbody`) now contains the fixture path `notes/sync-fixture.md`; the `kb_overview` row is non-empty (phase-31 regeneration ran — DB check in the test).
|
|
||||||
- Re-click → a second run completes with `0 added · 1 unchanged` (idempotent pull + hash-skip).
|
|
||||||
- **`test_double_trigger_409`** (integration-level, may live in `tests/integration/test_sync_api.py` if E2E timing is too flaky — the integration test already covers this; include here only if deterministic): start a sync, immediately `POST /api/sync` again → 409.
|
|
||||||
3. `README.md` — new "Sync from the UI" subsection under the import workflow: what the button does (clone/pull → import `--prune`-equivalent → KB overview refresh), the states (Syncing…/Synced/error), the 409 behavior, and prerequisites (`BOR_GIT_SOURCES` set; git available).
|
|
||||||
4. `.agent/user_stories/admin-sync-button.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
|
|
||||||
5. Commit: `git commit --no-gpg-sign -m "feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status"`; move `.agent/phases/todo/32_admin_sync_button/` → `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` + git on PATH; mock LLM — no live aipi).
|
|
||||||
- Regression in isolation: `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py`, `test_global_tuning.py`.
|
|
||||||
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All E2E tests green in isolation, including the idempotent second run and the `kb_overview` freshness check.
|
|
||||||
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
|
|
||||||
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Phase 33 — Cache Busting (un-stick the pages)
|
|
||||||
|
|
||||||
**Source:** `TODO.md L6 — "We need better cache busting, the pages are too sticky"`
|
|
||||||
**Story:** `.agent/user_stories/cache-busting.md`
|
|
||||||
**Context:** `app/main.py` serves the whole `frontend/` directory through one `StaticFiles(html=True)` catch-all mount; the five HTML pages reference assets **without any version** (`href="/assets/styles.css"`, `src="assets/markdown.js"`, `src="/assets/app.js"`, …), so browsers happily keep stale CSS/JS/HTML after a deploy — the "too sticky" report. The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) asserts no `https://` references — appending `?v=` keeps every reference same-origin, so it stays green. SSE/API live under `/api/*` and must be untouched (SSE already ships `Cache-Control: no-cache` itself).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
A deploy must be visible without a hard refresh: HTML pages are **always revalidated** (`Cache-Control: no-cache`) and reference their assets with a version token (`?v=<token>`); assets are served **immutable for 1 year** (the token in the URL identifies the content, so long caching is safe). Zero new services, zero build-step changes, no CDN.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `32_admin_sync_button` (todo) — sequencing only; no shared code (this phase is transport-layer and independent of the RAG work).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_asset_version_token.md` — `app/core/caching.py::asset_version()`: git short SHA (homelab checkouts have a `.git`), stable mtime+size content-hash fallback, computed once per process.
|
|
||||||
2. `02_caching_middleware.md` — the response middleware (HTML `no-cache` + `?v=` rewrite; `/assets/*` immutable) wired into `create_app` + integration tests.
|
|
||||||
3. `03_e2e_and_docs.md` — `tests/e2e/test_cache_busting.py` (Playwright header assertions), README, story file, commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: version token (git path, fallback path, failure path), the asset-reference rewrite (both `href`/`src` and leading-slash-less `assets/…` refs, no double-`?v=`).
|
|
||||||
- Integration: page headers + rewritten references; asset headers; no-CDN test green; SSE endpoint responses untouched (existing chat SSE tests green).
|
|
||||||
- Coverage: **>90%** on `app/` (the new `app/core/caching.py` fully covered); TOTAL ≥ pre-change.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_cache_busting.py` — one story, run **in isolation**; real Chromium asserting the headers and the versioned request URLs the browser actually makes.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Every HTML page (`/`, `/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) is served with `Cache-Control: no-cache` and its asset references carry `?v=<token>` (token non-empty, stable across requests, changes when the frontend content changes).
|
|
||||||
- [ ] `/assets/*` responses carry `Cache-Control: public, max-age=31536000, immutable`.
|
|
||||||
- [ ] `/api/*` (incl. the SSE chat stream) responses are byte-for-byte header-wise unaffected beyond what they already send; no-CDN integration test green.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green in isolation; `test_smoke.py` + one RAG E2E stay green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] `.agent/user_stories/cache-busting.md` exists; README documents the caching behavior + how the token changes on deploy.
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching`); `.agent/phases/todo/33_cache_busting/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **Version token** — `asset_version()`: if the project checkout has a `.git` (the homelab reality), the token is `git rev-parse --short HEAD` (a commit = a deploy, so the token flips on every deploy); otherwise a stable hash of the frontend tree (sorted `relpath + mtime_ns + size`, first 12 hex chars) so dev checkouts still bust. Computed **once per process** (`lru_cache`) — zero per-request git/file cost.
|
|
||||||
- **Rewrite scope** — only the five known HTML pages are rewritten (a small regex over `href="…assets/…"` / `src="…assets/…"` appending `?v=` when absent). No templating layer, no build step, no changes to the static files themselves (the `Containerfile` esbuild stage is untouched).
|
|
||||||
- **Asset caching** — `/assets/*` are cached `immutable` for 1 year **because** the URL carries the token; the unversioned path keeps working (StaticFiles ignores the query string), so old tabs and tests referencing `/assets/x.js` directly still resolve.
|
|
||||||
- **Middleware boundary** — the middleware touches exactly two shapes: the five page paths (body rewrite + `no-cache`) and `/assets/*` (header only). Everything else — all `/api/*` including SSE — passes through byte-identical (SSE keeps its own `no-cache`). Implemented as a Starlette middleware that only rewrites `text/html` responses under the page paths; if `Response.body()` turns out to misbehave on the `FileResponse` streaming path, the fallback is five explicit FastAPI routes that read + rewrite the files (identical observable behavior — the executor picks whichever passes the tests).
|
|
||||||
- **A11 untouched** — no CDN, no new packages, no new services (A12 untouched).
|
|
||||||
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
# Task 01 — app/core/caching.py: the asset version token
|
|
||||||
|
|
||||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "We need better cache busting, the pages are too sticky"`
|
|
||||||
**Story:** `.agent/user_stories/cache-busting.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
One source of truth for the asset version token: git short SHA when the checkout is a repo (a commit = a deploy), a stable content-hash fallback otherwise — computed once per process.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/core/caching.py` (new) —
|
|
||||||
- `def asset_version(static_dir: str | None = None) -> str`:
|
|
||||||
- `static_dir` defaults to `get_settings().static_dir` (resolves to `frontend`); the git repo root is its parent.
|
|
||||||
- **Git path:** if `(static_dir parent / ".git")` exists → `subprocess.run(["git", "-C", str(root), "rev-parse", "--short", "HEAD"], capture_output=True, text=True, timeout=5)` → the short SHA (e.g. `3841bd5`).
|
|
||||||
- **Fallback / failure path** (no `.git`, git missing, non-zero exit, timeout, unreadable): `hashlib.sha256` over the sorted list of `f"{relpath}:{mtime_ns}:{size}"` for every regular file under `static_dir`, first **12 hex chars**. A missing/empty `static_dir` → `"dev"`.
|
|
||||||
- `@functools.lru_cache(maxsize=None)` on the resolved-argument wrapper (settings are process-stable; the token must not be recomputed per request). Document that a process restart or new commit changes the token.
|
|
||||||
2. `tests/unit/test_caching.py` (new):
|
|
||||||
- git path: a `tmp_path` repo (`git init -q` + a commit of a dummy file, with a `frontend/` subdir inside) → token == `git rev-parse --short HEAD` of that repo; second call returns the same value (cache).
|
|
||||||
- fallback: a plain `tmp_path/frontend` with two files → 12-hex token; unchanged tree → same token; touch/modify a file (mtime or size change) + `asset_version.cache_clear()` → different token.
|
|
||||||
- failure: `static_dir` with a `.git` present but `git` removed from PATH (monkeypatch `subprocess.run` to raise `FileNotFoundError`) → falls back to the content hash, no exception.
|
|
||||||
- empty/missing dir → `"dev"`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: Work step 2.
|
|
||||||
- Coverage: **>90%** on `app/core/caching.py` (all branches).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All unit tests green (git, fallback, failure, empty — with `cache_clear()` between parametrized cases).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# Task 02 — Caching middleware + wiring + integration tests
|
|
||||||
|
|
||||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "the pages are too sticky" (HTML revalidation + versioned asset URLs + immutable assets)`
|
|
||||||
**Story:** `.agent/user_stories/cache-busting.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Apply the caching behavior at the transport layer: HTML pages `no-cache` with `?v=<token>` on every local asset reference; `/assets/*` immutable 1-year; everything else (all `/api/*`, including SSE) untouched.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/core/caching.py` (extend task 01's module):
|
|
||||||
- `HTML_PAGES: tuple[str, ...] = ("/", "/index.html", "/sources.html", "/document.html", "/login.html", "/tuning.html")`.
|
|
||||||
- `_ASSET_REF_RE = re.compile(r'((?:href|src)="(?:/)?assets/[^"?#]+)(")')` — matches `<link rel="stylesheet" href="/assets/styles.css">`, `<script src="assets/markdown.js"></script>` (no leading slash!), and `<script type="module" src="/assets/app.js">`; the rewrite appends `?v=<token>` before the closing quote, only when the reference has no query/hash yet (idempotent — never a double `?v=`).
|
|
||||||
- `def rewrite_asset_refs(html: str, token: str) -> str` — the pure, unit-testable rewrite.
|
|
||||||
- `def configure_caching(app: FastAPI) -> None` — one `@app.middleware("http")` (or equivalent Starlette middleware) that, **after** the response is produced:
|
|
||||||
- `path.startswith("/assets/")` → `response.headers["Cache-Control"] = "public, max-age=31536000, immutable"` (header only — never touch the body).
|
|
||||||
- `request.url.path` in `HTML_PAGES` **and** response content-type is `text/html` → `Cache-Control: no-cache` + `response.body = rewrite_asset_refs(body, asset_version())` (body via `await response.body()` — works for the buffered StaticFiles/FileResponse HTML responses; on any error or non-`text/html`, fall through to the unmodified response with only `no-cache`).
|
|
||||||
- everything else → completely untouched (no header, no body work). The `/api/*` SSE stream in particular must not be read or rewritten.
|
|
||||||
2. `app/main.py` — `from app.core.caching import configure_caching`; call `configure_caching(app)` inside `create_app()` after the routers (order: middleware wraps the whole app — call it before `return app`).
|
|
||||||
3. `tests/unit/test_caching.py` (extend) — `rewrite_asset_refs`:
|
|
||||||
- versioned: `href="/assets/styles.css"` → `href="/assets/styles.css?v=abc123"`; `src="assets/markdown.js"` (no slash) → versioned; `src="/assets/app.js"` (module script) → versioned.
|
|
||||||
- idempotent: an already-`?v=`-tagged reference is not double-tagged; a `#fragment` or existing `?query` reference is left alone.
|
|
||||||
- non-asset references untouched (`href="/sources.html"`, `href="data:…"`, `href="/login.html?next=…"`).
|
|
||||||
4. `tests/integration/test_api.py` (extend):
|
|
||||||
- `GET /` → 200, `cache-control: no-cache`; body contains `href="/assets/styles.css?v=<token>"` with a non-empty token matching `asset_version()`; the unversioned string `href="/assets/styles.css">` is **gone** from the body.
|
|
||||||
- each of the other four pages (`/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) → `no-cache` + at least one versioned asset reference.
|
|
||||||
- `GET /assets/styles.css` → 200, `cache-control` contains `immutable` and `max-age=31536000`.
|
|
||||||
- `GET /api/health` → response has **no** `cache-control` injected (baseline: FastAPI's default) — assert equality with the pre-middleware behavior; the SSE chat endpoint (`tests/integration/test_chat_api.py`) stays green unmodified.
|
|
||||||
- the existing no-CDN test (`test_html_pages_served_locally_no_cdn`) stays green — the rewritten references are still same-origin.
|
|
||||||
5. `tests/unit/test_main.py` — app creation still succeeds with the middleware wired (existing creation tests stay green; add an assertion that `create_app()`'s middleware stack includes the caching middleware by name).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: Work steps 3 + 5; Integration: Work step 4.
|
|
||||||
- Coverage: **>90%** on `app/core/caching.py`; `app/` TOTAL ≥ pre-change.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All new unit + integration tests green; the full suite green (especially `test_chat_api.py` — SSE unaffected).
|
|
||||||
- [ ] Manual check: `curl -si localhost:8000/ | grep -i cache-control` → `no-cache`; the HTML body shows `?v=…` asset refs; `curl -si localhost:8000/assets/styles.css | grep -i cache-control` → immutable.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Task 03 — Story E2E (Playwright header assertions) + README + commit
|
|
||||||
|
|
||||||
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — (whole item: better cache busting — the pages are too sticky)"`
|
|
||||||
**Story:** `.agent/user_stories/cache-busting.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The story gate: a real-browser E2E that asserts what the browser actually receives (HTML `no-cache`, versioned asset request URLs, immutable asset headers), plus README docs, story file, and the phase commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_cache_busting.py` (new) — collect responses with `page.on("response")`:
|
|
||||||
- `test_html_pages_are_no_cache_and_versioned` — navigate to `/`:
|
|
||||||
- the document response's `cache-control` header is `no-cache`;
|
|
||||||
- the `styles.css` request URL contains `?v=` and the response's `cache-control` contains `immutable` + `max-age=31536000`;
|
|
||||||
- the `app.js` request URL contains the **same** token value as the CSS one (single token per process);
|
|
||||||
- the served HTML (`page.content()`) contains no unversioned `/assets/styles.css"` reference.
|
|
||||||
- `test_other_pages_share_the_token` — navigate to `/sources.html` then `/login.html`: each document response is `no-cache`; both pages' CSS requests carry the same token.
|
|
||||||
- `test_api_responses_unaffected` — `page.request.get("/api/health")` → no `cache-control: no-cache`/immutable injection (the endpoint's baseline headers only); a chat SSE POST still streams (reuse the minimal chat-request helper from an existing E2E — the stream must complete with `done`).
|
|
||||||
2. `README.md` — new short "Caching / deploys" section: HTML is always revalidated; assets are cached 1 year immutable and carry `?v=<token>`; the token is the git short SHA (falls back to a content hash in non-git checkouts) and flips on every commit/deploy — no hard refresh needed anymore; API/SSE caching is unchanged.
|
|
||||||
3. `.agent/user_stories/cache-busting.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
|
|
||||||
4. Commit: `git commit --no-gpg-sign -m "perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching"`; move `.agent/phases/todo/33_cache_busting/` → `.agent/phases/complete/`.
|
|
||||||
5. **Deploy note (post-commit, owner action)**: after this phase lands, the *first* deploy also requires browsers to see the new HTML once (revalidation) — one normal navigation; thereafter every commit is picked up automatically.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` for the app boot; the mock LLM keeps the SSE check deterministic).
|
|
||||||
- Regression in isolation: `test_smoke.py`, `test_chat_rag.py`.
|
|
||||||
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All E2E tests green in isolation (headers + token consistency + SSE unaffected).
|
|
||||||
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
|
|
||||||
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
|
|
||||||
- [ ] `podman compose up -d` (full app) smoke: a fresh browser profile loads the site and every asset request is versioned (manual confirmation recorded in the commit message or phase notes).
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# Phase 34 — One Navbar on Every Page
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
**Context:** The shared header (phase 19, `frontend/assets/header.js`) + the Tuning nav link (phase 29) already standardize nav + auth on chat / sources / tuning — but `document.html` still uses the separate `.doc-header` variant (back + title + actions, **no nav at all**), `login.html` misses the Tuning link, and two functional controls remain page-scoped: the Tuning steering toggle + panel (chat only, logic in `app.js`) and the Sync sources button (Sources only, logic in `sources.js`). Owner confirmation (2026-08-26): the bar must be identical on **all** pages — nav, Tuning toggle, Sync, New chat, and the auth pair all present everywhere; the locked A10 UI revision stays (admin-only controls hidden for anonymous, active for the admin on every tab).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the header bar **identical on all five pages** (chat, sources, document viewer, tuning, login): one shared markup block, one owner of all functional control behavior (`header.js`), the viewer's back link + title preserved in a second titlebar row, and the phase-12/19 height contract (64px desktop / 58px ≤640px) applied to the standard row on every page.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `19_shared_header` (complete) — the `header.js` module, the nav/auth markup + ids, the cached-one-whoami contract, the ship-hidden/reveal-for-admin pattern.
|
|
||||||
- `29_tuning_nav_link` (complete) — the admin-only `#nav-tuning` reveal pattern this phase completes on the remaining pages.
|
|
||||||
- `15_steering_notes` + `27_global_tuning` (complete) — the steering toggle/panel logic being moved into the shared module; chat-page behavior must not change.
|
|
||||||
- `32_admin_sync_button` (complete) — the sync button state machine + `GET/POST /api/sync` being moved into the shared module; Sources-page behavior (result line + error banner) must not change.
|
|
||||||
- `13_document_back_navigation` (complete) — the `#doc-back` target-resolution behavior the viewer titlebar must preserve.
|
|
||||||
- `16_admin_auth` (complete) — the whoami gate, the soft-gate pages, the sign-out binding.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_steering_moves_to_module.md` — the steering toggle + panel logic moves from `app.js` into `header.js` (exported `refreshSteering()`); the chat per-bubble Tune form keeps working.
|
|
||||||
2. `02_sync_and_chat_moves_to_module.md` — the sync state machine moves from `sources.js` into `header.js` (`bor:sync-status` event); one module-owned New chat binding; the sign-in `?next=` rewrite.
|
|
||||||
3. `03_full_header_all_pages.md` — all five pages ship the identical header block; `#steering-panel` exists on every page; the viewer becomes standard row + titlebar row; login gains the full header.
|
|
||||||
4. `04_viewer_titlebar_styles.md` — the two-row viewer header styles, the sync button's failed state on non-Sources pages, theme/contrast/focus preserved.
|
|
||||||
5. `05_e2e_and_contract_update.md` — the story E2E suite `test_nav_consistency.py`; `test_header_consistency.py` + `test_shared_header.py` updated to the new viewer contract; regression pass; commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: frontend-only — no new `app/` logic; the no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) must still pass (all new markup is same-origin, no new tags).
|
|
||||||
- Coverage: **>90%** on `app/` — unchanged by this phase (no Python change).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_nav_consistency.py` — the story gate, run in isolation; plus the two contract suites updated in task 05 and the regression list below.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The same visible header controls appear on **all five pages** in the same order — brand, nav [Chat, Sources, Tuning], Tuning toggle, Sync sources (admin), New chat, exactly one of Sign in / Sign out — verified in `test_nav_consistency.py` for both the admin and the anonymous role.
|
|
||||||
- [ ] The document viewer shows the standard bar (row 1) + back link and title (row 2); `#doc-back` target resolution (phase 13) unchanged.
|
|
||||||
- [ ] The login page carries the full header (nav incl. Tuning, Tuning toggle, Sync, New chat, auth pair).
|
|
||||||
- [ ] Chat page: the steering panel + per-bubble Tune + inline form behave exactly as before; Sources page: the sync button state machine + `#sync-result` line + `#sync-error-banner` behave exactly as before.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (>90%).
|
|
||||||
- [ ] Regressions green in isolation: `test_header_consistency.py`, `test_shared_header.py`, `test_document_back_navigation.py`, `test_document_viewer.py`, `test_steering.py`, `test_global_tuning.py`, `test_sync_button.py`, `test_tuning_nav_link.py`, `test_smoke.py`, `test_chat_rag.py`, `test_admin_auth.py`.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean (no Python change, but run the gate).
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks / labels / contrast ≥4.5:1 / focus-visible preserved; no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/34_consistent_navbar/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A10 UI revision preserved** — admin-only controls (Sources / Tuning nav links, Sync button) ship hidden and are revealed only for the signed-in admin; anonymous visitors get the reduced bar, identically on every page (owner confirmation 2026-08-26 — "hidden for anon, visible for admin").
|
|
||||||
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
|
|
||||||
- **Phase 19 module contract extended, not replaced** — `header.js` keeps the cached one-whoami-per-page promise; it gains ownership of the controls' behavior, not a second whoami.
|
|
||||||
- **Viewer bar superseded** — the phase-19 single-row viewer bar (PLAN.md §7.1 "the viewer bar = back + title + the same actions") is replaced by the two-row layout at the owner's request (this TODO). `PLAN.md` is not edited (Protocol B); this phase directory records the revision.
|
|
||||||
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Task 01 — Steering toggle + panel move into header.js
|
|
||||||
|
|
||||||
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make `frontend/assets/header.js` the owner of the steering toggle + panel behavior (today in `frontend/assets/app.js`), so the toggle can sit in every page's header (task 03) with zero page-script duplication. The chat page's behavior — panel open/close, list, count badge, per-note delete, per-bubble Tune form — must be byte-for-byte the same from the user's perspective.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/header.js` — add the steering logic (runs at module import, like the existing sign-out binding):
|
|
||||||
- `loadSteering()` — `GET /api/steering`; non-2xx (the anonymous 403, unreachable API) → empty list (the current chat-page anonymous state); render via `renderSteeringPanel(notes)`.
|
|
||||||
- `renderSteeringPanel(notes)` — newest-first `<li class="steering-note">` rows: the note as `textContent` in a `span.steering-note-text` (XSS contract unchanged — never innerHTML for the note), a per-note Remove `button.steering-delete` with `aria-label="Delete tuning note: …"`; toggle `#steering-empty`'s `hidden` on `notes.length`; set the `#steering-count` badge text.
|
|
||||||
- `deleteSteeringNote(id, btn)` — disable the row button, `DELETE /api/steering/{id}`, re-load the list, announce through `#steering-announcer` (`role="status"`).
|
|
||||||
- The `#steering-toggle` click binding — open/close `#steering-panel`, flip `aria-expanded`, move focus into the panel on open (the chat-page a11y contract; read `app.js`'s current implementation first and mirror it exactly, including any close-on-Esc / outside-click behavior it has).
|
|
||||||
- **Export `refreshSteering()`** (fetch + render) — task 01's `app.js` change wires the per-bubble Tune form's success path to it.
|
|
||||||
- Update the file's header comment (it now owns the steering controls).
|
|
||||||
2. `frontend/assets/app.js` — remove the steering **panel** section (the `#steering-toggle` / `#steering-count` / `#steering-panel` / `#steering-list` / `#steering-empty` / `#steering-announcer` refs, `loadSteering`, `renderSteeringPanel`, `deleteSteeringNote`, `announceSteering`, the toggle binding) — **keep** the per-bubble `appendTuneButton` + `openTuneForm` (a chat-specific feature): the inline form's success path calls `refreshSteering()` imported from `./header.js` instead of the removed `loadSteering()`. Keep `TUNE_ICON` and the form's fetch/error handling untouched.
|
|
||||||
3. Update the comments that describe the panel as chat-page-owned (app.js header comment, index.html steering comments) — the panel now belongs to the shared module; index.html's markup stays for now (task 03 copies it to the other pages).
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
- All elements are looked up null-safe (`querySelector` + guard) — a page that (still) lacks the panel markup is a no-op, mirroring how `initSharedHeader()` already works. This keeps the app functional between tasks.
|
|
||||||
- Do not change the steering API (`app/api/steering.py`), the panel markup in `index.html`, or the `#steering-panel` styles.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No Python change; the no-CDN integration test is unaffected.
|
|
||||||
- Coverage: `app/` gate unaffected (no Python change).
|
|
||||||
- The moved logic is behavior-verified by the regression suites in task 05 (`test_steering.py`, `test_global_tuning.py`); until then `uv run pytest` (unit + integration) must stay green.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `header.js` exports `refreshSteering()` and owns the toggle binding, panel render, per-note delete, count badge, and announcer.
|
|
||||||
- [ ] `app.js` no longer contains the panel logic; the per-bubble Tune button + inline form remain and call `refreshSteering()` on save.
|
|
||||||
- [ ] The chat page (`/`) still loads, opens, lists, and deletes steering notes exactly as before (manual smoke via the dev server or the regression suites in task 05).
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Task 02 — Sync state machine + New chat + sign-in next move into header.js
|
|
||||||
|
|
||||||
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make `header.js` the owner of the Sync button state machine (today in `sources.js`), the single New chat binding (today duplicated across `app.js` / `sources.js` / `tuning.js` / `document.js`), and the sign-in `?next=` derivation — so the same markup on any page (task 03) behaves identically.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/header.js` — add the sync state machine (read `sources.js`'s sync section first and mirror its contract exactly):
|
|
||||||
- **Boot (admin only):** `await fetchIsAdmin()` on the cached whoami — non-admins never poll (the status endpoint is admin-only). One `GET /api/sync/status`: `running` → enter the running state + start polling (the phase-32 reload-mid-sync re-attach); terminal → render the last result.
|
|
||||||
- **Click `#sync-btn`:** `POST /api/sync` → 202 enters running; 409 attaches to the running state (one sync at a time).
|
|
||||||
- **Poll** `GET /api/sync/status` every 2000 ms — one live timer, stopped on a terminal state. **No client-side hard timeout** (phase-32 locked decision — a sync can outlive the page; the state machine simply keeps polling).
|
|
||||||
- **Button states (§7.4 never-stale):** idle → label "Sync"; running → `disabled` + `aria-busy="true"` + spinner class (`.sync-icon.is-spinning`) + label "Syncing…"; success → label "Synced HH:MM"; failed → error state with the sanitized error string in the button's `title` + `aria-label` (on non-Sources pages that is where the failure is visible — the Sources page's own banner is driven by the event below).
|
|
||||||
- **On every state change** dispatch `window.dispatchEvent(new CustomEvent("bor:sync-status", { detail: <status> }))` where `detail` is the `GET /api/sync/status` object — step 2 points the Sources page's banner/result line at it.
|
|
||||||
2. `frontend/assets/header.js` — **one** New chat binding (module scope, null-safe): if `#messages` exists (chat page) → `window.dispatchEvent(new CustomEvent("bor:new-chat"))` and let the page script act; otherwise `clearChatStorage()` + `location.href = "/"` (the existing non-chat behavior — "new chat" means go to the chat, fresh).
|
|
||||||
3. `frontend/assets/header.js` — **sign-in `?next=` rewrite:** in `initSharedHeader()` (or the module-scope boot), set `#sign-in-link`'s `href` to `/login.html?next=<current pathname>` (default `/`) — the admin lands back on the page they signed in from.
|
|
||||||
- ASSUMPTION: on the chat page this changes the static fallback `?next=/sources.html` to `/` at runtime — landing on the page you signed in from ("return to where you were"). The page markup keeps its current href as the no-JS fallback.
|
|
||||||
4. `frontend/assets/sources.js` — remove the sync state machine (the `#sync-btn` click handler, the 2 s poll loop, the button-state helpers, the boot re-attach). **Keep** `#sync-result` + `#sync-error-banner` rendering, now driven by a `window.addEventListener("bor:sync-status", …)` subscription: `running` → clear the result line, hide the banner; `success` → render the last-result counts in `#sync-result` (reuse the existing formatting, "added" always shown); `failed` → show `#sync-error-banner` with the error text; `idle` → hide the banner, clear the result.
|
|
||||||
5. `frontend/assets/app.js` — replace the direct `#new-chat-btn` click binding with `window.addEventListener("bor:new-chat", startNewChat)` (the `startNewChat` function itself is unchanged).
|
|
||||||
6. `frontend/assets/sources.js`, `frontend/assets/tuning.js`, `frontend/assets/document.js` — remove their `#new-chat-btn` click bindings (the module owns them). Update the file-header comments (document.js: the module now owns New chat; sources.js: sync is module-owned, the banner is event-driven).
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
- Null-safe element lookups throughout (a page that doesn't (yet) have `#sync-btn` is a no-op — the app stays functional between tasks).
|
|
||||||
- The module must keep exactly **one** whoami per page load (the cached promise) — the sync boot may await it but must not add a fetch.
|
|
||||||
- Do not touch `app/api/sync.py` — the API contract is unchanged.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No Python change; the no-CDN integration test is unaffected.
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Behavior parity is verified by the regression suites in task 05 (`test_sync_button.py`, `test_shared_header.py`, `test_chat_rag.py`); until then `uv run pytest` must stay green.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `header.js` owns: the sync state machine (+ `bor:sync-status` event), the single New chat binding (`bor:new-chat` on chat, clear+navigate elsewhere), and the sign-in `next` rewrite.
|
|
||||||
- [ ] `sources.js` no longer contains the sync state machine — `#sync-result` / `#sync-error-banner` render off the event; no `#new-chat-btn` binding remains in any page script.
|
|
||||||
- [ ] On the Sources page the full phase-32 cycle (click → polling → success counts / failure banner, reload re-attach) still works — confirmed in task 05 via `test_sync_button.py`.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Task 03 — The identical full header on all five pages
|
|
||||||
|
|
||||||
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Ship the **identical header block** on all five pages — brand, nav [Chat, Sources, Tuning], Tuning toggle, Sync sources, New chat, Sign in / Sign out — and the `#steering-panel` section on every page; the document viewer keeps back + title in a second titlebar row; the login page finally carries the full header.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
The canonical block is `index.html`'s current header **plus** the `#sync-btn` copied verbatim from `sources.html` (hidden by default, `#sync-label` + `.sync-icon` inside). Place the Sync button **after** the Tuning toggle and **before** the New chat button on every page.
|
|
||||||
|
|
||||||
1. `frontend/index.html` (chat) — add the `#sync-btn` block to the header (the only missing control); everything else already ships. `#steering-panel` stays where it is (after `#kb-banner` in `<main>`).
|
|
||||||
2. `frontend/sources.html` — add the Tuning toggle block (copied from `index.html`: `#steering-toggle` + `#steering-count`) after the nav; add the `#steering-panel` section (copied from `index.html`, incl. the `#steering-announcer` paragraph) as the **first child of `<main>`**; keep the existing `#sync-btn` where it is.
|
|
||||||
3. `frontend/tuning.html` — add the Tuning toggle block + the `#sync-btn` block to the header (same order as chat); add the `#steering-panel` section as the first child of `<main>`.
|
|
||||||
4. `frontend/document.html` — restructure the header:
|
|
||||||
- **Row 1** becomes the standard `.app-header` / `.header-inner` bar, byte-for-byte the same block as the other pages: brand, `<nav class="app-nav">` with `Chat` + `#nav-sources` (hidden) + `#nav-tuning` (hidden), Tuning toggle, `#sync-btn` (hidden), New chat, Sign in (`?next=/document.html` static fallback) / Sign out.
|
|
||||||
- **Row 2** — a new `.doc-titlebar` container inside the same `<header>`, carrying the existing `#doc-back` link + `#doc-title` + `#doc-meta` (moved out of the old `.doc-header-inner` title block, markup otherwise unchanged — `renderDocument` addresses them by id, so `document.js` needs no render change).
|
|
||||||
- ASSUMPTION: **no nav link gets `is-active` / `aria-current` on the viewer** — a document is a detail view reachable from chat or Sources (phase 13's `back` param), so no single nav target is "current". The back link carries the navigation affordance.
|
|
||||||
- The old `.doc-header-actions` wrapper is dropped — its buttons now live in row 1's standard `.header-inner`.
|
|
||||||
5. `frontend/login.html` — full header: the nav gains the `#nav-tuning` link (after `#nav-sources`, same hidden-by-default markup as the other pages); add the Tuning toggle + `#sync-btn` + New chat + the Sign in / Sign out pair (sign-in static fallback `?next=/login.html`); add the `#steering-panel` section as the first child of `<main>`.
|
|
||||||
6. `frontend/assets/header.js` — comment updates only: the viewer now has a nav (its "the viewer has no nav" notes are stale); the module's "missing element is a no-op" contract still holds for any page missing an element. No behavior change — the reveal code already handles `#nav-sources` / `#nav-tuning` / `#sync-btn` / the auth pair wherever they exist.
|
|
||||||
7. Update the stale HTML comments in the touched headers (phase-19/29 comments describing the old page-specific layouts) to reference this phase + the owner confirmation (2026-08-26).
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
- Keep every existing id / class / aria attribute exactly as it exists today (the E2E suites key off them); only ADD missing blocks and move the viewer's title elements.
|
|
||||||
- Ship-hidden stays ship-hidden: `#nav-sources`, `#nav-tuning`, `#sync-btn`, and exactly one of the auth pair are `hidden` in the markup on every page — `header.js` reveals at load (one whoami, cached).
|
|
||||||
- Preserve indentation/markup style so the five headers stay diff-identical (that identity is what task 05's E2E asserts).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No Python change; the no-CDN integration test must still pass (same-origin markup only).
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Manual smoke before task 05: with the dev server, as admin and as anonymous, each of the five pages shows the full bar (admin) / reduced bar (anonymous) with no console errors.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All five page headers contain the same control inventory in the same order: brand, nav [Chat, Sources, Tuning], `#steering-toggle`, `#sync-btn`, `#new-chat-btn`, `#sign-in-link` + `#sign-out-btn`.
|
|
||||||
- [ ] `#steering-panel` (+ `#steering-announcer`) exists on all five pages (chat: after `#kb-banner`; others: first child of `<main>`).
|
|
||||||
- [ ] `document.html` = standard row + `.doc-titlebar` row with `#doc-back` / `#doc-title` / `#doc-meta`; no nav link carries `is-active` there.
|
|
||||||
- [ ] `login.html` carries the full header incl. the `#nav-tuning` link and the auth pair.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Task 04 — Two-row viewer header styles + sync failed state
|
|
||||||
|
|
||||||
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Style the document viewer's new two-row header (standard row + titlebar row) so row 1 is visually indistinguishable from the other pages' bars, and give the Sync button a visible failed state on pages that have no error banner (every page except Sources).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/styles.css` — **viewer header:**
|
|
||||||
- The viewer's `<header>` keeps the sticky app-frame behavior; **row 1** reuses the existing `.app-header` / `.header-inner` rules verbatim (64px desktop, 58px at ≤640px — the phase-12/19 pinned heights apply because row 1 *is* the standard bar).
|
|
||||||
- New `.doc-titlebar` rules for row 2: `.container`-width inner row with `#doc-back` + the title block; its own height (title line + meta line), `border-top` separator in the existing hairline color, same surface color (`#121a2e`/`#0a0e17` family per the phase-08 palette); `#doc-title` truncates with an ellipsis + `title` attribute instead of the old pill-clipping rule.
|
|
||||||
- The old `.doc-header` / `.doc-header-inner` / `.doc-header-actions` / title-clipping rules are removed or reduced to the two-row structure (keep class names that task 05's updated contract suites reference — check which selectors the suites use before deleting: `.doc-header` may remain as the header element's class wrapping both rows).
|
|
||||||
- The `.steering-panel` positioning rules must work from the new placement on the non-chat pages (first child of `<main>`) — the panel is an in-flow section, so this is expected to be a no-op; verify visually and in the E2E.
|
|
||||||
2. `frontend/assets/styles.css` — **sync failed state (non-Sources pages):** the failed `#sync-btn` gets an error treatment from the phase-08 palette (error ink `#fca5a5` on the error surface `#2d1318`, border `#f59e0b`-free — the error chip uses `#fca5a5`/`#2d1318`, ≈9.1:1) so a failed sync is visible on every page, complementing the `title`/`aria-label` error text set by `header.js`. Reuse the existing `.sync-btn` state classes/styles if phase 32 already defines a failed look; otherwise add it.
|
|
||||||
3. `frontend/assets/styles.css` — **login page:** the full header needs no new rules (it reuses `.app-header`), but confirm the login card layout still centers correctly with the full bar (no header-height regression at ≤640px).
|
|
||||||
4. Accessibility checks (AGENTS.md rule 5): `:focus-visible` 3px outline on the new titlebar back link (it already has the existing `.doc-back` styles — preserve); contrast ≥4.5:1 for title/meta text; `prefers-reduced-motion` still stills the sync spinner.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- CSS-only — no Python change; the no-CDN integration test is unaffected.
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Visual pass (dev server, admin + anonymous, desktop + 640px): all five pages, viewer row 1 identical to chat's bar; viewer row 2 shows back + title + meta; the sync button's failed state is visible and readable.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The viewer's row 1 renders pixel-consistent with the chat/sources bars (same height 64px / 58px, same paddings, same controls).
|
|
||||||
- [ ] `.doc-titlebar` renders back + title (ellipsis) + meta badges on one or two tidy lines; sticky with the header.
|
|
||||||
- [ ] A failed sync shows an error-colored `#sync-btn` on non-Sources pages with the sanitized error in `title` / `aria-label`.
|
|
||||||
- [ ] No console layout breakage on the login page (card still centered, 58px bar at ≤640px).
|
|
||||||
- [ ] `uv run pytest` green (no-CDN test included); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Task 05 — Story E2E + contract-suite updates + regression pass
|
|
||||||
|
|
||||||
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
|
|
||||||
**Story:** `.agent/user_stories/nav-consistency.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the contract with the story's dedicated Playwright suite — identical visible header control inventory on all five pages for each role — update the two pre-existing contract suites that encoded the old viewer bar, and run the full regression list.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_nav_consistency.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (see `tests/e2e/conftest.py`); a fixture document for the viewer URL (the `source=docs&path=homelab%2Fkubernetes.md` pattern from `test_shared_header.py`); admin session via `tests/e2e/auth_helpers.py`.
|
|
||||||
- **Admin inventory (all five pages):** `/`, `/sources.html`, `/document.html?source=…&path=…`, `/tuning.html`, `/login.html` — each header contains, visible: the three nav links (Chat, `#nav-sources`, `#nav-tuning`), `#steering-toggle`, `#sync-btn`, `#new-chat-btn`, and `#sign-out-btn` visible with `#sign-in-link` hidden. Assert the same **id + class inventory and DOM order** of the header controls on every page (normalize: the current-page `is-active` nav marker and the sign-in `?next=` value legitimately differ per page).
|
|
||||||
- **Anonymous inventory (all five pages):** nav present with Chat visible and `#nav-sources` / `#nav-tuning` hidden (locked A10 UI revision); `#sync-btn` hidden; `#sign-in-link` visible, `#sign-out-btn` hidden; `#steering-toggle` visible.
|
|
||||||
- **Viewer specifics:** row 1 height equals the chat page's header height (64px desktop / 58px ≤640px); the titlebar row is visible with `#doc-back` + `#doc-title` (rendered document title) + `#doc-meta` badges; clicking `#doc-back` honors the `back` param (phase 13 — one positive + one rejection case).
|
|
||||||
- **Steering works off-chat:** as admin, on `/tuning.html` — seed zero notes (truncate `steering_notes` via a `SessionLocal` like the other suites), click `#steering-toggle` → `#steering-panel` visible + `aria-expanded="true"` + empty state shown; add a note through the panel? (the panel has no add form — it lists notes; assert the toggle open/close cycle + the count badge reads 0) — keep this deterministic, no chat needed.
|
|
||||||
- **Sync present, not triggered:** as admin on `/tuning.html` assert `#sync-btn` is visible (do NOT click it — a real sync would clone real repos; the full state machine is `test_sync_button.py`'s job).
|
|
||||||
2. `tests/e2e/test_header_consistency.py` (UPDATE to the new contract): the viewer assertions change — `.doc-header` is now the two-row header; assert **row 1** (the standard bar) is 64px desktop / 58px ≤640px and identical to the chat/sources bars (the existing `_box_height(page, ".doc-header")` measurement must be pointed at the standard row — use the row-1 selector, e.g. `.app-header .header-inner` inside the viewer header), and assert the titlebar row is present (height > 0). The chat/sources assertions are unchanged.
|
|
||||||
3. `tests/e2e/test_shared_header.py` (UPDATE to the new contract): the "the viewer has no nav — no Sources link in the DOM" assertions flip — the viewer now carries the same nav contract (`.app-nav` with Chat + hidden `#nav-sources` + hidden `#nav-tuning`, revealed for admin). The auth-pair + New chat assertions for the viewer stay (they move from `.doc-header-actions` to the standard bar — update the selectors).
|
|
||||||
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_header_consistency.py`, `test_shared_header.py`, `test_document_back_navigation.py`, `test_document_viewer.py`, `test_steering.py`, `test_global_tuning.py`, `test_sync_button.py`, `test_tuning_nav_link.py`, `test_smoke.py`, `test_chat_rag.py`, `test_admin_auth.py`. Fix fallout in the suites above where the old contract is encoded; fix app code where behavior genuinely changed.
|
|
||||||
5. Full gate: `uv run pytest` (unit + integration), `uv run pytest --cov=app --cov-report=term-missing` (TOTAL unchanged, >90%), `uv run ruff check . && uv run pyright`.
|
|
||||||
6. **UI Structure Check** (AGENTS.md rule 5) on the five headers + the new titlebar: landmarks (`<header>`, `<nav aria-label>`, `<main>`), labels, contrast ≥4.5:1, focus-visible, no CDN (rule 6 — the no-CDN integration test covers it).
|
|
||||||
7. **Commit** (A17): stage only this phase's files (`frontend/**`, `tests/e2e/**`), message `feat(ui): one consistent navbar on every page (TODO.md L3)`, always `--no-gpg-sign`. Move `.agent/phases/todo/34_consistent_navbar/` to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `tests/e2e/test_nav_consistency.py` green **in isolation** — the story gate (A16: one story, one file).
|
|
||||||
- Unit/integration: no new `app/` logic — the existing suite (incl. the no-CDN test) stays green.
|
|
||||||
- Coverage: **>90%** on `app/` — unchanged (no Python change).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_nav_consistency.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `test_header_consistency.py` + `test_shared_header.py` updated and green; every suite in the task 05 regression list green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (>90%).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# Phase 35 — Admin Page to Add / Remove Git Sources
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L4 — "I need a page only the admin can access where I can add and remove git sources for docs"
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
**Context:** Phase 28 introduced git-based sources (`BOR_GIT_SOURCES` env var + `scripts/git_sync.clone_or_pull`) and phase 32 the one-click Sync button (`POST /api/sync`) — but the *list itself* can only be changed by editing `.env` and restarting. This phase makes the list admin-managed: a Postgres-backed table, an admin-only CRUD API, and a dedicated admin page, with the sync pipeline and `import_docs` resolving the effective list from the DB (env var demoted to an empty-table fallback).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Deliver a page **only the admin can access** (`/git-sources.html`, soft-gated like Sources) to **add and remove git sources**, stored in a new `git_sources` table; the Sync button (phase 32) and `import_docs` (phase 28) use the stored list, `BOR_GIT_SOURCES` remains a fallback while the table is empty, and phase 32's fail-loud "no git sources configured" is preserved when both are empty.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `28_git_based_sources` (complete) — `scripts/git_sync.clone_or_pull`, `repo_name`, the `BOR_GIT_SOURCES` settings + `git_source_list`, the import resolution order (`--source` wins).
|
|
||||||
- `32_admin_sync_button` (complete) — the `POST /api/sync` / `GET /api/sync/status` pipeline this phase re-points at the DB list; the Sync button the page's hint refers to.
|
|
||||||
- `16_admin_auth` (complete) — `require_admin` (the router-level pattern from `app/api/sync.py`), the soft-gate page pattern (`sources.html`), the `fetchIsAdmin()` frontend gate.
|
|
||||||
- `34_consistent_navbar` (todo) — the identical five-page header this phase's admin-only "Git sources" nav link plugs into (phase 29 pattern).
|
|
||||||
- `29_tuning_nav_link` (complete) — the admin-only ship-hidden nav-link pattern to copy.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_model_and_migration.md` — `GitSource` model + migration `0006_git_sources.py` (reversible).
|
|
||||||
2. `02_git_sources_api.md` — admin-only `GET/POST /api/git-sources` + `DELETE /api/git-sources/{id}` with validation, the env-fallback listing, and the integration suite.
|
|
||||||
3. `03_sync_and_importer_use_db.md` — `effective_git_sources()` shared by `app/api/sync.py` and `scripts/import_docs.py` (DB wins, env fallback, fail-loud unchanged) + test updates.
|
|
||||||
4. `04_admin_page.md` — `/git-sources.html` + `git-sources.js` (soft-gated, list / add / remove, env note, sync hint) + styles.
|
|
||||||
5. `05_nav_link.md` — the admin-only "Git sources" nav link on all five pages + the `header.js` reveal.
|
|
||||||
6. `06_e2e_and_docs.md` — the story E2E suite `test_git_sources_admin.py`, `test_nav_consistency.py` nav-inventory update, README/`.env.example` notes, regressions, commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: `tests/unit/` for `effective_git_sources` (DB-wins / env-fallback / both-empty); `tests/integration/test_git_sources_api.py` for the CRUD contract (403/201/409/422/404, env fallback); the migration up/down test following the 0004/0005 pattern; the existing `test_sync_api.py` + `test_import_docs_git.py` suites stay green with the resolution indirection.
|
|
||||||
- Coverage: **>90%** on `app/` for the new module + API.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_git_sources_admin.py` — the story gate, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Migration 0006 applied (`uv run alembic upgrade head`); `git_sources` table exists with `url` unique.
|
|
||||||
- [ ] `GET /api/git-sources` (admin) lists DB rows; while the table is empty it returns the env list with `from_env: true`; anonymous gets 403 on all three routes.
|
|
||||||
- [ ] `POST` creates (201, trimmed, shape-validated, 409 duplicate without echoing the URL); `DELETE` removes (204/404).
|
|
||||||
- [ ] `POST /api/sync` and `import_docs` resolve the list via `effective_git_sources` (origin logged `db|env`); both-empty still fails loudly; `--source` override unchanged.
|
|
||||||
- [ ] `/git-sources.html`: anonymous sees the sign-in gate; the admin sees list + add + remove with a never-stale button and inline errors; the admin-only "Git sources" nav link is visible on all five pages for the admin and hidden for anonymous.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation; regressions (task 06 list) green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A3 / A13 honoured** — the list lives in Postgres via Alembic (no JSON file, no new store).
|
|
||||||
- **A10 extended per the phase-16 pattern** — a new admin-only router behind `require_admin`; the public API surface stays stateless; no new auth mechanism.
|
|
||||||
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
|
|
||||||
- **Env var demoted, not removed** — `BOR_GIT_SOURCES` keeps working exactly as today while the table is empty (the fallback); once the table has rows it is ignored (the UI is the source of truth). Phase 32's fail-loud empty-config behavior is preserved.
|
|
||||||
- **Scope boundary** — adding/removing a repo does NOT immediately clone, import, or prune: the existing Sync button performs that (removal prunes on the next sync, `prune=True`). The page's hint says so.
|
|
||||||
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# Task 01 — GitSource model + migration 0006
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the `git_sources` table (one row per admin-managed repo URL) via the model + a reversible Alembic migration, following the exact conventions of migrations 0003–0005.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/models.py` — add the model (next to `SteeringNote`, docstring citing this phase + the A13 convention):
|
|
||||||
```python
|
|
||||||
class GitSource(Base):
|
|
||||||
"""One admin-managed git source (phase 35).
|
|
||||||
|
|
||||||
The UI-maintained list of repo URLs the Sync button (phase 32) and
|
|
||||||
import_docs (phase 28) clone/pull. DB rows win over the
|
|
||||||
BOR_GIT_SOURCES env var, which is a fallback while this table is
|
|
||||||
empty (see app.rag.git_sources.effective_git_sources).
|
|
||||||
"""
|
|
||||||
__tablename__ = "git_sources"
|
|
||||||
|
|
||||||
id: Mapped[uuid.UUID] = mapped_column(
|
|
||||||
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
|
|
||||||
)
|
|
||||||
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
|
|
||||||
added_at: Mapped[datetime] = mapped_column(
|
|
||||||
DateTime(timezone=True), server_default=func.now()
|
|
||||||
)
|
|
||||||
```
|
|
||||||
(Import `Text` — already imported in the file; verify.)
|
|
||||||
2. `alembic/versions/0006_git_sources.py` — new migration:
|
|
||||||
- Read `alembic/versions/0005_kb_overview.py` first and chain from its actual `revision` id (the filenames are not the revision ids).
|
|
||||||
- `upgrade()`: `CREATE TABLE git_sources (id UUID PRIMARY KEY, url TEXT NOT NULL, added_at TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL)` + `CREATE UNIQUE INDEX uq_git_sources_url ON git_sources (url)` (use `sa.Uuid` / the same column types the other migrations use — mirror their style, including `op.create_table` kwargs and the `UniqueConstraint`-vs-index choice 0003/0004 made).
|
|
||||||
- `downgrade()`: drop the index + table.
|
|
||||||
3. Apply it to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`.
|
|
||||||
4. Migration test — follow the existing pattern (see how 0004/0005 are integration-tested — `tests/integration/` migration suite): assert 0006 up creates the table + unique constraint and down drops it (round-trip on the test DB).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: the migration up/down test above; `uv run pytest` green overall.
|
|
||||||
- Coverage: model-only for now — the `app/` gate stays >90% (models are thin).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `alembic/versions/0006_git_sources.py` exists, chains off 0005's real revision id, and is reversible.
|
|
||||||
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB; `git_sources` visible (`\d git_sources` equivalent).
|
|
||||||
- [ ] The 0006 up/down integration test passes; full `uv run pytest` green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Task 02 — Admin-only git sources CRUD API
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The admin CRUD contract for the stored list: `GET /api/git-sources` (DB rows, or the env fallback while the table is empty), `POST /api/git-sources` (validated create), `DELETE /api/git-sources/{id}` — all behind `require_admin`, exactly like `app/api/sync.py`.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/schemas.py` — add (mirroring the steering schemas' style):
|
|
||||||
- `GitSourceIn` — `url: str = Field(min_length=1, max_length=500)` + a `mode="before"` trim validator (whitespace-only → 422, same trick as `SteeringNoteIn`).
|
|
||||||
- `GitSourceOut` — `id: uuid.UUID | None`, `url: str`, `added_at: datetime | None` (both nullable: env-fallback rows carry neither).
|
|
||||||
- `GitSourceList` — `sources: list[GitSourceOut]`, `from_env: bool` (`True` only when the table is empty and the list comes from `BOR_GIT_SOURCES`).
|
|
||||||
2. `app/api/git_sources.py` (NEW) — `router = APIRouter(prefix="/git-sources", tags=["git-sources"], dependencies=[Depends(require_admin)])` (copy the sync.py pattern + docstring style):
|
|
||||||
- `GET ""` → `GitSourceList`: DB rows ordered by `(added_at, id)`; if the table is empty → the `get_settings().git_source_list` env URLs as rows with `id=None, added_at=None` and `from_env=True`; `from_env=False` whenever DB rows exist (the env var is then ignored — the phase's locked decision).
|
|
||||||
- `POST ""` (201) → create:
|
|
||||||
- Shape validation (module-level `URL_RE = re.compile(r"^(https?://|ssh://|git@)")` with a docstring): the trimmed URL must match — covers the phase-28 real URLs (HTTPS + `git@` SSH); scp-style `host:repo` is deliberately rejected (422).
|
|
||||||
- ASSUMPTION: the accepted shapes are exactly `http://`, `https://`, `ssh://`, `git@…`; the 422 detail is generic ("not a valid git URL (expected https://, ssh:// or git@…)") and never echoes the input.
|
|
||||||
- Duplicate (same trimmed URL already stored) → 409 with a generic detail ("a git source with this URL already exists") — **never echo the URL** (URLs may embed `user:pass@` credentials; phase 32's masking discipline).
|
|
||||||
- Success → insert, commit, return the created `GitSourceOut`.
|
|
||||||
- `DELETE "/{source_id}"` → 204; unknown id → 404 `git source not found`.
|
|
||||||
3. `app/main.py` — register the router alongside the existing `include_router` calls (check how `sync` is included and mirror it).
|
|
||||||
4. `tests/integration/test_git_sources_api.py` (NEW) — follow `tests/integration/test_sync_api.py`'s auth pattern (`_login` via `POST /api/login` with the fixture admin password, admin client as context manager):
|
|
||||||
- Anonymous → 403 `{"detail": "admin only"}` on GET, POST, and DELETE.
|
|
||||||
- Admin + empty table + env set (monkeypatch the settings `git_sources`) → GET returns the env rows, `from_env=True`, null ids.
|
|
||||||
- Admin + empty table + env empty → GET returns `sources=[]`, `from_env=True`.
|
|
||||||
- POST: valid `https://…` → 201 + the row appears in GET with `from_env` now `False`; a duplicate → 409 and the detail contains no URL; an invalid shape (`not a url`, `host:repo`) → 422; whitespace-only / >500 chars → 422; a `git@github.com:…` URL → 201 (accepted).
|
|
||||||
- DB rows win over env: seed a row AND set the env → GET returns only the DB rows, `from_env=False`.
|
|
||||||
- DELETE: known id → 204 + gone from GET (back to env fallback if the table is now empty); unknown id → 404.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the suite above — the full CRUD contract, auth split, fallback semantics.
|
|
||||||
- Coverage: **>90%** on the new module (`app/api/git_sources.py` + schemas).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All three routes exist under `/api/git-sources`, admin-only (403 anonymous), registered in `app/main.py`.
|
|
||||||
- [ ] `tests/integration/test_git_sources_api.py` green; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
|
|
||||||
- [ ] No credential-echo path: 409/422 details never contain the submitted URL (a test asserts this).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# Task 03 — Sync + import_docs resolve the effective list (DB wins, env fallback)
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
One shared resolver — DB rows win, `BOR_GIT_SOURCES` is a fallback only while the table is empty, fail-loud unchanged when both are empty — used by **both** the in-app sync pipeline (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py`), so the admin page's list is what actually gets cloned and indexed.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/git_sources.py` (NEW) — the single resolver (importable by both the app and the CLI — `scripts` already imports `app.rag.*`):
|
|
||||||
```python
|
|
||||||
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
|
|
||||||
"""(urls, origin) — DB rows in (added_at, id) order win; while the
|
|
||||||
table is empty the BOR_GIT_SOURCES env list is the fallback; both
|
|
||||||
empty → ([], "env")."""
|
|
||||||
```
|
|
||||||
- DB rows: `select(GitSource).order_by(GitSource.added_at, GitSource.id)`.
|
|
||||||
- Fallback: `get_settings().git_source_list` (the phase-28 CSV parse — reuse, don't re-implement).
|
|
||||||
2. `app/api/sync.py` — `_run_sync` replaces `settings.git_source_list` with the resolver:
|
|
||||||
- Open a `SessionLocal()` (close in `finally`) around the resolution — the background task has no request session.
|
|
||||||
- Log the origin: the existing `sync: started repos=N` line gains `origin=db|env`.
|
|
||||||
- Both-empty: keep the fail-loud `GitSyncError("no git sources configured …")` (extend the message to mention both origins — e.g. `(git_sources table empty and BOR_GIT_SOURCES unset)`; if `tests/integration/test_sync_api.py::test_no_git_sources_configured_fails_loudly` asserts the old text, update that expectation — it is this phase's file to update).
|
|
||||||
3. `scripts/import_docs.py` — the git-URL resolution branch (today `settings.git_source_list`) resolves via the same function: open a short `SessionLocal()` at resolution time (the import needs the DB anyway — no DB-down fallback to design). The `--source` override still wins (manual mode), the log line records the origin (`git sources: N repo(s) origin=db|env` before the clone loop).
|
|
||||||
4. **Test updates:**
|
|
||||||
- `tests/unit/` — NEW unit tests for `effective_git_sources`: DB rows win (seeded table + env set → DB list, origin `db`); env fallback (empty table + env set); both empty → `([], "env")`. (Use a test DB session or a stubbed session following the existing unit-test conventions.)
|
|
||||||
- `tests/integration/test_sync_api.py` — where it stubs `settings.git_source_list` to drive sync scenarios, keep those scenarios working through the new indirection: either seed the `git_sources` table or monkeypatch `effective_git_sources` (whichever the file's existing fixture style favors); add one scenario asserting a DB row is used over the env when both are set (clone/import mocked — the file already mocks them).
|
|
||||||
- `tests/integration/test_import_docs_git.py` — same treatment for the CLI path (`--source` override scenario untouched); add the DB-over-env scenario at the CLI level.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `effective_git_sources` — all three branches.
|
|
||||||
- Integration: sync + import_docs suites green with the resolver in the path, incl. the new DB-over-env scenarios and the updated fail-loud expectation.
|
|
||||||
- Coverage: **>90%** on the new module + the modified call sites.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `effective_git_sources` exists in `app/rag/git_sources.py` and is the ONLY place (besides the API's GET fallback, which may call it too) that combines DB + env.
|
|
||||||
- [ ] `_run_sync` and `import_docs` both resolve through it; origin visible in their logs.
|
|
||||||
- [ ] Both-empty still raises the fail-loud error (sync) / the CLI's existing no-sources behavior (import_docs) — assertions kept/updated.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
# Task 04 — The /git-sources.html admin page
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The page itself: a soft-gated (anonymous → sign-in gate, exactly like `sources.html`) full-width manager with an add form, a sources list with per-row Remove, an env-fallback note, and a hint pointing at the Sync button. Never-stale buttons, inline errors, WCAG 2.1 AA basics, dark tech theme.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/git-sources.html` (NEW) — modeled on `frontend/sources.html` (same frame, same gate pattern, the standard full header from phase 34):
|
|
||||||
- Head: same meta/favicon/stylesheet pattern; `<title>Git sources · Brain of Reese</title>`; skip-link; the identical header block (nav [Chat, Sources, Tuning] + Tuning toggle + `#sync-btn` + New chat + auth pair — the "Git sources" nav link itself arrives in task 05, so this file lands without it for now).
|
|
||||||
- `<main id="main">` — `#steering-panel` first (phase 34 contract), then:
|
|
||||||
- **Gate** `#git-sources-gate` — the `#sources-gate` soft-gate markup pattern (sign-in card + link `/login.html?next=/git-sources.html`), visible for anonymous, hidden for admin.
|
|
||||||
- **Content** `#git-sources-content` (hidden until admin):
|
|
||||||
- `.page-head` — `<h1>Git sources</h1>` + sub: "The repositories the Sync button clones and indexes. Add or remove them here — no `.env`, no restart."
|
|
||||||
- **Env note** `#git-sources-env-note` (hidden by default; shown when the API returns `from_env: true`): "These sources currently come from `BOR_GIT_SOURCES` in `.env` — adding or removing one here switches management to the database."
|
|
||||||
- **Add form** `#git-source-form` — visible label (or visually-hidden label per the tuning-page pattern — use a visible `<label for="git-source-url">Add a git source</label>`), input `#git-source-url` (type text, `maxlength="500"`, `autocomplete="off"`, placeholder `https://github.com/you/homelab.git`, `required`), submit button `#git-source-add` ("Add source"), error line `#git-source-error` (`role="alert"`, hidden) — §7.4 never-stale: the button disables + label changes while the POST is in flight, re-enables on success/failure (the form is kept on failure, same as the tuning forms).
|
|
||||||
- **List** `#git-sources-list` (a full-width table or list per §7.1 — **no skinny single-column list**: use the Sources-page table pattern — columns: URL (mono `<code>`), Added, actions) + empty state `#git-sources-empty` ("No git sources stored yet." — and, with `from_env`, the env note already explains where the active list comes from).
|
|
||||||
- **Hint box** (`role="note"`): "Use the **Sync sources** button in the header (or on the Sources page) to clone the repos and refresh the index — removing a repository prunes its documents from the index on the next sync."
|
|
||||||
2. `frontend/assets/git-sources.js` (NEW) — the page module (loaded `type="module"`, imports `./header.js` like its siblings):
|
|
||||||
- Boot: `const admin = await initSharedHeader()` (one cached whoami) — anonymous → show the gate, stop; admin → hide the gate, `loadSources()`.
|
|
||||||
- `loadSources()` — `GET /api/git-sources` → render the table rows (`textContent` only — URLs may contain credentials; never innerHTML the URL), the added date (localized, `—` for null), the per-row Remove button (`.git-source-remove`, `aria-label="Remove git source: <url>"`), the env note's `hidden` on `from_env`, the empty state. Non-2xx → the content area shows a `role="alert"` error state with a retry (never a stuck page).
|
|
||||||
- Add submit — client-side non-empty check; disable `#git-source-add` (label "Adding…"); `POST /api/git-sources` with `{url}`; success → clear the input, re-enable (label "Add source"), `loadSources()`, focus the new row (a11y); failure → `#git-source-error` with the server detail (422 shape-aware like the tuning forms), re-enable, input kept.
|
|
||||||
- Remove click — `window.confirm("Remove this git source from the list? Its documents stay indexed until the next sync prunes them.")` — cancel → nothing; ok → disable the row button, `DELETE /api/git-sources/{id}`, `loadSources()`; failure → row error state + re-enable.
|
|
||||||
- Focus management + keyboard: all controls focus-visible (theme CSS covers it), the list rows' buttons are real `<button>`s.
|
|
||||||
3. `frontend/assets/styles.css` — the `.git-source-*` rules + gate reuse:
|
|
||||||
- The table: full-width in the 72rem container (the Sources-page table styles are a good starting point — reuse classes where they fit), mono URL cells with horizontal scroll on overflow (long URLs with credentials), rows ≥44px touch targets, `:focus-visible` 3px outline.
|
|
||||||
- Env note: an info chip in the theme palette (brand-soft `#232b52` surface, brand-ink `#a5b4fc` text ≈6.9:1); hint box: the page-sub styling family; error/alert states reuse the existing `#fca5a5`/`#2d1318` error treatment.
|
|
||||||
- Gate: reuse the `#sources-gate` styles (the page is the same shape as Sources — one gate visual language).
|
|
||||||
4. `frontend/assets/header.js` — no change (the page's controls are the standard shared ones; `initSharedHeader` already handles everything that ships in the header).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Frontend-only — no Python change this task; the no-CDN integration test must still pass (same-origin markup; the E2E in task 06 exercises the page).
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Manual smoke (dev server, signed in): add a real-looking URL → row appears; remove → confirm → row gone; invalid URL → inline 422 error, button re-enabled; signed out → gate only.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `/git-sources.html` served at the route; anonymous sees only the gate (list/form absent or inert); admin sees list + form + env note + hint.
|
|
||||||
- [ ] Add / remove round-trip works against the task-02 API; every in-flight state disables its control and re-enables on resolution (never stale); errors are inline `role="alert"`.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks, labeled controls, contrast ≥4.5:1, focus-visible; full-width table (no skinny list); no CDN (rule 6).
|
|
||||||
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Task 05 — Admin-only "Git sources" nav link on all five pages
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make the new page reachable from everywhere: the admin-only **"Git sources"** nav link in all five identical headers (phase 34's contract), revealed for the admin by `header.js` — the exact phase-29 pattern.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. The five page headers — `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/tuning.html`, `frontend/git-sources.html` — inside `<nav class="app-nav" aria-label="Primary">`, **immediately after** the `#nav-sources` link, add (mirroring the `#nav-tuning` markup, comment citing this phase + owner permission 2026-08-26):
|
|
||||||
```html
|
|
||||||
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
|
|
||||||
```
|
|
||||||
- Rules (phase-29 contract): `hidden` by default on every page (anonymous-safe ship-hidden); NO `is-active` / `aria-current` on the four pages that aren't the Git sources page.
|
|
||||||
- **Exception:** on `frontend/git-sources.html` the link carries `class="nav-link is-active"` + `aria-current="page"` (the current page, like Tuning on `tuning.html`).
|
|
||||||
- Nav order on every page becomes: Chat, Sources, **Git sources**, Tuning.
|
|
||||||
2. `frontend/assets/header.js` — next to the `navTuning` reveal block, add the same ship-hidden/reveal-for-admin contract:
|
|
||||||
```js
|
|
||||||
const navGitSources = document.querySelector("#nav-git-sources");
|
|
||||||
if (navGitSources) navGitSources.hidden = !admin;
|
|
||||||
```
|
|
||||||
Update the file-header comment (the admin-only link list now includes Git sources).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Frontend-only — no Python change; the no-CDN integration test is unaffected (same-origin `<a>`).
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Manual smoke: admin sees "Git sources" on all five pages → each navigates to `/git-sources.html` (with `is-active` there); anonymous never sees it (ships hidden, no flash).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All five headers contain `#nav-git-sources` after `#nav-sources`, `hidden` by default; `git-sources.html`'s carries `is-active` + `aria-current="page"`.
|
|
||||||
- [ ] `header.js` reveals it for the admin on the cached whoami (no extra request) and hides it for anonymous on every page.
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Task 06 — Story E2E + nav-inventory update + docs + commit
|
|
||||||
|
|
||||||
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
|
|
||||||
**Story:** `.agent/user_stories/git-sources-admin.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the story end-to-end with its dedicated Playwright suite, keep phase 34's nav-consistency contract in sync with the new link, document the env-var demotion, and close the phase with the full gate + one commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_git_sources_admin.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (`tests/e2e/conftest.py`); admin session via `tests/e2e/auth_helpers.py`; **no git, no network** — this suite is UI + API only (the clone/import path is mocked at the integration level, task 03).
|
|
||||||
- **Anonymous:** `/git-sources.html` shows the sign-in gate (list + add form absent/inert); `#nav-git-sources` hidden on the five pages; the API 403s (assert via the page context's `fetch` or `context.request` — follow `test_admin_auth.py`'s pattern for anonymous API assertions).
|
|
||||||
- **Admin — nav:** `#nav-git-sources` visible on all five pages; clicking it from `/` lands on `/git-sources.html` with the link `is-active`.
|
|
||||||
- **Admin — list:** seed two rows via the API (or `SessionLocal`) before load → both rows render (mono URL text, added date); the env note is hidden (DB rows exist).
|
|
||||||
- **Admin — add:** submit `https://example.com/reese/new-repo.git` → the row appears, the input clears, the button re-enables (never stale); submit a duplicate → inline `role="alert"` error, no new row, button re-enabled; submit `not a valid url` → inline 422 error, button re-enabled.
|
|
||||||
- **Admin — remove:** click a row's Remove → accept the confirm dialog (Playwright `page.on("dialog")`) → the row disappears; cancel a second removal → the row stays.
|
|
||||||
- **Admin — env fallback:** truncate `git_sources`, set the E2E app's `BOR_GIT_SOURCES` (follow how `test_sync_button.py` controls the env on the app fixture), reload → the env rows render + `#git-sources-env-note` visible.
|
|
||||||
- The sync-origin behavior (DB over env in the pipeline) is integration-level (task 03) — do not trigger a real sync in this suite.
|
|
||||||
2. `tests/e2e/test_nav_consistency.py` (phase 34 — UPDATE): the admin nav inventory now includes **"Git sources"** (four links, order Chat, Sources, Git sources, Tuning); the anonymous hidden set gains `#nav-git-sources`; the per-page inventory comparison stays order-sensitive.
|
|
||||||
3. **Docs:**
|
|
||||||
- `.env.example` — the `BOR_GIT_SOURCES` comment: now the **empty-table fallback**; the primary management UI is the admin Git sources page (phase 35).
|
|
||||||
- `README.md` — the import/update workflow section: the git-sources list is managed on the admin page (stored in Postgres); `BOR_GIT_SOURCES` only applies while that list is empty; `--source` still overrides for manual runs.
|
|
||||||
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_git_sources_admin.py` (new), `test_nav_consistency.py` (updated), `test_sync_button.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
|
|
||||||
5. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
|
|
||||||
6. **UI Structure Check** (AGENTS.md rule 5) on the new page (full-width table, labels, contrast, focus-visible, aria-live on the list updates) + no CDN (rule 6).
|
|
||||||
7. **Commit** (A17): stage this phase's files (`app/**`, `alembic/**`, `frontend/**`, `tests/**`, `README.md`, `.env.example`), message `feat(sources): admin page to add and remove git sources (TODO.md L4)`, always `--no-gpg-sign`. Move `.agent/phases/todo/35_git_sources_admin/` to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `tests/e2e/test_git_sources_admin.py` green **in isolation** (A16: one story, one file).
|
|
||||||
- Unit/integration: from tasks 01–03 — all green under `uv run pytest`.
|
|
||||||
- Coverage: **>90%** on `app/` (new API module + resolver fully covered).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `test_nav_consistency.py` updated for the fourth nav link and green; every suite in the task 06 regression list green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] README + `.env.example` document the fallback semantics.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Phase 36 — Document Summary Shown Together With the Original
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L5 — "When I click on a document with a summary I should be able to see the summary and the original document together."
|
|
||||||
**Story:** `.agent/user_stories/summary-in-viewer.md`
|
|
||||||
**Context:** Phase 30 stores a lite-model summary on `documents.summary` for every non-markdown document (plus an indexed `is_summary` chunk) — but the viewer never shows it: `GET /api/documents/content` omits the field and the shared renderer `renderDocument` (`frontend/assets/document.js`, used by BOTH the full-page viewer `document.html` and the chat/sources modal, phase 26) only renders the raw content. Markdown documents carry no summary (phase 30) and must render exactly as before.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
When a document **has** a summary, show it and the original content together — a labeled Summary panel above the content, on both viewer surfaces at once (shared renderer); documents without a summary are unchanged.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `30_document_summaries` (complete) — the `documents.summary` column (migration 0004), the summarizer, the `summary_kb` E2E fixture + the deterministic mock `SUMMARY_MODE` digest.
|
|
||||||
- `10_story_document_viewer` + `26_document_modal_viewer` (complete) — the viewer page, the modal, and the shared `renderDocument(doc, {titleEl, metaEl, contentEl})` core both surfaces render through.
|
|
||||||
- `16_admin_auth` (complete) — the soft rule this phase must not touch: the content endpoint stays public + stateless (catalog gated, viewer public).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_content_api_summary_field.md` — `DocContent.summary` + the endpoint returns it; integration tests.
|
|
||||||
2. `02_viewer_summary_panel.md` — the shared renderer draws the `.doc-summary` panel (both surfaces) + theme-matched styles.
|
|
||||||
3. `03_e2e_and_regression.md` — the story E2E suite `test_summary_in_viewer.py`; regressions; full gate; commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: the content endpoint returns the summary for a summarized non-markdown doc and `null` for a markdown doc; anonymous access unchanged.
|
|
||||||
- Coverage: **>90%** on `app/` (the touched endpoint stays covered).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_summary_in_viewer.py` — the story gate, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `GET /api/documents/content` returns `summary` (string or null); no auth/shape change beyond the added nullable field; the endpoint is still public.
|
|
||||||
- [ ] A summarized document shows the labeled Summary panel **above** the original content in the full-page viewer AND the modal; the original content (including content the summary digest doesn't contain) is fully visible.
|
|
||||||
- [ ] A markdown document (no summary) renders exactly as before on both surfaces — no empty panel.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation; regressions (task 03 list) green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): the panel is a labeled section, contrast ≥4.5:1, no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A7 / A15 untouched** — retrieval, context assembly, and the SSE contract are unchanged; this is a display + API-field phase.
|
|
||||||
- **Phase 16 soft rule untouched** — `GET /api/documents/content` stays public + stateless (anyone who can open a document sees its summary; the catalog stays admin-gated).
|
|
||||||
- **Phase 30 untouched** — summaries are still generated at import, still markdown-excluded, still fail-soft (NULL possible); this phase only surfaces the existing field.
|
|
||||||
- **Shared-renderer principle (phase 26)** — the panel is drawn in `renderDocument`, so the page and the modal can never drift.
|
|
||||||
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; summary text rendered with `textContent` (XSS contract unchanged).
|
|
||||||
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Task 01 — The content endpoint returns the summary
|
|
||||||
|
|
||||||
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
|
|
||||||
**Story:** `.agent/user_stories/summary-in-viewer.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Surface the existing `documents.summary` field (phase 30) on the viewer's data contract: `DocContent` gains a nullable `summary` and `GET /api/documents/content` returns it — endpoint stays public, stateless, and otherwise byte-identical.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/schemas.py` — `DocContent` gains:
|
|
||||||
```python
|
|
||||||
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
|
|
||||||
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
|
|
||||||
#: summary generation failed but the document was still indexed.
|
|
||||||
summary: str | None = None
|
|
||||||
```
|
|
||||||
(Place it after `format` / before `content`, with the docstring mirroring `Document.summary`'s.)
|
|
||||||
2. `app/api/docs.py` — `get_document_content` returns `summary=doc.summary` in the `DocContent(...)` construction. Nothing else changes (no auth, no query change — `Document` is already selected in full).
|
|
||||||
3. Integration tests — extend the existing `/api/documents/content` test module (find it in `tests/integration/` — the phase-10 content-endpoint tests):
|
|
||||||
- A non-markdown document row seeded with `summary="…"` → response JSON carries `summary` verbatim.
|
|
||||||
- A markdown document row with `summary=None` → `"summary": null`.
|
|
||||||
- Anonymous (no admin cookie) still gets 200 (the phase-16 soft rule — public viewer) for both.
|
|
||||||
- The existing assertions (404 on unknown pair, content/format fields) stay green unmodified.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the cases above; `uv run pytest` green overall.
|
|
||||||
- Coverage: **>90%** on `app/` — the touched endpoint stays covered (the new field is exercised by the new assertions).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `GET /api/documents/content?source=…&path=…` returns the summary for a summarized doc and `null` for a markdown doc; anonymous access unchanged (200).
|
|
||||||
- [ ] No other field, status code, or auth behavior of the endpoint changed.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
# Task 02 — The shared renderer draws the Summary panel (both surfaces)
|
|
||||||
|
|
||||||
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
|
|
||||||
**Story:** `.agent/user_stories/summary-in-viewer.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
`renderDocument` — the single rendering core both the full-page viewer (`document.html`) and the chat/sources modal (`document-modal.js`) go through (phase 26) — draws a labeled Summary panel above the original content whenever `doc.summary` is non-empty; `null`/empty renders nothing, so markdown documents and fail-soft rows are byte-for-byte unchanged.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/document.js` — in `renderDocument(doc, { titleEl, metaEl, contentEl })`, **after** the meta row is populated and **before** the content is built, append the summary section to `contentEl` (which is then filled with the usual `.doc-md` / `<pre class="doc-raw">` content after it):
|
|
||||||
```js
|
|
||||||
if (doc.summary && doc.summary.trim() !== "") {
|
|
||||||
const section = document.createElement("section");
|
|
||||||
section.className = "doc-summary";
|
|
||||||
section.setAttribute("aria-label", "Summary");
|
|
||||||
const title = document.createElement("h2");
|
|
||||||
title.className = "doc-summary-title";
|
|
||||||
title.textContent = "Summary";
|
|
||||||
const body = document.createElement("p");
|
|
||||||
body.className = "doc-summary-text";
|
|
||||||
body.textContent = doc.summary; // text node — XSS contract unchanged
|
|
||||||
section.append(title, body);
|
|
||||||
contentEl.appendChild(section);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- The meta badge row (`#doc-meta` / `#doc-modal-meta`) is untouched; the panel sits between meta and content on **both** surfaces because both call this one function.
|
|
||||||
- Update the file-header comment (the renderer now also owns the optional summary panel) and the `renderDocument` doc comment.
|
|
||||||
2. `frontend/assets/styles.css` — `.doc-summary` (dark tech theme, phase-08 palette):
|
|
||||||
- A clearly-distinct "summary, not content" look: surface `#121a2e` with a 3px brand left border (`#6d78f2`) or a brand-soft (`#232b52`) header strip — pick one and keep it simple; `border-radius` matching the existing content cards; padding ~1rem; `margin-bottom` separating it from the content.
|
|
||||||
- `.doc-summary-title` — small-caps/label treatment: `#a5b4fc` (brand-ink on the brand-soft chip, ≈6.9:1) or `#a5b4fc` on surface (verify ≥4.5:1 — if short, use the chip).
|
|
||||||
- `.doc-summary-text` — `var(--ink)` (`#e8ebf4`) on the surface (≈14.5:1); wraps inside the same width the content uses (the ≤46rem centered column for md docs, the raw-content width otherwise — the panel is a child of `contentEl`, so it inherits the column; verify for the `<pre class="doc-raw">` case where the content is wider).
|
|
||||||
- No animation (nothing for `prefers-reduced-motion` to still); the section is static content — no focusability needed (it carries `aria-label` + heading).
|
|
||||||
3. Do NOT touch `document-modal.js` (it calls `renderDocument` — the panel comes for free), `markdown.js`, or the page scripts.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Frontend-only — no Python change; the no-CDN integration test is unaffected.
|
|
||||||
- Coverage: `app/` gate unaffected.
|
|
||||||
- Manual smoke (dev server with an imported non-md doc, e.g. a yaml from the fixture KB via the importer + mock, or a hand-seeded `documents.summary` row): modal from the Sources table AND the full page both show panel + content; a markdown doc shows no panel.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] A document with a summary shows the labeled panel above the original content in **both** the modal and the full-page viewer; the original content is fully visible (nothing hidden/collapsed).
|
|
||||||
- [ ] A document without a summary (`null` or empty/whitespace) renders exactly as before on both surfaces — no panel, no empty box.
|
|
||||||
- [ ] The summary text is written with `textContent` (XSS contract); the meta row is unchanged.
|
|
||||||
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# Task 03 — Story E2E + regression pass + commit
|
|
||||||
|
|
||||||
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
|
|
||||||
**Story:** `.agent/user_stories/summary-in-viewer.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the story end-to-end with its dedicated Playwright suite — summary panel + original content visible together on both surfaces, no panel for markdown docs — then the full gate + one commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_summary_in_viewer.py` (NEW — the story gate, run in isolation). Reuse `tests/e2e/test_document_summaries.py`'s machinery as closely as possible (its fixture KB + import helpers are the reference):
|
|
||||||
- Fixtures: the standard E2E app + DB + the deterministic mock LLM (`tests/e2e/mock_llm.py` — its `SUMMARY_MODE` answer is a byte-stable 24-token digest); import `tests/fixtures/summary_kb/` so `quadlet/qwen-llamacpp.yaml` gets its stored summary + `notes/qwen-llamacpp-notes.md` stays summary-less (the existing tail sentinel `RESE-SUMMARY-SENTINEL-7f3a` sits on the yaml's LAST line — **outside** the 24-token digest, so it is a marker for "the original, not the summary").
|
|
||||||
- **Full page:** open `/document.html?source=summary_kb&path=quadlet%2Fqwen-llamacpp.yaml` → `.doc-summary` visible with the deterministic digest text AND the original content visible with the sentinel (`RESE-SUMMARY-SENTINEL-7f3a` present in the rendered content) — summary and original **together**.
|
|
||||||
- **Modal:** from the Sources table (admin session via `tests/e2e/auth_helpers.py`), click the yaml's row → the modal shows the same panel + content (sentinel present, digest present); then "Full page" still lands on the dedicated page with the panel (the two surfaces agree).
|
|
||||||
- **No-summary control:** the markdown doc (`notes/qwen-llamacpp-notes.md`) → no `.doc-summary` element on the full page and in the modal; the content renders as before.
|
|
||||||
- **API shape (cheap, via the page context's `fetch` or `context.request`):** `GET /api/documents/content` for the yaml carries `summary` (string), for the md doc `null`; anonymous fetch → 200 (soft rule unchanged).
|
|
||||||
- The E2E must not depend on a real LLM (the mock's digest is deterministic — the same pattern `test_document_summaries.py` relies on).
|
|
||||||
2. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_summary_in_viewer.py` (new), `test_document_viewer.py`, `test_document_summaries.py`, `test_document_back_navigation.py`, `test_chat_rag.py` (the source-chip modal path), `test_smoke.py`.
|
|
||||||
3. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
|
|
||||||
4. **UI Structure Check** (AGENTS.md rule 5): the panel is a labeled section (`aria-label` + heading), theme contrast ≥4.5:1, it does not break the centered 46rem chat-column-width content layout, no CDN (rule 6 — the no-CDN integration test covers it).
|
|
||||||
5. **Commit** (A17): stage this phase's files (`app/schemas.py`, `app/api/docs.py`, `frontend/assets/document.js`, `frontend/assets/styles.css`, `tests/**`), message `feat(viewer): show document summary together with the original (TODO.md L5)`, always `--no-gpg-sign`. Move `.agent/phases/todo/36_summary_in_viewer/` to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `tests/e2e/test_summary_in_viewer.py` green **in isolation** (A16: one story, one file).
|
|
||||||
- Unit/integration: from task 01 — all green under `uv run pytest`.
|
|
||||||
- Coverage: **>90%** on `app/` (the endpoint change is covered).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] Every suite in the task 03 regression list green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Task 01 — Live tool-calling probe
|
|
||||||
|
|
||||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "The agent should be able to list the available sources as a tool and the read the ones it thinks are relevant"`
|
|
||||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Verify against the live aipi endpoint whether the `turbo` chat model supports OpenAI-style `tools` + streaming `tool_calls` before building the loop — the phase-17 "verified live" convention — and record which path the phase takes.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `scripts/llm_probe.py` — add a `--tools` flag: when set, send (a) a non-streaming `chat/completions` request with one trivial function (e.g. `get_time`, no parameters) and a user message that makes calling it natural, and (b) the same with `stream=True`; for each, print whether `finish_reason` is `tool_calls`, the parsed `function.name`/`arguments`, and (streaming) whether the calls arrive as `delta.tool_calls` chunks with `index`/`id`/partial `function.arguments` (the OpenAI wire convention). Reuse the existing env reading (`BOR_LLM_BASE_URL` / API key / `BOR_LLM_CHAT_MODEL`) and the script's existing output style.
|
|
||||||
2. Run it against the live endpoint (`uv run python -m scripts.llm_probe --tools`) and classify the verdict:
|
|
||||||
- **supported** → the phase uses OpenAI `tools`/`tool_calls` (tasks 02–03 as written).
|
|
||||||
- **not supported** → the phase uses the prompt-based structured-call fallback (task 03 documents it): the model is instructed to emit a single JSON block (`{"tool": "list_documents"}` / `{"tool": "read_document", "source": …, "path": …}`) before answering; `run_agent` parses it out of the content stream; the SSE `tool` contract and the budgets are identical.
|
|
||||||
- **intermittent** → treat as not supported (fail-loud house style) and note it.
|
|
||||||
3. Record the verdict + date where it will be read later: the task-03 `app/rag/agent.py` module docstring (task 03 writes it) and the phase commit message (task 06) — e.g. `probe: turbo tool_calls=streaming-ok 2026-08-26`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- The probe is a CLI script (no `app/` coverage impact). If the parsing of the probe response is factored into a function, add a small unit test for it.
|
|
||||||
- `uv run pytest` green (no regressions); ruff + pyright clean.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run python -m scripts.llm_probe --tools` runs and prints a clear supported / not-supported verdict for both the non-streaming and the streaming request.
|
|
||||||
- [ ] The verdict (with date) is available for task 03's docstring and task 06's commit message.
|
|
||||||
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# Task 02 — LLM client: `tools` + tool-call streaming
|
|
||||||
|
|
||||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…read the ones it thinks are relevant… These values should be configured by environment variables"` (the client plumbing the env-tuned loop runs on)
|
|
||||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Teach `LLMClient.chat_stream` to pass an OpenAI `tools` list and to accumulate streaming `tool_calls` deltas into typed pieces — with `tools=None` producing a byte-identical request to today.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/llm.py`
|
|
||||||
- New frozen dataclass next to `StreamPiece`:
|
|
||||||
```python
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ToolCallPiece:
|
|
||||||
"""One model-requested tool call accumulated from stream deltas (phase 37)."""
|
|
||||||
id: str # the model's tool_call id; synthesized "call_<index>" when absent
|
|
||||||
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
|
|
||||||
arguments: dict[str, Any]
|
|
||||||
```
|
|
||||||
- `chat_stream(self, messages, tools: list[dict[str, Any]] | None = None)`:
|
|
||||||
- When *tools* is not None, pass `tools=tools` to `chat.completions.create`; when None, do **not** include the key (byte-identical request to today).
|
|
||||||
- In the chunk loop, accumulate `delta.tool_calls` (a list of partials keyed by `index`; `id` and `function.name` arrive on the first partial for an index, `function.arguments` arrives in fragments to concatenate).
|
|
||||||
- At stream end (or when a chunk carries `finish_reason == "tool_calls"`), for each accumulated call **in index order** yield `ToolCallPiece(id, name, json.loads(arguments) or {})`.
|
|
||||||
- Malformed `arguments` JSON → raise `LLMError` (fail-loud house style — a silently dropped tool call would corrupt the loop).
|
|
||||||
- Return annotation becomes `AsyncIterator[StreamPiece | ToolCallPiece]`; update the docstring (wire convention + a pointer to the task-01 probe verdict).
|
|
||||||
- **If the task-01 verdict is "not supported"** (prompt-based fallback): skip the `tools` parameter and the delta accumulation entirely; keep `StreamPiece` unchanged; the JSON-block parse helper lands in `app/rag/agent.py` (task 03) instead.
|
|
||||||
2. `app/api/chat.py` — update the phase-17 typing import line (`StreamPiece # noqa: F401`) to also import `ToolCallPiece` so pyright sees the union; the dispatch wiring itself is task 04.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit (`tests/unit/` — extend the existing `chat_stream` tests): synthetic chunk sequences — (a) a content-only stream is unchanged (no `ToolCallPiece`; the captured `create()` kwargs have no `tools` key); (b) a tool-call stream with partials across chunks (name on the first, arguments in 2–3 fragments) → one `ToolCallPiece` with the merged JSON; (c) two calls in one stream (indices 0 and 1) → both, in index order; (d) malformed arguments JSON → `LLMError`; (e) `tools=[…]` present in the request when passed.
|
|
||||||
- Coverage: **>90%** on the modified module.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `chat_stream(messages)` (no tools) — all existing unit tests green unchanged.
|
|
||||||
- [ ] The new tool-call accumulation tests green; full `uv run pytest` green.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
# Task 03 — The basic agent loop (`app/rag/agent.py`)
|
|
||||||
|
|
||||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "it gets one opportunity to list documents and then one opportunity to add exactly one extra document to its context before being required to answer. These values should be configured by environment variables."`
|
|
||||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
A testable agent loop: the model gets the two tools while budgets last, the app executes them against Postgres, and once both budgets are spent the tools are dropped so the model is required to answer.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — two settings (documented, `BOR_` prefix per house style, near the RAG-tuning block):
|
|
||||||
```python
|
|
||||||
#: Per-turn opportunities to call the `list_documents` agent tool (phase 37);
|
|
||||||
#: 0 disables the tool entirely (pre-phase behavior with both at 0).
|
|
||||||
agent_list_calls: int = 1
|
|
||||||
#: Per-turn opportunities to call `read_document` (phase 37); 0 disables.
|
|
||||||
agent_read_calls: int = 1
|
|
||||||
```
|
|
||||||
2. `app/rag/agent.py` — new module. The module docstring carries: the phase, the loop contract, and the task-01 probe verdict + date (which path is in use — OpenAI tool_calls vs the prompt-based fallback).
|
|
||||||
- `AGENT_TOOLS`: the two OpenAI function definitions — `list_documents` (no parameters; description: "List every document indexed in the knowledge base, one `source/path — title` line each") and `read_document` (`source` + `path` required; description: "Add the full content of exactly one more indexed document to your context").
|
|
||||||
- DB accessors (module-level functions so unit tests can monkeypatch them):
|
|
||||||
- `list_catalog(db) -> list[tuple[str, str, str]]` — `select(Document.source, Document.path, Document.title).order_by(Document.source, Document.path)` (same order as `GET /api/docs`).
|
|
||||||
- `find_document(db, source, path) -> Document | None`.
|
|
||||||
- `@dataclass AgentHolder: read_docs: list[Document] = field(default_factory=list); tool_calls: int = 0` — the API layer (task 04) reads it after the stream.
|
|
||||||
- `async def run_agent(llm, db, *, system_prompt, user_message, seed_docs, settings, holder) -> AsyncIterator[StreamPiece | ToolCallPiece]`:
|
|
||||||
1. `messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_message}]`; `list_left = settings.agent_list_calls`; `read_left = settings.agent_read_calls`; `tools = AGENT_TOOLS if (list_left or read_left) else None`; `rounds = 0`; cap `max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls` (belt-and-braces — every tool round must consume a budget, so the cap only catches pathological streams).
|
|
||||||
2. Loop: stream `llm.chat_stream(messages, tools=tools)`; yield every piece as it arrives. Collect any `ToolCallPiece` (handle the first; if a stream yields both content and a tool call — rare — the content stays (it was already emitted) and the tool still runs).
|
|
||||||
- **No tool call** → return (the answer was streamed).
|
|
||||||
- **`list_documents`**: `list_left > 0` → `list_left -= 1`, `holder.tool_calls += 1`, result = `f"{n} documents:\n" + "\n".join(f"{s}/{p} — {t}")` (uncapped in v1 — **ASSUMPTION: the catalog is not truncated; the UI never shows it, only the model does**). Else result = `"No listing budget left — answer with what you have."` (no budget consumed).
|
|
||||||
- **`read_document`**: arguments must carry `source` and `path`. If that document is already in `seed_docs` or `holder.read_docs` → result = `"Already in your context."` (no budget consumed, no append). Else if `read_left > 0` → `doc = find_document(db, source, path)`; found → `read_left -= 1`, `holder.tool_calls += 1`, `holder.read_docs.append(doc)`, result = `f"Document {source}/{path}:\n{doc.content}"` (**full text, never truncated — the A7-revised contract**); not found → result = `f"No document at {source}/{path} — check the list_documents output."` (no budget consumed). `read_left == 0` → result = `"No reading budget left — answer with what you have."`.
|
|
||||||
- **Unknown tool name** → result = `"Unknown tool."` (no budget consumed).
|
|
||||||
- Append to `messages`: the assistant tool-call message (`{"role": "assistant", "content": None, "tool_calls": [{"id": tc.id, "type": "function", "function": {"name": tc.name, "arguments": json.dumps(tc.arguments)}}]}`) + the tool result (`{"role": "tool", "tool_call_id": tc.id, "content": result}`); then `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` (once both budgets are spent, the next request must be answered).
|
|
||||||
- `rounds += 1`; if `rounds >= max_rounds` → force one final `chat_stream(messages, tools=None)` (yield its pieces) and return.
|
|
||||||
- **Prompt-based fallback (only if the task-01 verdict is "not supported")**: instead of `ToolCallPiece`s, scan each streamed content turn for a leading JSON block matching `{"tool": …}` (strip leading whitespace, parse with `json`; the block must be the first non-whitespace content of the turn). On a match: strip the block from the emitted stream (re-emit only trailing content, if any), then run the identical budget/result machinery. The SSE `tool` events and the budgets are unchanged either way.
|
|
||||||
3. `app/rag/prompts.py` — add the `<tools>` instructions section to the **HIGH prompt only** (a new constant, appended after the mode body; the LOW/deflection prompt stays byte-identical). Wording (tune against the live model if needed, but keep it stable — the E2E mock keys off the `<tools>` marker's *presence*, not the wording):
|
|
||||||
> If the documents in your context reference other files, or you need content that is not included above, call `list_documents` to see what is indexed, then `read_document` to pull in exactly one more document. Answer as soon as you have what you need — do not read more than one extra document.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit (`tests/unit/test_agent.py`, new): a scripted fake LLM (canned stream sequences) + monkeypatched `list_catalog`/`find_document` + a `Settings` with budgets set:
|
|
||||||
- list → read → answer: event order (tool pieces before content), `holder.read_docs == [doc]`, `holder.tool_calls == 2`, the request after budgets are spent carries `tools=None`, the follow-up request contains the assistant tool-call + tool-result messages.
|
|
||||||
- Budgets: `read_left` exhausted → a second `read_document` gets "No reading budget left" and appends nothing; `agent_list_calls=0` + `agent_read_calls=0` → exactly one request with `tools=None` (byte-identical single-call path).
|
|
||||||
- Edge: reading a doc already in `seed_docs` → "Already in your context." (no budget consumed); unknown path → "No document at …"; unknown tool name → "Unknown tool."; the round cap forces a final no-tools answer.
|
|
||||||
- The HIGH prompt gains the `<tools>` section; the LOW prompt is byte-identical to pre-phase (assert against the existing prompt-test fixtures).
|
|
||||||
- Integration (`tests/integration/`): `list_catalog` ordering + `find_document` hit/None against real Postgres (follow the existing DB-test patterns).
|
|
||||||
- Coverage: **>90%** on the new module.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `app/rag/agent.py` exists with the loop above; the module docstring records the probe verdict + date (task 01).
|
|
||||||
- [ ] The budget matrix (0/1) unit tests green, including "budgets 0/0 ⇒ one request, no tools".
|
|
||||||
- [ ] HIGH prompt carries `<tools>`; LOW prompt byte-identical to pre-phase.
|
|
||||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# Task 04 — `POST /api/chat`: `tool` SSE events + agent wiring
|
|
||||||
|
|
||||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "This will require some reconfiguring of the UI since it will now need to show 'calling tool' in addition to 'thinking' and it will need a basic agent loop."`
|
|
||||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Wire the loop into the chat endpoint: the new SSE `tool` event, the agent on grounded turns only, and the read document reflected in `done.sources`, `query_log`, and the per-turn log line.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/schemas.py` — add next to `ChatThinkingEvent`:
|
|
||||||
```python
|
|
||||||
class ChatToolEvent(BaseModel):
|
|
||||||
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension)."""
|
|
||||||
type: Literal["tool"] = "tool"
|
|
||||||
name: str # "list_documents" | "read_document"
|
|
||||||
argument: str | None = None # "source/path" for read_document
|
|
||||||
```
|
|
||||||
2. `app/api/chat.py`
|
|
||||||
- In `stream()`, after `plan_turn`: when `not plan.deflected` → run `run_agent(llm, db, system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.docs, settings=settings, holder=holder)`; when deflected → keep the current direct `chat_stream` (A8 byte-identical).
|
|
||||||
- Event mapping: `StreamPiece` thinking/delta exactly as today (including the `BOR_STREAM_THINKING` kill-switch and the `thinking_chars` count); `ToolCallPiece` →
|
|
||||||
`yield sse_event(ChatToolEvent(name=tc.name, argument=f"{tc.arguments.get('source')}/{tc.arguments.get('path')}" if tc.name == "read_document" else None).model_dump())`.
|
|
||||||
- `done` event: `sources` = `plan.docs + holder.read_docs` deduped by `(source, path)`, order preserved; the `ChatDoneEvent` shape otherwise unchanged.
|
|
||||||
- `query_log.sources`: the same combined list (replaces the current `source_paths` build).
|
|
||||||
- Per-turn log line: add `tool_calls={holder.tool_calls}` after `thinking_chars=` (PLAN §9 required-line extension — the phase 17/30/31 precedent).
|
|
||||||
- Extend the module docstring with the phase-37 section (the flow, the grounded-only scope, the budgets, the "both budgets 0 ⇒ pre-phase behavior" note).
|
|
||||||
3. `.agent/PLAN.md` — record two revision notes in the project's established format (phase 17/24 precedent, "owner permission 2026-08-26"): the §4 SSE contract gains `{"type":"tool","name":…,"argument":…}` (client rule: render as a "calling tool" line/state; `delta`/`done` unchanged) and the §9 per-turn log line gains `tool_calls=N`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration (`tests/integration/test_chat*.py` — extend the existing SSE-contract suite, mock LLM that emits tool calls): (a) grounded turn with tool calls → event sequence `thinking?/tool/tool/delta…/done`, `done.sources` includes the read doc, the `query_log` row's sources match, the log line carries `tool_calls=2`; (b) deflected turn → no `tool` events, the sequence byte-identical to today; (c) budgets 0/0 → no `tool` events, the mock received no `tools` parameter, single-request path.
|
|
||||||
- Coverage: **>90%** on the modified modules.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `POST /api/chat` (mock LLM, grounded) streams `tool` frames ahead of the `delta` frames; the deflected path is unchanged (existing suites green).
|
|
||||||
- [ ] `done.sources`/`query_log` include the read document (deduped); the log line carries `tool_calls=N`.
|
|
||||||
- [ ] The PLAN.md §4 + §9 revision notes exist (dated, owner permission 2026-08-26).
|
|
||||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Task 05 — UI: the "calling tool" state
|
|
||||||
|
|
||||||
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…it will now need to show 'calling tool' in addition to 'thinking'"`
|
|
||||||
**Story:** `.agent/user_stories/agent-document-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The chat shell shows a "calling tool" state (button + status label) and a visible tool line in the bubble, distinct from the Thinking scratchpad; tool lines persist with the chat record (phase 14) and re-render after reload.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/app.js` — in the turn handler where the SSE frames are dispatched (the same switch that handles `thinking` and `delta`):
|
|
||||||
- New `tool` branch:
|
|
||||||
- **Status/label:** keep `uiState = thinking` (the button stays disabled — never stale, PLAN §7.4) and set the typing-indicator label to "Brain of Reese is listing documents" (`name: "list_documents"`) / "Brain of Reese is reading <argument>" (`name: "read_document"`); the elapsed-seconds hint (`thinkingClock` aria-label) keeps working through tool frames.
|
|
||||||
- **Bubble line:** append a tool line to the same wrap the thinking block uses (above the answer, beside/below the Thinking summary): `<span class="tool-call">🔎 Listing documents</span>` / `<span class="tool-call">📄 Reading <code>source/path</code></span>` — a real visible line with its own icon + color, distinct from the Thinking block (styles step). Multiple calls append multiple lines, in order.
|
|
||||||
- Tolerate tool frames interleaved with thinking frames (append-only, the same rule as thinking); a tool frame after the first `delta` (should not happen in v1 — the loop completes before the answer stream) still appends rather than crashes.
|
|
||||||
- **Persistence (phase 14 convention):** the saved record gains an optional `tools: [{name, argument}]` array next to `thinking`; re-render the tool lines when a record is loaded (same code path as the thinking re-render).
|
|
||||||
- Keep the new label strings as plain literals — phase 39 centralizes brand strings; do **not** introduce a brand helper here.
|
|
||||||
2. `frontend/assets/styles.css` — `.tool-call` style: inline row, icon + text, an accent color distinguishable from the thinking block's, `code` styling for the path, contrast ≥ 4.5:1 in both themes; no layout shift on append (the centered 46rem chat column is untouched — no new container).
|
|
||||||
3. UI Structure Check (AGENTS.md rule 5) before finalizing: semantic landmarks unchanged; the tool lines live inside the `#messages` `aria-live="polite"` region (announced to screen readers); focus-visible unaffected (the lines are not interactive); no new top-level landmarks.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No new Python logic — gated by the story E2E (task 06) plus the existing frontend-adjacent suites staying green.
|
|
||||||
- No CDN (AGENTS.md rule 6): no new `<script>`/`<link>` tags.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] During a tool call the button label shows the "calling tool" text (not a stale "Thinking…") and the tool lines render above the answer.
|
|
||||||
- [ ] After a page reload, the persisted record re-renders the tool line(s).
|
|
||||||
- [ ] `uv run pytest` green (no regressions); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Task 01 — `git_sources.kind` + `path` (migration 0007)
|
|
||||||
|
|
||||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
|
||||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Extend the phase-35 `git_sources` table with a source-kind discriminator (`git` | `local`) and an optional local path, via a reversible migration — the foundation for the API, pipeline, and page tasks.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `alembic/versions/0007_git_sources_kind.py` — read `alembic/versions/0006_git_sources.py` first and chain from its **actual** `revision` id (filenames are not revision ids — the phase-35 task-01 convention):
|
|
||||||
- `upgrade()`: `ALTER TABLE git_sources ADD COLUMN kind TEXT NOT NULL DEFAULT 'git'` + `ADD CONSTRAINT ck_git_sources_kind CHECK (kind IN ('git', 'local'))`; `ALTER TABLE git_sources ADD COLUMN path TEXT`; a unique index on `path` — a partial unique index (`WHERE path IS NOT NULL`) where the 0006 style allows it, otherwise a plain unique index (Postgres treats NULLs as distinct, and the API enforces local-only paths anyway — mirror whatever 0006 chose for `url`).
|
|
||||||
- `downgrade()`: drop the index, constraint, and columns in reverse.
|
|
||||||
2. `app/models.py` — extend the phase-35 `GitSource` model: `kind: Mapped[str]` (default `"git"`) + `path: Mapped[str | None]` (nullable, unique) and update the docstring (phase 38: kind discriminator; local rows carry `path`, git rows keep `url`).
|
|
||||||
3. Apply to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`; verify the columns + constraint + that existing rows read as `kind='git'`, `path=NULL`.
|
|
||||||
4. Migration test — follow the 0004/0005/0006 pattern in `tests/integration/`: up adds the columns/constraint/index (existing rows keep `kind='git'`), down drops them (round-trip on the test DB).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the up/down test above; full `uv run pytest` green.
|
|
||||||
- Coverage: model + migration only — the `app/` gate stays >90%.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `alembic/versions/0007_git_sources_kind.py` chains off 0006's real revision id; up/down reversible.
|
|
||||||
- [ ] `uv run alembic upgrade head` applies cleanly; existing rows read `kind='git'`, `path=NULL`.
|
|
||||||
- [ ] The 0007 up/down integration test passes; full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Task 02 — The admin API: the local kind
|
|
||||||
|
|
||||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
|
||||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Extend the phase-35 git-sources CRUD to accept and return `kind=local` rows with fail-loud path validation, keeping the git contract byte-identical.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. The phase-35 API module (the file behind `/api/git-sources` — identify it in the repo; phase 35 created it):
|
|
||||||
- `POST /api/git-sources` body: `{kind?: "git"|"local" (default "git"), url?, path?}`:
|
|
||||||
- `kind=git` → exactly today's `url` validation (trimmed, 1–500 chars, the `https?://` / `ssh://` / `git@` shape, 409 on duplicate without echoing the URL).
|
|
||||||
- `kind=local` → `path` required: trimmed; `Path(p).expanduser()`; must be **absolute after expansion** and an **existing directory on the server** → else `422 {detail: "local source path is not a directory: <path>"}` (a missing path is a user error — fail loud at add-time so the owner sees it immediately; the path is not a secret, so echo it); 409 on a duplicate `path` (detail may name the path).
|
|
||||||
- Wrong field combinations (git without url, local without path, both kinds' fields) → 422.
|
|
||||||
- `GET /api/git-sources` rows gain `kind` + `path` (git rows: `path: null`; the env-fallback rows report `kind: "git"`; `from_env: true` semantics unchanged — env rows are git-only).
|
|
||||||
- `DELETE /api/git-sources/{id}` — unchanged (removal prunes on the next sync, as today).
|
|
||||||
2. The phase-35 request/response models (wherever they live — `app/schemas.py` or the module) — extend for the new fields; keep the OpenAPI docs accurate.
|
|
||||||
3. Integration tests (extend `tests/integration/test_git_sources_api.py`): anonymous → 403 on all routes (regression); `kind=local` + an existing temp dir → 201 with the stored row (`kind=local`, `path` stored expanded); a relative path → 422; a missing path → 422 naming the path; a duplicate path → 409; a git row still validates exactly as before (regression); GET mixes kinds in added order.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: the matrix above; the existing git-kind suite green unchanged.
|
|
||||||
- Coverage: **>90%** on the modified module.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The local-kind POST matrix (201 / 422 / 409) green; the git contract unchanged.
|
|
||||||
- [ ] GET rows carry `kind` + `path`; the env fallback stays git-only.
|
|
||||||
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
# Task 03 — Sync + `import_docs`: git and local together
|
|
||||||
|
|
||||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
|
||||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The canonical mirror action (the Sync button) and the CLI import DB **git + local** rows in one run: git rows clone/pull as today, local rows are walked directly; a missing local directory aborts the run loudly before anything is imported.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. The phase-35 resolution module (`app/rag/git_sources.py` or wherever `effective_git_sources()` lives — identify it in the repo) — extend:
|
|
||||||
- `effective_sources(db) -> list[GitSource]` — the DB rows of **both** kinds (DB-wins / env-fallback semantics unchanged: while the table is empty, the env git URLs surface as synthetic `kind='git'` rows with `from_env`). Keep a backward-compatible alias (`effective_git_sources`) if other modules import the old name.
|
|
||||||
- Log the list origin as today (`origin=db|env`) plus the kind counts (`git=N local=M`).
|
|
||||||
2. `app/api/sync.py::_run_sync`:
|
|
||||||
- For each resolved row: `kind=git` → `clone_or_pull(url, sources_root / repo_name(url))` (unchanged); `kind=local` → `Path(row.path).expanduser()`, verify `.is_dir()` **at sync time** (the directory may have moved/deleted since add-time) → else raise a sync error `local source missing: <path>` (no credentials involved, but run it through the existing `_sanitize_error` for consistency).
|
|
||||||
- `import_sources(combined, llm, prune=True)` over the **single combined list** (git checkouts + local dirs) — pruning covers the union (phase-32 semantics); the overview regeneration is unchanged (change-gated).
|
|
||||||
- The fail-loud empty-config check: no git rows, no local rows, and no env URLs → `"no sources configured (git or local)"` (replaces phase 32's git-only message).
|
|
||||||
3. `scripts/import_docs.py` — `_resolve_sources` gains the DB path (after `--source`, which still wins over everything): no `--source` and the DB table non-empty → the combined list (git cloned/pulled + local direct); table empty → env git URLs (as today) → the legacy `DEFAULT_SOURCES`. A missing local dir → abort with the path named, before importing anything (the same pre-import fail-loud as a failing git clone).
|
|
||||||
4. Unit/integration:
|
|
||||||
- Unit: `effective_sources` — mixed kinds, DB-wins, env fallback git-only, both-empty (extend the phase-35 tests).
|
|
||||||
- Integration (the `test_sync_api.py` pattern, with a **temp local dir** — a host temp dir containing one fixture `.md`, since the app server runs on the same host): local-only, git-only, and mixed syncs — the local file lands in the KB (`GET /api/docs` as admin); a missing local path → status `failed` with `local source missing: …` in the sanitized error; `import_docs` (no `--source`) with a DB local row imports it; `--source` still wins over the DB.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- The suites above; `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` (phase 35's list) stay green through the indirection.
|
|
||||||
- Coverage: **>90%** on the modified modules.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] A mixed git + local sync imports both in one run; union pruning works (a file deleted from the local dir is pruned on the next sync).
|
|
||||||
- [ ] A missing local dir fails the run loudly (the status error names the path) and imports nothing.
|
|
||||||
- [ ] `import_docs`: DB git + local resolution; `--source` wins; the env fallback is git-only; the both-empty message is updated.
|
|
||||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Task 04 — The page: the local-directory form + badges
|
|
||||||
|
|
||||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
|
||||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The sources page (phase 35) can add a **local directory** next to git repos, the list shows which is which, and the hint reflects the combined Sync semantics.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. The phase-35 page (the git-sources HTML template + its page script — identify the actual paths in the repo):
|
|
||||||
- A second add form, **"Local directory"**: a labeled text input (placeholder `~/Notes`), an Add button, an inline error slot — the same never-stale-button + inline-error pattern as the git form (PLAN §7.4); on success the form clears and the list re-fetches.
|
|
||||||
- List rows: a kind badge — `Git` / `Local` (a small styled span, distinguishable by color **and** text, not color alone — WCAG) — plus the mono value (git URL as today; the full local path) + added date + Remove (Remove is unchanged — it prunes on the next sync).
|
|
||||||
- The hint text: "Sync clones/pulls the git repos and imports the local directories together (files removed from a source are pruned)."
|
|
||||||
- Anonymous: the sign-in gate unchanged; the admin-only nav link unchanged (phase 35/29).
|
|
||||||
2. `frontend/assets/styles.css` — the badge styles + the second form (reuse the existing form styles; contrast ≥ 4.5:1 in both themes).
|
|
||||||
3. UI Structure Check (AGENTS.md rule 5) before finalizing: landmarks intact, both inputs labeled, focus-visible on the new Add button, the page stays inside the shared layout (no new top-level structure).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Gated by the story E2E (task 05); no CDN (AGENTS.md rule 6 — no new external tags).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The admin adds a local directory through the page; the row appears with the Local badge; an invalid path shows the 422 detail inline and the button recovers (never stale).
|
|
||||||
- [ ] The git form + all existing page behavior unchanged.
|
|
||||||
- [ ] `uv run pytest` green (no frontend unit layer — the E2E is the gate); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Phase 39 — Configurable app name (brand)
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."
|
|
||||||
**Story:** `.agent/user_stories/configurable-brand.md`
|
|
||||||
**Context:** `app/config.py` (`app_name`, already `BOR_APP_NAME`, currently used only for the OpenAPI title at `app/main.py:48`), the five templates (`frontend/*.html` — the `.brand-text` spans, the `<title>`s, the meta descriptions, the index empty-state h1, the `aria-label`s), `frontend/assets/app.js` (status labels ~L107/112, the elapsed-hint aria ~L540), `frontend/assets/document.js` (page titles L147/152), `app/api/health.py` (the public stateless endpoint pattern), `app/core/caching.py` (phase 33: `?v=` rewriting of `assets/…` refs — automatic for any new asset file), the `Containerfile` esbuild stage (explicit per-asset lines, L17–23).
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Make **one env var** (`BOR_APP_NAME`, default "Brain of Reese") drive the app's display name everywhere — titles, the header brand, the status labels, the aria text, the greeting — via a public `/api/config` endpoint + a small `brand.js`, with zero behavior change when the variable is unset.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `34_consistent_navbar` (todo — runs before this phase) — the standard five-page header (the `.brand-text` nodes this phase re-skins).
|
|
||||||
- `01_infrastructure` (complete) — the public stateless endpoint pattern (`app/api/health.py`).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_config_endpoint.md` — public `GET /api/config` → `{app_name, version}` + tests.
|
|
||||||
2. `02_frontend_branding.md` — `brand.js` + the template/JS de-hard-coding + the Containerfile build line.
|
|
||||||
3. `03_e2e_docs_commit.md` — the story E2E (its own app instance with the overridden name), `.env.example` + README, commit, move the phase dir.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: the endpoint (200, the values, anonymous access, exactly two keys — no settings may leak later).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_configurable_brand.py` — a **second** app instance booted with `BOR_APP_NAME` overridden; run in isolation.
|
|
||||||
- Coverage: **>90%** on `app/`.
|
|
||||||
- Regression: the default-name behavior is byte-identical — the existing suites (which assert "Brain of Reese" titles/labels against the shared conftest server) stay green **unchanged**.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` by default; the response key set is exactly `{app_name, version}`.
|
|
||||||
- [ ] With `BOR_APP_NAME="Brain of Testy"`: the index title "Brain of Testy"; the header `.brand-text` renders `Brain of <strong>Testy</strong>`; the empty-state h1 "Hey! I'm Brain of Testy."; the `#messages` aria-label "Conversation with Brain of Testy"; the chat status label "Brain of Testy is thinking"; the other pages' titles carry the name; the document viewer title "… · Brain of Testy".
|
|
||||||
- [ ] With the variable unset: the existing E2E + unit suites green unchanged (no rename leak).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_configurable_brand.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **The existing `app_name` setting is the source of truth (owner permission 2026-08-26)** — `BOR_APP_NAME` (default "Brain of Reese"); no new variable, no setting rename.
|
|
||||||
- **A11 honoured** — runtime fetch + JS application (no build-time template injection, no Jinja, no CDN); the static templates stay static.
|
|
||||||
- **A10 honoured** — `/api/config` is public and stateless; the response carries no secrets (exactly `app_name` + `version`).
|
|
||||||
- **Runtime fetch, brief flash accepted (owner permission 2026-08-26)** — the default name renders immediately and is replaced when `/api/config` answers (LAN latency; no re-paint machinery for in-flight turns — a mid-turn label keeps the previous name for that turn).
|
|
||||||
- **A16/A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Task 01 — `GET /api/config`
|
|
||||||
|
|
||||||
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
|
|
||||||
**Story:** `.agent/user_stories/configurable-brand.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
A public, stateless endpoint that hands the frontend its display name (+ version) — the single source the brand layer reads.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/api/config.py` — a new router mirroring the `app/api/health.py` pattern:
|
|
||||||
```python
|
|
||||||
@router.get("/config")
|
|
||||||
def app_config(settings: Settings = Depends(get_settings)) -> dict[str, str]:
|
|
||||||
"""Public app metadata for the frontend brand layer (phase 39)."""
|
|
||||||
return {"app_name": settings.app_name, "version": settings.app_version}
|
|
||||||
```
|
|
||||||
Public (no `require_admin` — the brand must render for anonymous users too, before any sign-in); stateless (A10); the response carries **exactly** these two keys (no other setting may leak in later — the test asserts the key set).
|
|
||||||
2. `app/main.py` — register it with the other routers (`app.include_router(config_router, prefix="/api")`), before the static mount (API routes take precedence — same order as the existing routers).
|
|
||||||
3. Tests (follow the health test's location and pattern): 200 for anonymous; default values (`"Brain of Reese"`, the current `app_version`); with an overridden `Settings` (`app_name="Brain of Testy"`) the response follows; the response key set == `{"app_name", "version"}`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- The tests above; full `uv run pytest` green.
|
|
||||||
- Coverage: **>90%** on the new module (keep the handler + docstring tight so it stays covered).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` (the current default version).
|
|
||||||
- [ ] The suite green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# Task 02 — `brand.js` + de-hard-coding
|
|
||||||
|
|
||||||
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
|
|
||||||
**Story:** `.agent/user_stories/configurable-brand.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Every visible brand string on every page resolves from one place (`window.BOR_BRAND`, fed by `/api/config`) — the default "Brain of Reese" renders immediately, and a fetch failure falls back to the default (the page never breaks).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/brand.js` — a new small **classic** script (vanilla, no CDN — A11; not a module, so its top level runs at parse time):
|
|
||||||
- Top level: `window.BOR_BRAND = "Brain of Reese"` (synchronous default — module scripts execute after parsing, so the page scripts can read it from the first line).
|
|
||||||
- DOM application (deferred: `if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", applyBrand); else applyBrand();`):
|
|
||||||
1. `fetch("/api/config", { cache: "no-store" })`; on success with a non-empty `app_name`: `window.BOR_BRAND = app_name`, then:
|
|
||||||
2. `document.title = document.title.replaceAll("Brain of Reese", name)`.
|
|
||||||
3. Every `.brand-text` node: if the name starts with `"Brain of "` → `innerHTML = 'Brain of <strong>' + escapeHTML(rest) + '</strong>'` (the current look for the default name); else → `textContent = name` (plain, no bold). **HTML-escape the name** (an operator-controlled string must not inject markup).
|
|
||||||
4. A `TreeWalker` over the document's text nodes: replace the literal "Brain of Reese" with the name (catches the index empty-state h1 "Hey! I'm Brain of Reese." and any prose).
|
|
||||||
5. An attribute pass over `aria-label`, `placeholder`, and meta `content` attributes containing the literal → replace (the `#messages` aria-label "Conversation with Brain of Reese", the input label, the meta descriptions).
|
|
||||||
- On fetch failure: keep the default, `console.warn` (the `loadHealth` house style — progressive enhancement, never break the page).
|
|
||||||
- Small local `escapeHTML` helper (the `markdown.js` pattern — do not import across modules unless the build makes it easy).
|
|
||||||
2. The templates (`frontend/index.html`, `sources.html`, `tuning.html`, `document.html`, `login.html`) — add `<script src="assets/brand.js"></script>` **before** the page's module script on every page (classic script → runs at parse time; the module scripts execute later). No other template changes — the walker + attribute pass is the single mechanism; do **not** add `data-brand` markers. The phase-33 `?v=` rewriting picks the new ref up automatically (`app/core/caching.py` matches any `src="…assets/…"`).
|
|
||||||
3. `frontend/assets/app.js` — replace the "Brain of Reese" literals with `window.BOR_BRAND` reads: the `UI_STATE` labels (~L107: "… is thinking" / "… is answering"), `TYPING_LABEL` (~L112), the elapsed-hint aria-label (~L540). Pattern: `const brand = () => window.BOR_BRAND || "Brain of Reese";` + template strings. (Mid-turn staleness: a label set before the fetch lands keeps the old name for that turn — accepted, see the phase's locked decisions.)
|
|
||||||
4. `frontend/assets/document.js` — the page titles (L147/152: `${doc.title} · Brain of Reese` / `"Document not found · …"`) → use the same `window.BOR_BRAND` read (document.js is a module — `window.BOR_BRAND` is set by then).
|
|
||||||
5. `Containerfile` — add the esbuild line for the new file next to the others (L17–23 pattern, classic script like `markdown.js`): `esbuild ./assets/brand.js --minify --outfile=/out/assets/brand.js`.
|
|
||||||
6. Verify the no-op property: with the default settings the rendered DOM text is byte-identical to pre-phase on all five pages (the replace is a no-op for the default name) — the existing suites' title/label assertions are the guard; if any assert a string this task moved onto `window.BOR_BRAND`, the default path must render the identical bytes.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No Python logic — the story E2E (task 03) is the gate; the existing suites (which assert the default "Brain of Reese" titles/labels against the shared conftest server) must stay green **unchanged**.
|
|
||||||
- No CDN (rule 6): no new external tags. UI Structure Check (rule 5): no landmark/contrast change — the brand text keeps its existing classes and styling (the `innerHTML` rewrite only re-emits the same structure with the new name).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `BOR_APP_NAME` unset → all five pages render exactly as today (existing suites green).
|
|
||||||
- [ ] `BOR_APP_NAME="Brain of Testy"` → title/header/greeting/labels/aria all carry the new name (asserted by the story E2E, task 03).
|
|
||||||
- [ ] The Containerfile build includes brand.js; the asset ref is versioned like its siblings (phase 33).
|
|
||||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Task 03 — Story E2E + docs + commit
|
|
||||||
|
|
||||||
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
|
|
||||||
**Story:** `.agent/user_stories/configurable-brand.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The story's isolated Playwright suite against an app instance booted with the overridden name, the env docs, and the phase commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_configurable_brand.py` (the story gate — one story, one file, run in isolation):
|
|
||||||
- A **second app instance** — the shared `app_server` conftest fixture keeps the default name (the other suites' title/label assertions depend on it). Copy the conftest `app_server` env block (same DB, the mock-LLM base URL, `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET`, `BOR_STATIC_DIR`, `BOR_RELEVANCE_THRESHOLD`) with two changes: `BOR_APP_NAME="Brain of Testy"` and a distinct port (`APP_PORT + 1` per the conftest convention). A session-scoped fixture **inside the test file**, started after `mock_llm` is available.
|
|
||||||
- Assertions (custom instance): index `document.title` == `"Brain of Testy"`; the `.brand-text` `innerHTML` == `Brain of <strong>Testy</strong>`; the empty-state h1 text == `"Hey! I'm Brain of Testy."`; the `#messages` `aria-label` == `"Conversation with Brain of Testy"`; the sources page title `"Sources · Brain of Testy"`; the login page title `"Sign in · Brain of Testy"`; one chat turn with a pre-token window (the `think out loud` marker) → the button label shows `"Brain of Testy is thinking"`.
|
|
||||||
- Default-name assertion (cheap regression in the same file): the shared conftest server's index title still == `"Brain of Reese"`.
|
|
||||||
- The chat-turn assertion works on the default (possibly empty) KB — a deflected answer is fine; the label assertion is pre-token, so no DB seeding is required.
|
|
||||||
2. `.env.example` — document `BOR_APP_NAME` in the App section (the display name on all pages; default "Brain of Reese").
|
|
||||||
3. README — the configuration section: `BOR_APP_NAME` (what it affects: titles, the header brand, the status labels, the aria text; the default; the bold-split rendering rule: names starting "Brain of " bold the remainder, any other name renders in normal weight).
|
|
||||||
4. Regression pass: `uv run pytest` + the coverage gate (>90%) + the isolated story E2E + the suites that assert brand strings (`test_smoke.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_chat_persistence.py`) green.
|
|
||||||
5. Commit — one atomic `--no-gpg-sign` Conventional Commits commit for the whole phase (AGENTS.md rule 8), e.g. `feat(brand): configurable app name — BOR_APP_NAME drives /api/config + the frontend brand layer`; move the phase directory to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- The gates above are this task's quality bar (A16: one story, one isolated E2E file, coverage >90%).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The story E2E is green in isolation, deterministic across two consecutive runs (custom-name instance + the default-name assertion).
|
|
||||||
- [ ] The step-4 regression list green; coverage >90%.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# Task 01 — Toggle ships hidden, admin-only reveal
|
|
||||||
|
|
||||||
**Phase:** `40_tuning_toggle_flash` · **Source:** `TODO.md:3` — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
|
|
||||||
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Ship `#steering-toggle` `hidden` in all six pages and unhide it in `header.js` only when whoami says admin — zero flash for anonymous, identical admin UX.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/git-sources.html`, `frontend/login.html`, `frontend/tuning.html` — add the `hidden` attribute to the existing `#steering-toggle` `<button>` (the element that ships `aria-expanded="false" aria-controls="steering-panel"`; keep every other attribute, icon, label, and count badge byte-identical). The `#steering-panel` section already ships `hidden` — do not touch it.
|
|
||||||
2. `frontend/assets/header.js` — in `initSharedHeader()`, in the `if (admin)` branch, add `if (steeringToggle) steeringToggle.hidden = false;` **before** `if (steeringPanel) refreshSteering();`. The anonymous branch (`steeringToggle?.remove(); steeringPanel?.remove();`) stays byte-identical. Update the module docstring: the steering controls are now ship-hidden / reveal-for-admin (2026-08-27, `TODO.md` L3), matching the admin-only nav links.
|
|
||||||
3. `tests/unit/test_steering_toggle_visibility.py` (new) — source pins in the house style (regex/substring over the HTML + JS files, see `tests/unit/test_sync_button.py`):
|
|
||||||
- `#steering-toggle` carries `hidden` in **all six** pages;
|
|
||||||
- `header.js` contains the admin unhide (`steeringToggle.hidden = false`) inside `initSharedHeader`;
|
|
||||||
- the anonymous removal (`steeringToggle?.remove()`) is still present;
|
|
||||||
- `#nav-tuning` still ships `hidden` (the contract this phase relies on).
|
|
||||||
4. Grep the existing suites for exact-markup pins of the toggle (`tests/unit/test_shared_header.py`, `tests/unit/test_steering.py`, `tests/e2e/test_global_tuning.py`, `test_steering.py`) and update any assertion that breaks purely because of the new `hidden` attribute — behavior assertions stay.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the new pin file above; full unit suite green.
|
|
||||||
- Coverage: **>90%** on `app/` (no Python change — TOTAL must be unchanged; run `uv run pytest --cov=app --cov-report=term-missing`).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All six pages ship `#steering-toggle` with `hidden`; `header.js` reveals for admin and still removes for anonymous.
|
|
||||||
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
|
|
||||||
- [ ] No behavior change in completed work (admin steering flow byte-identical: open/close, count badge, delete, announcer).
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
# Task 01 — Server core: round cap replaces the budgets (config + loop + unit/integration)
|
|
||||||
|
|
||||||
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
|
|
||||||
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
One coherent server-side change, landed atomically so the suite is green at the checkpoint: `agent_max_rounds` (`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no tools) replaces both per-tool budgets in config, the agent loop, and every test that pins them.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` —
|
|
||||||
- **delete** the `agent_list_calls` and `agent_read_calls` fields (with docstrings);
|
|
||||||
- **add** in their place, same "RAG tuning" section:
|
|
||||||
```python
|
|
||||||
#: Hard cap on the agent tool rounds per grounded turn (phase 45,
|
|
||||||
#: revising phase 37's per-tool budgets — owner permission
|
|
||||||
#: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
|
|
||||||
#: as it wants"). Every tool call the model emits consumes a
|
|
||||||
#: round; at the cap the loop forces one final no-tools answer.
|
|
||||||
#: ``0`` disables the tools entirely — the turn is a single
|
|
||||||
#: request with ``tools=None`` (the pre-phase-37 path — the kill
|
|
||||||
#: switch).
|
|
||||||
agent_max_rounds: int = 10
|
|
||||||
```
|
|
||||||
- optional: a `field_validator` rejecting negative values (note it in the docstring if added).
|
|
||||||
2. `app/rag/agent.py` —
|
|
||||||
- `run_agent`: `max_rounds = settings.agent_max_rounds`; `tools = AGENT_TOOLS if max_rounds > 0 else None` (the kill switch — at 0 the loop makes exactly one request with `tools=None`, byte-identical to the pre-phase-37 path);
|
|
||||||
- delete `list_left` / `read_left` and the budget-driven `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` transition — `tools` stays `AGENT_TOOLS` while rounds remain;
|
|
||||||
- after each executed call: `rounds += 1`; the existing cap branch becomes the **only** forced-exit: `if rounds >= max_rounds:` → the `logger.warning("agent round cap reached …")` + final `chat_stream(messages, tools=None)` (update the warning text: it is no longer belt-and-braces — it is the cap);
|
|
||||||
- `_execute_tool(db, call, seed_docs, holder)`: drop the `list_left` / `read_left` parameters and the `LIST_EXHAUSTED` / `READ_EXHAUSTED` early returns; keep the `ALREADY_IN_CONTEXT`, `UNKNOWN_TOOL`, `MISSING_READ_ARGS` rejections (non-budget — a repeated rejected call still consumes a *round* in the loop, so a pathological stream is bounded by `max_rounds`); return type simplifies to `str`;
|
|
||||||
- delete the `LIST_EXHAUSTED` / `READ_EXHAUSTED` constants;
|
|
||||||
- `AGENT_TOOLS`: `read_document` description "Add the full content of exactly one more indexed document to your context" → "Add the full content of one more indexed document to your context";
|
|
||||||
- module docstring: the budget paragraph (points 1, 3, 4) rewritten for the round cap (owner revision 2026-08-27, `TODO.md` L8); `run_agent` docstring updated (`seed_docs` note unchanged); `AgentHolder` unchanged (`tool_calls` still counts executed calls — now including re-lists);
|
|
||||||
- the per-call `logger.info("agent tool=… budget list_left=… read_left=…")` line becomes `logger.info("agent tool=%s args=%s round=%d/%d", …)` (or equivalent — the per-turn `tool_calls=N` field in `app/api/chat.py` is untouched).
|
|
||||||
3. `tests/unit/test_agent.py` — **rewrite** the budget tests around the round cap (keep the file's fake-LLM harness):
|
|
||||||
- an always-`list_documents`-calling mock with `agent_max_rounds=3`: exactly 3 tool rounds execute, then one forced `tools=None` request streams the answer; `holder.tool_calls == 3`;
|
|
||||||
- `agent_max_rounds=0`: exactly one request, `tools=None`, no tool lines, `holder.tool_calls == 0` (kill switch);
|
|
||||||
- an always-`read_document`-with-unknown-path mock (every call rejected — `No document at …`): the loop runs to `max_rounds` and forces the final answer (rejections no longer end the loop early via budgets, the cap bounds them);
|
|
||||||
- the existing rejections tests (`Unknown tool.`, `MISSING_READ_ARGS`, `Already in your context.`) keep passing — update their `_settings(...)` calls (`agent_max_rounds=…` instead of the budget kwargs);
|
|
||||||
- a **re-list** test: `list_documents` called twice in one turn executes both (the second returns the catalog again) and counts 2 in `holder.tool_calls`.
|
|
||||||
4. `tests/unit/test_config.py` — default 10; `BOR_AGENT_MAX_ROUNDS=0` / `=5` overrides; (negative validator, if added); delete the old budget assertions.
|
|
||||||
5. `tests/integration/test_chat_api.py` — the `agent_list_calls=0, agent_read_calls=0` fixture kwargs (~line 661) become `agent_max_rounds=0`; any other budget kwarg in the file the same; the tool SSE-event and `done.sources` assertions stay untouched.
|
|
||||||
6. `.env.example` — the two `BOR_AGENT_*_CALLS` lines become one: `# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)` (file's optional-setting comment style).
|
|
||||||
7. Grep the repo for `agent_list_calls|agent_read_calls|BOR_AGENT_(LIST|READ)_CALLS|LIST_EXHAUSTED|READ_EXHAUSTED` — zero hits outside `.agent/phases/complete/**` (history).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit + integration: full `uv run pytest` green at this checkpoint (the mock/E2E multi-read flow lands in task 02 — the existing 3-step mock flow still works unmodified, so `test_agent_document_tools.py` E2E is not yet run by the gate).
|
|
||||||
- Coverage: **>90%** on `app/` — `agent.py` + `config.py` fully covered by the rewritten tests.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] No per-tool budgets anywhere in `app/` or `tests/`; `agent_max_rounds` is the single knob (default 10, `0` = kill switch).
|
|
||||||
- [ ] Re-lists execute; non-budget rejections intact; the cap bounds pathological streams; `tool_calls=N` log field and `tool` SSE event unchanged.
|
|
||||||
- [ ] `uv run pytest` + coverage gate green at this checkpoint.
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
# Task 02 — Mock: deterministic multi-read tool flow
|
|
||||||
|
|
||||||
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
|
|
||||||
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The E2E mock gains a deterministic **multi-read** agent flow (list → read #1 → read #2 → answer) so "as many tool calls as it wants" is provable statelessly, without disturbing the existing 3-step flow.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/mock_llm.py` —
|
|
||||||
- the existing phase-37 flow (documented in the module docstring and `_tool_flow`): marker `TOOLS_TRIGGER` ("use your tools") + `<tools>` system section → step classification **statelessly from the messages**: no tool results yet → `list`; one `tool`-role message with the catalog prefix → `read` (first catalog doc, parsed from the listing via the `rsplit("/", 1)` convention); one `tool`-role message with the `Document <source/path>:` prefix → forced answer.
|
|
||||||
- add a **multi-read variant**: when the user message contains **both** `TOOLS_TRIGGER` and a new marker `MULTI_READ_TRIGGER = "read two documents"`, the classifier reads the *count* of `tool`-role messages whose content starts with `"Document "` (the read-result prefix, `app.rag.agent`'s `_execute_tool` output):
|
|
||||||
- 0 read results (+ no catalog yet) → `list`;
|
|
||||||
- 0 read results (catalog present) → `read` the **first** catalog doc;
|
|
||||||
- 1 read result → `read` the **second** catalog doc (the listing minus the already-read doc — parse the catalog lines the same way the existing read step does, skipping the path already read);
|
|
||||||
- 2 read results → forced answer: the existing answer shape (tail echo) plus a deterministic line naming **both** read paths (e.g. `"I read <path1> and <path2>."` — byte-stable) so the E2E can assert the model actually used both;
|
|
||||||
- the single-read flow (no `MULTI_READ_TRIGGER`) stays byte-identical — the variant must be a strict superset (the existing `test_agent_document_tools.py` E2E keeps passing unmodified).
|
|
||||||
- update the module docstring's tool-flow documentation (the multi-read steps + the marker).
|
|
||||||
2. `uv run pytest` green (mock-only change; the existing 3-step E2E is not run by the unit gate but must stay conceptually intact — the regression run in task 03 proves it).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: full suite green; if a mock-specific unit test file exists (check `tests/unit/`), add the multi-read classification case there (catalog → read #1 → read #2 → answer) so the new branch is unit-covered; otherwise the E2E (task 03) covers it.
|
|
||||||
- Coverage: **>90%** on `app/` (unchanged — `tests/` only).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `TOOLS_TRIGGER` + `MULTI_READ_TRIGGER` → deterministic 4-step flow (list, read #1, read #2, answer naming both paths); the 3-step flow is unchanged for marker-less turns.
|
|
||||||
- [ ] Full unit/integration suite green.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Phase 46 — Mobile hamburger nav
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L9 — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
|
|
||||||
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
|
|
||||||
**Context:** The shared bar (phase 19/34 — the same header block on all six pages) carries brand + up to four text nav pills (Chat, `#nav-sources`, `#nav-git-sources`, `#nav-tuning` — the latter three ship `hidden`, revealed for admin by `header.js`) + four action controls (steering toggle, `#sync-btn`, `#new-chat-btn`, sign in/out). At ≤640px the phase-34/35 squeeze rules (0.72rem pills, 0.05rem gaps) leave a bar that is squished and hard to hit. The fix: on mobile the nav links move into an animated hamburger dropdown; the action pills stay in the bar.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
At ≤640px the nav links live in a `#nav-toggle`-opened dropdown menu (slide+fade, reduced-motion-still) with comfortable targets and the same auth visibility; at >640px the bar is byte-identical to today.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `45_agent_unlimited_tools` (todo) — sequential only (no shared files).
|
|
||||||
- `34_consistent_navbar` / `35_git_sources_admin` (complete) — the six-page bar contract, the admin-only link reveal, and the mobile squeeze rules being superseded.
|
|
||||||
- `07_story_responsive_polish` (complete) — the ≤640px conventions (44px targets, safe areas).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_hamburger_markup_all_pages.md` — `#nav-toggle` + `id="app-nav"` on all six pages; the mobile CSS (dropdown, animation, reduced-motion).
|
|
||||||
2. `02_toggle_behavior.md` — `header.js` open/close behavior (aria, Esc, link-close, resize-close) + source pins.
|
|
||||||
3. `03_hamburger_e2e_and_commit.md` — story E2E suite (mobile + desktop + reduced motion) + regressions + commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: new `tests/unit/test_hamburger_nav.py` — `#nav-toggle` (with `aria-controls="app-nav"`, `aria-expanded`, `aria-label="Menu"`) present in **all six** pages and absent-visible on desktop (CSS `display: none` outside the media query); `<nav class="app-nav" id="app-nav">` in all six; the mobile CSS block carries the dropdown rules + `.is-open` state + the 180ms transition + the reduced-motion override; the old nav-pill squeeze rules are gone/superseded; `header.js` carries the toggle binding (click, Esc, delegated link close, matchMedia close).
|
|
||||||
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_mobile_hamburger_nav.py`, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] 375px: hamburger visible (44px target), inline nav hidden, no horizontal bar overflow; menu opens with animation, closes via link/Esc/outside; anonymous sees only "Chat" in the menu, admin sees all four links.
|
|
||||||
- [ ] `reducedMotion: "reduce"`: no transition, open/close still instant and correct.
|
|
||||||
- [ ] >640px: no hamburger, inline pills exactly as today (phase-34/35 contract intact).
|
|
||||||
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation (DB up).
|
|
||||||
- [ ] Regression E2E suites green in isolation: `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): the toggle is a labeled 44px button with `aria-expanded`/`aria-controls`; the menu keeps `nav aria-label="Primary"`; focus-visible on the new control; contrast ≥4.5:1; no CDN.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **Owner-locked (2026-08-27, roadmap A5):** nav links only in the menu; action pills stay in the bar; slide-down + fade 180ms; `prefers-reduced-motion` stills it; the existing ≤640px breakpoint (no new one).
|
|
||||||
- **A11 untouched** — no new assets/CDN; the menu reuses the existing `<nav>` element (no duplicated links, so the whoami reveal keeps working unchanged).
|
|
||||||
- **Phase-34 contract kept** — the same bar on every page; the auth visibility rules apply inside the menu exactly as before.
|
|
||||||
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
# Task 01 — Hamburger markup (six pages) + mobile dropdown CSS
|
|
||||||
|
|
||||||
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
|
|
||||||
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
All six pages carry the identical `#nav-toggle` button + `id="app-nav"`, and the ≤640px stylesheet turns the nav into an animated full-width dropdown — desktop untouched.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **Markup — all six pages** (`frontend/index.html`, `sources.html`, `document.html`, `git-sources.html`, `login.html`, `tuning.html`), each in its `<header class="app-header">` block:
|
|
||||||
- insert the toggle button **immediately before** the `<nav>`:
|
|
||||||
```html
|
|
||||||
<!-- Phase 46 (owner permission 2026-08-27, `TODO.md` L9): the
|
|
||||||
mobile hamburger — visible ≤640px only (CSS); opens the nav as
|
|
||||||
an animated dropdown. Behavior: assets/header.js. -->
|
|
||||||
<button type="button" class="nav-toggle" id="nav-toggle"
|
|
||||||
aria-expanded="false" aria-controls="app-nav" aria-label="Menu">
|
|
||||||
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M4 7h16M4 12h16M4 17h16"/></svg>
|
|
||||||
</button>
|
|
||||||
```
|
|
||||||
- give the existing nav the id: `<nav class="app-nav" id="app-nav" aria-label="Primary">` (every other attribute/child byte-identical — the links keep their `hidden` attributes; **no links duplicated**).
|
|
||||||
- keep the six pages visually/structurally identical (the phase-34 contract) — the toggle is part of the shared bar block, positioned the same on every page.
|
|
||||||
2. `frontend/assets/styles.css` —
|
|
||||||
- **global (outside media queries):** `.nav-toggle { display: none; }` (desktop: absent);
|
|
||||||
- **inside the existing `@media (max-width: 640px)` block:**
|
|
||||||
- `.nav-toggle { display: inline-flex; align-items: center; justify-content: center; width: 44px; height: 44px; padding: 0; color: var(--ink); background: none; border: 0; border-radius: var(--radius-sm); cursor: pointer; }` + `:focus-visible` inherits the global 3px outline + a hover state matching the other pills (the `.steering-toggle:hover` family);
|
|
||||||
- `.app-nav` becomes the dropdown: `position: absolute; top: 100%; left: 0; right: 0; flex-direction: column; gap: 0; background: var(--surface); border-bottom: 1px solid var(--line); box-shadow: <existing shadow token or 0 8px 24px rgba(0,0,0,.4)>; padding: 0.5rem 0; z-index: <above the header content — check the header's z-index and use header+1>;` — note the containing block is the sticky `.app-header` (`.header-inner` is not positioned), so the menu spans the header's full width, edge to edge — intended on mobile;
|
|
||||||
- **closed state (default):** `visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none; transition: opacity 180ms ease, transform 180ms ease, visibility 0s linear 180ms;`
|
|
||||||
- **open state:** `.app-nav.is-open { visibility: visible; opacity: 1; transform: none; pointer-events: auto; transition: opacity 180ms ease, transform 180ms ease, visibility 0s; }`
|
|
||||||
- **menu rows:** `.app-nav .nav-link { padding: 0.75rem 1.25rem; font-size: 1rem; }` (comfortable 44px+ targets, readable — this **supersedes** the ≤640px pill-squeeze rules for `.nav-link` and `.app-nav` gap in that block: delete/replace the `.nav-link { padding: 0.3rem 0.25rem; font-size: 0.72rem; }` and `.app-nav { gap: 0.05rem; }` rules, keeping the rest of the block);
|
|
||||||
- **reduced motion:** inside the file's existing `@media (prefers-reduced-motion: reduce)` block (the one covering the 640px rules — or a new one after it): `.app-nav { transition: none; }`;
|
|
||||||
- the **900px tablet block is untouched** (inline nav still in use at 641–900px); the action-pill rules in the 640px block are untouched; the header height (`--header-h: 58px`) is untouched.
|
|
||||||
- verify at 360px: brand (clipped clean as today) + hamburger + the four icon action pills fit without horizontal overflow (the old four text pills are gone from the bar — there is now room; if the bar still overflows, the brand ellipsis target absorbs it exactly as before).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the new `tests/unit/test_hamburger_nav.py` starts here with the markup + CSS pins (the JS pins land in task 02): toggle markup (attributes) in all six pages; `id="app-nav"` in all six; `.nav-toggle { display: none }` outside media queries; the mobile block carries the dropdown, `.is-open`, the 180ms transition pair, the reduced-motion override, and the superseded squeeze rules are gone; full suite green (behavior not yet wired — the menu is closed by default and CSS-inert without JS, so no E2E regressions at this checkpoint).
|
|
||||||
- Coverage: **>90%** on `app/` (unchanged).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Six identical toggles + `id="app-nav"`; desktop rendering byte-identical (`.nav-toggle` hidden, nav inline as before).
|
|
||||||
- [ ] Mobile: closed dropdown is invisible and non-interactive; `.is-open` (added by task 02's JS) will be the only opener.
|
|
||||||
- [ ] Full suite green at this checkpoint.
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
# Task 02 — Toggle behavior in the shared header module
|
|
||||||
|
|
||||||
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
|
|
||||||
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
The menu opens and closes with correct ARIA state, Esc/link/outside dismissal, and desktop-resize cleanup — one module-owned binding, like the existing sign-out/steering bindings in `header.js`.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/assets/header.js` — a new module-import binding (same pattern as the sign-out binding: look up at import, guard null-safe, no page-script involvement):
|
|
||||||
```js
|
|
||||||
/* ---------- mobile hamburger (phase 46; module-owned) ----------
|
|
||||||
* ≤640px only (CSS hides the button elsewhere): #nav-toggle opens the
|
|
||||||
* nav as a dropdown (#app-nav .is-open — the animated state, task 01
|
|
||||||
* CSS). One binding for all six pages; a page without either element
|
|
||||||
* is a no-op, like the rest of this module. The nav LINKS keep their
|
|
||||||
* ship-hidden whoami contract (hidden links stay hidden inside the
|
|
||||||
* menu) — this binding only toggles the container. */
|
|
||||||
const navToggle = document.querySelector("#nav-toggle");
|
|
||||||
const appNav = document.querySelector("#app-nav");
|
|
||||||
|
|
||||||
function setNavMenu(open) {
|
|
||||||
if (!appNav || !navToggle) return;
|
|
||||||
appNav.classList.toggle("is-open", open);
|
|
||||||
navToggle.setAttribute("aria-expanded", open ? "true" : "false");
|
|
||||||
}
|
|
||||||
|
|
||||||
if (navToggle && appNav) {
|
|
||||||
navToggle.addEventListener("click", () =>
|
|
||||||
setNavMenu(!appNav.classList.contains("is-open")));
|
|
||||||
// A link click navigates (or closes same-page) — shut the menu.
|
|
||||||
appNav.addEventListener("click", (e) => {
|
|
||||||
if (e.target.closest("a")) setNavMenu(false);
|
|
||||||
});
|
|
||||||
// Esc closes while open (document-level; the menu is the only
|
|
||||||
// document-level overlay this module owns).
|
|
||||||
document.addEventListener("keydown", (e) => {
|
|
||||||
if (e.key === "Escape" && appNav.classList.contains("is-open")) {
|
|
||||||
setNavMenu(false);
|
|
||||||
navToggle.focus(); // focus returns to the opener
|
|
||||||
}
|
|
||||||
});
|
|
||||||
// Resize back to desktop: the inline nav reappears — no stale open
|
|
||||||
// state (the .is-open class is scoped by the ≤640px CSS anyway, but
|
|
||||||
// dropping it keeps aria-expanded honest).
|
|
||||||
const mq = window.matchMedia("(max-width: 640px)");
|
|
||||||
const onMqChange = () => { if (!mq.matches) setNavMenu(false); };
|
|
||||||
if (mq.addEventListener) mq.addEventListener("change", onMqChange);
|
|
||||||
else mq.addListener(onMqChange); // older engines, defensive
|
|
||||||
}
|
|
||||||
```
|
|
||||||
- update the module docstring: add the hamburger bullet (phase 46, owner permission 2026-08-27, `TODO.md` L9).
|
|
||||||
2. `tests/unit/test_hamburger_nav.py` — extend (from task 01) with the JS pins: `header.js` contains the `#nav-toggle` binding, `setNavMenu` (or equivalent) syncing **both** `.is-open` and `aria-expanded`, the delegated `a`-click close, the `Escape` close (with focus return), and the `matchMedia("(max-width: 640px)")` change-close; assert the binding is null-safe (`navToggle && appNav` guard).
|
|
||||||
3. `uv run pytest` green at this checkpoint.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: the extended pin file; full suite green.
|
|
||||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] One module-owned binding: click toggles (aria-expanded in sync), Esc closes + refocuses the toggle, a link click closes, desktop resize closes; pages lacking the elements are a no-op.
|
|
||||||
- [ ] The auth visibility contract is untouched — the binding toggles the container only; `hidden` links stay hidden.
|
|
||||||
- [ ] Full suite green at this checkpoint.
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
# Task 03 — Hamburger E2E + regressions + commit
|
|
||||||
|
|
||||||
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
|
|
||||||
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the full mobile contract in the browser (visibility, contents per auth state, navigation, dismissal, animation, reduced motion, desktop regression) and commit the phase.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_mobile_hamburger_nav.py` (new) — mock-only, DB up. Mobile tests use a 375×812 page (new page per test, or `page.set_viewport_size` — the conftest `page` fixture is 1280×800, so create mobile pages via the `browser` fixture); per the story's Playwright Mapping Rule:
|
|
||||||
- `test_mobile_hamburger_visible_and_bar_roomy` — 375px: `#nav-toggle` visible (box ≥44px in both dimensions), `aria-expanded="false"`, the inline nav links are **not** visible in the bar (menu closed — bounding boxes outside the header band or opacity 0), and no horizontal overflow (`document.documentElement.scrollWidth <= window.innerWidth`);
|
|
||||||
- `test_anonymous_menu_contents` — anonymous at 375px: click `#nav-toggle` → `aria-expanded="true"`, exactly **one** visible link in `#app-nav` ("Chat"); `#nav-sources` / `#nav-git-sources` / `#nav-tuning` remain `hidden` inside the menu;
|
|
||||||
- `test_admin_menu_contents` — login (e2e.auth_helpers) at 375px: open → Chat / Sources / Git sources / Tuning all visible (the whoami reveal works inside the menu);
|
|
||||||
- `test_link_click_navigates_and_closes` — admin at 375px: open, click "Sources" → URL becomes `/sources.html` and on the arrival page the menu is closed (`aria-expanded="false"`, no `.is-open`);
|
|
||||||
- `test_esc_and_outside_close` — open, press `Escape` → closed **and** focus is back on `#nav-toggle`; open again, click a neutral point (e.g. the page footer/main) → closed. (If the outside-click close is not implemented per task 02's contract — it is not: only Esc/link/resize close — assert instead that the menu stays open on an outside click and **note the accepted behavior** in the test docstring; the story's AC 4 lists Esc + link + resize, not backdrop click. Do NOT add a backdrop-close — it is out of the locked scope.)
|
|
||||||
- `test_animation_and_reduced_motion` — motion allowed: computed `transition-duration` on `#app-nav` includes `0.18s` (opacity/transform pair); open → the class/aria flip; `reducedMotion: "reduce"` (new context via the `browser` fixture): computed transition is `none`/`0s` and open/close still works;
|
|
||||||
- `test_desktop_unchanged` (regression) — 1280×800: `#nav-toggle` not visible (`display: none`), the inline nav renders in the bar exactly as before (admin: all four links visible inline).
|
|
||||||
2. Regression pass (isolation runs): `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
|
|
||||||
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(header): hamburger dropdown nav on mobile (owner permission)`, staging this phase's files; move `.agent/phases/todo/46_mobile_hamburger_nav/` → `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation.
|
|
||||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The story E2E suite passes in isolation (all seven tests); the five regression suites pass in isolation.
|
|
||||||
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Task 01 — Config: the ten new formats (allowed + default)
|
|
||||||
|
|
||||||
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
|
|
||||||
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
`app/config.py` allows and imports the ten new formats by default; the env validator keeps rejecting truly unknown extensions.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` —
|
|
||||||
- `_ALLOWED_IMPORT_EXTENSIONS`: add `"container", "network", "volume", "image", "pod", "kube", "swap", "os", "endpoint", "j2"` (with a comment: A9 revised 2026-08-27, owner permission — the full Podman quadlet family + Jinja templates);
|
|
||||||
- `import_extensions` default: `"md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2"` (the original seven first, the ten appended — order is cosmetic, the set is what matters; keep the field's docstring and the `mode="after"` validator **unchanged** — it already normalizes/lowercases/dedups and rejects unknowns, so it accepts the new names automatically);
|
|
||||||
- update the module-level comment on `_ALLOWED_IMPORT_EXTENSIONS` (it cites A9 revised 2026-08-21 — append the 2026-08-27 revision).
|
|
||||||
2. `.env.example` — the commented `BOR_IMPORT_EXTENSIONS` line updates to the new default CSV (it currently documents the old default).
|
|
||||||
3. `tests/unit/test_config.py` —
|
|
||||||
- the allowed set contains all seventeen formats;
|
|
||||||
- the default `import_extensions` includes the ten new names (assert each);
|
|
||||||
- `import_extension_set` returns the dotted lowercased set (`.container`, `.j2`, …);
|
|
||||||
- the validator **accepts** a `BOR_IMPORT_EXTENSIONS` containing the new names (e.g. `md,container,j2`) and **still rejects** an unknown one (e.g. `md,xyz`) — the never-widen contract with the widened base set.
|
|
||||||
4. `uv run pytest tests/unit/test_config.py -v` green; full unit suite green (the chunker/importer don't know the new suffixes yet — task 02; no test at this checkpoint imports a new-format file).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: as above; coverage **>90%** on `app/` (config covered).
|
|
||||||
- No behavior change for existing formats (the default CSV only grows — every previously-imported file still matches).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Default settings import all seventeen formats; env narrowing/widening rules behave (new names allowed, unknowns rejected).
|
|
||||||
- [ ] `.env.example` documents the new default.
|
|
||||||
- [ ] Full suite green at this checkpoint.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# Task 02 — Chunker dispatch + fixture files
|
|
||||||
|
|
||||||
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
|
|
||||||
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Every new suffix dispatches to plain-text paragraph packing (`chunk_text`), and the fixture tree carries realistic quadlet + jinja files for the unit/integration/E2E layers.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/rag/chunker.py` —
|
|
||||||
- `_FORMAT_CHUNKERS`: add the ten entries (owner decision R1: plain-text packing — no TOML/Jinja-aware splitter):
|
|
||||||
```python
|
|
||||||
".container": chunk_text, ".network": chunk_text, ".volume": chunk_text,
|
|
||||||
".image": chunk_text, ".pod": chunk_text, ".kube": chunk_text,
|
|
||||||
".swap": chunk_text, ".os": chunk_text, ".endpoint": chunk_text,
|
|
||||||
".j2": chunk_text,
|
|
||||||
```
|
|
||||||
- update the `_FORMAT_CHUNKERS` comment (currently "A9, revised: md, markdown, txt, yaml, yml, json, py") and the module docstring's per-format list (add: "**container / network / volume / image / pod / kube / swap / os / endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and Jinja templates; plain-text paragraph packing (`chunk_text`) — no format-specific splitter (owner decision).").
|
|
||||||
- `chunk_document`'s unknown-suffix fallback stays as-is (belt-and-braces).
|
|
||||||
2. **Fixture files** (under `tests/fixtures/docs/homelab/` — the tree the import E2E imports; hidden dirs are skipped, so no dot-dirs):
|
|
||||||
- `quadlet/compose.container` — realistic quadlet TOML (≥ ~1 500 chars to exercise sub-splitting past a single paragraph pack): `[Unit]` (Description/Wants), `[Service]` (Restart=always), `[Container]` (Image, Ports, Environment, Network, Volume mounts, Exec), comments. Include a unique sentinel token on its own line, e.g. `# RESE-QUADLET-SENTINEL-77aa`.
|
|
||||||
- `quadlet/lan.network` — small: `[Unit]` + `[Network]` (Driver=bridge, IPAMDriver, Subnets) + sentinel `# RESE-NETWORK-SENTINEL-11bb`.
|
|
||||||
- `quadlet/cache.volume` — small: `[Unit]` + `[Volume]` (Driver, Device) + sentinel `# RESE-VOLUME-SENTINEL-22cc`.
|
|
||||||
- `templates/deploy.j2` — a Jinja snippet with `{% for %}` / `{{ var }}` / `{# comment #}` constructs (realistic: an ansible-style service template) + sentinel `RESE-JINJA-SENTINEL-33dd` (no `#` prefix — it lives in a `{% set %}` line or a comment the importer keeps).
|
|
||||||
- keep the existing fixture files byte-identical (other E2E suites import this tree — `test_import_documents.py` asserts exact chunk counts: **run that suite's expectations check**: the tree grew by 4 files, so the phase-02 import E2E's document/chunk count assertions will change — update `tests/e2e/test_import_documents.py`'s count constants in this task, or fold that update into task 04's E2E work; whichever you choose, the full E2E regression pass in task 04 must be green. Prefer updating the constants here so task 03's integration test and task 04 share the same fixture state.)
|
|
||||||
3. `tests/unit/test_chunker.py` —
|
|
||||||
- dispatch: for **each** of the ten suffixes, `chunk_document(content, "x/<name>.<suffix>")` produces the same chunks as `chunk_text(content, …)` (parametrize over the suffix list);
|
|
||||||
- the `.container` fixture (read the file in the test, house pattern) chunks into ≥2 chunks, every chunk ≤ `HARD_MAX_CHARS`, and the sentinel token survives in some chunk;
|
|
||||||
- the `.j2` fixture chunks; Jinja braces are just text (no special handling — assert a `{{` line appears verbatim in a chunk);
|
|
||||||
- the unknown-suffix fallback is unchanged (a `.whatever` file still chunk-paragraphs).
|
|
||||||
4. `uv run pytest tests/unit/test_chunker.py -v` green; full unit suite green.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: as above; coverage **>90%** on `app/` (chunker dispatch covered).
|
|
||||||
- No behavior change for the seven original formats (their chunker bindings are untouched).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Ten dispatch entries; docstrings/comments cite the A9 revision; fixtures exist with their sentinels and the ≥1 200-char container file.
|
|
||||||
- [ ] `test_import_documents.py` count constants updated (or explicitly deferred to task 04 — state the choice in the task's completion note).
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
# Task 03 — Importer parity: walk, delta, prune, titles
|
|
||||||
|
|
||||||
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
|
|
||||||
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the new formats ride the existing import machinery unchanged: the default walk picks them up, delta detection re-imports on change, prune drops them on removal, and titles fall back to the file stem.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/unit/test_importer.py` — extend (house style — the walk function `iter_importable_files` is pure and unit-tested against temp trees):
|
|
||||||
- a temp tree containing one file per new extension (all ten) + one unknown (`.xyz`) + one hidden-dir file (`.esphome/x.container`) + one exclusion (`node_modules/y.container`): the default-extensions walk returns exactly the ten new files — unknown/hidden/excluded filtered;
|
|
||||||
- the seven original extensions still walk (regression in the same test);
|
|
||||||
- title extraction: a `.container` file with no H1 gets the stem title (`extract_title` fallback — via the importer's title path, whatever the house test asserts titles through).
|
|
||||||
2. `tests/integration/` — a new `tests/integration/test_import_quadlet_jinja.py` (or extend `test_importer_e2e.py` if that file's harness fits better — choose and note):
|
|
||||||
- build a temp source dir with a `.container`, a `.volume`, and a `.j2` file (reuse the fixture files' content or small inline variants);
|
|
||||||
- run `import_sources([dir], fake_llm, prune=False)` with the house fake (`tests/fakes.py::FakeEmbedder` — it already implements `embed` + `chat`; give it an `embed_one` delegate if the import path calls one — check the `Embedder` protocol in `app/rag/importer.py` and satisfy exactly what it names):
|
|
||||||
- the three docs land in `documents` (source/path/title — stem titles) with non-zero `chunks` rows;
|
|
||||||
- **delta:** re-run with the `.j2` file's content changed → that doc `updated` (hash changed), the others `unchanged`;
|
|
||||||
- **prune:** delete the `.volume` file, re-run with `prune=True` → pruned count 1, the row gone, its chunks cascade-deleted.
|
|
||||||
- use the existing DB integration harness (the conftest app/db fixtures in `tests/conftest.py` — same pattern as `test_importer_e2e.py`).
|
|
||||||
3. `uv run pytest tests/unit/test_importer.py tests/integration/test_import_quadlet_jinja.py -v` green; full suite green (DB up for the integration part: `podman compose up -d db`).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit + integration as above; coverage **>90%** on `app/` (the importer is unchanged code — the new coverage comes from exercising it with the new formats; if TOTAL dips from task 01's config growth, add asserts — but no `app/` code change is expected in this task).
|
|
||||||
- No `app/` code change expected: if a gap is found (e.g. the walk already accepts any dotted suffix and the config set was the only gate), record that in the task completion note — the tests then prove the gate's location.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Default walk indexes the ten new formats; hidden-dir/exclusion/unknown filtering unchanged; stem titles.
|
|
||||||
- [ ] Delta + prune parity for the new formats (integration).
|
|
||||||
- [ ] Full suite green at this checkpoint.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Phase 48 — Nav rename: "Sources" → "RAG", "Git sources" → "Sources"
|
|
||||||
|
|
||||||
**Source:** owner request (chat, 2026-08-28) — "The 'Git sources' navbar item should be renamed to 'Sources' and the 'Sources' navbar item should be renamed to 'RAG'."
|
|
||||||
**Story:** `.agent/user_stories/nav-sources-rag-rename.md`
|
|
||||||
**Context:** the shared header (phase 19, `frontend/assets/header.js`) ships ONE nav on all six pages — `[Chat, #nav-sources "Sources" → /sources.html, #nav-git-sources "Git sources" → /git-sources.html, #nav-tuning "Tuning"]` — with the two admin-only links ship-hidden and revealed by id once `/api/whoami` says admin (phase 16 contract, phase 34 same-header-everywhere, phase 46 mobile hamburger). Both labels are plain `<a>` text in the six page files; `header.js` toggles only the `hidden` attribute and never reads the label, so the rename is markup-only.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Relabel the two admin-only nav items across all six pages — the document-catalog link (`#nav-sources`) becomes **"RAG"** and the source-manager link (`#nav-git-sources`) becomes **"Sources"** — with ids, hrefs, order, the ship-hidden/reveal contract, and every other page control (viewer back button, Sync button, page titles/h1s) unchanged.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `34_consistent_navbar` / `19_shared_header` (complete) — the same full header block on all six pages and the `header.js` reveal-by-id contract this phase preserves.
|
|
||||||
- `35_git_sources_admin` (complete) — the `#nav-git-sources` link (and the `/git-sources.html` page) it labels.
|
|
||||||
- `46_mobile_hamburger_nav` (complete) — the `LINK_TEXTS` assertion in `test_mobile_hamburger_nav.py` that this phase updates.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_nav_label_swap.md` — swap the two label texts in all six page files + the label mentions in `header.js`'s docstring/comments (no logic).
|
|
||||||
2. `02_existing_test_labels.md` — update the two test suites that assert the old label text; leave every other assertion (viewer back button, sync label, h1/title markers) untouched.
|
|
||||||
3. `03_story_e2e_regression_commit.md` — the story E2E (`test_nav_rename_sources.py`), the regression suites in isolation, ruff + pyright, the one `--no-gpg-sign` commit, and the phase-dir move.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit/integration: no new backend logic (frontend markup only). The existing no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) must still pass — the swap touches no tags or origins (the `/git-sources.html` "Git sources" marker keeps matching the page's unchanged `<h1>`).
|
|
||||||
- Coverage: frontend-only; the `app/` >90% gate is unaffected (unchanged).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_nav_rename_sources.py` — the story gate, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All six pages show **"RAG"** on `#nav-sources` (href `/sources.html`) and **"Sources"** on `#nav-git-sources` (href `/git-sources.html`); nav order and `is-active`/`aria-current` placement unchanged; no id/href/`hidden` change.
|
|
||||||
- [ ] Anonymous still sees neither link; the admin sees both (contract preserved, proven by the story E2E + `test_shared_header.py`).
|
|
||||||
- [ ] `uv run pytest` green (the two updated suites included); `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (no backend change).
|
|
||||||
- [ ] Regression E2E suites green in isolation: `test_nav_consistency.py`, `test_shared_header.py`, `test_mobile_hamburger_nav.py`, `test_git_sources_admin.py`, `test_header_consistency.py`, `test_sources_midstream_bug.py`, `test_smoke.py`.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_nav_rename_sources.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean (no Python behavior change, but run the gate).
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): labeled links, landmarks/contrast/focus unchanged; no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/48_nav_rename_sources/` moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **Label-only rename (owner request 2026-08-28)** — exactly the two nav item texts change. Element ids (`#nav-sources`, `#nav-git-sources`), hrefs, nav order, the ship-hidden/reveal contract, `header.js` behavior, and every other label (the document viewer's "Sources" back button, the "Sync sources" button, page `<title>`/`<h1>` — incl. "Knowledge base" and "Git sources") are OUT of scope. If the owner later wants the page titles/h1s to follow the nav labels, that is a follow-up phase, not this one.
|
|
||||||
- **A11 untouched** — vanilla HTML only, no CDN, no new packages, no new tags.
|
|
||||||
- **A10 untouched** — no endpoint, auth, or `header.js` logic change; the rename rides the existing reveal-by-id path.
|
|
||||||
- **No schema / migration** — purely a markup + test-label change.
|
|
||||||
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# Task 01 — Swap the two nav label texts on all six pages
|
|
||||||
|
|
||||||
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Rename the nav items themselves: `#nav-sources` ("Sources" → **"RAG"**) and `#nav-git-sources` ("Git sources" → **"Sources"") in the shared header markup of all six pages, plus the label mentions in `header.js`'s doc comments. Markup only — no ids, hrefs, classes, logic, or other text changes.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
On **each** of the six page files — `frontend/index.html`, `frontend/sources.html`, `frontend/git-sources.html`, `frontend/tuning.html`, `frontend/document.html`, `frontend/login.html` — make exactly two text swaps inside `<nav class="app-nav" aria-label="Primary">`:
|
|
||||||
|
|
||||||
1. The catalog link — change the visible text only:
|
|
||||||
```html
|
|
||||||
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
|
|
||||||
```
|
|
||||||
becomes
|
|
||||||
```html
|
|
||||||
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>RAG</a>
|
|
||||||
```
|
|
||||||
(On `sources.html` the link carries `class="nav-link is-active" aria-current="page"` — keep those attributes exactly as they are; swap only `Sources` → `RAG`.)
|
|
||||||
|
|
||||||
2. The manager link — change the visible text only:
|
|
||||||
```html
|
|
||||||
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
|
|
||||||
```
|
|
||||||
becomes
|
|
||||||
```html
|
|
||||||
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Sources</a>
|
|
||||||
```
|
|
||||||
(On `git-sources.html` the link carries `class="nav-link is-active" aria-current="page"` — keep them; swap only `Git sources` → `Sources`.)
|
|
||||||
|
|
||||||
3. `frontend/assets/header.js` — **comments only** (the reveal logic is id-based and must stay byte-identical): in the module docstring (the nav-link inventory: `"Sources" (#nav-sources, phase 19), "Git sources" (#nav-git-sources, phase 35)`) and in the inline comment above the `navGitSources` reveal, update the quoted labels to the new ones (`"RAG" (#nav-sources)`, `"Sources" (#nav-git-sources)`). No executable line of `header.js` changes.
|
|
||||||
|
|
||||||
Rules for all edits:
|
|
||||||
- Do **not** touch element ids, `href`s, `hidden` defaults, `class` attributes, `aria-current`, indentation, or any other text on the page (page `<title>`, `<h1>`, the viewer's "Sources" back button, the "Sync sources" button label — all stay).
|
|
||||||
- Do **not** add `is-active` anywhere or move the links — the physical nav order stays Chat, `#nav-sources`, `#nav-git-sources`, `#nav-tuning` (the labels just swap, so it now *reads* Chat, RAG, Sources, Tuning).
|
|
||||||
- The phase-16/19/35 ship-hidden contract is untouched: the links still ship `hidden` and `header.js` still reveals them by id for admins.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No new unit/integration logic (frontend markup only).
|
|
||||||
- Coverage: frontend-only; `app/` coverage unaffected.
|
|
||||||
- The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) still passes — no tags or origins change; its `("/git-sources.html", "Git sources")` marker still matches that page's unchanged `<h1>Git sources</h1>`.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] All six pages: `#nav-sources` renders text **RAG** and `#nav-git-sources` renders text **Sources**; ids/hrefs/`hidden`/`is-active` unchanged (a quick `grep -c 'id="nav-sources" hidden>RAG<'` per page returns 1, and `grep -c 'id="nav-git-sources" .*>Sources<'` returns 1).
|
|
||||||
- [ ] `git diff --stat` shows only the six HTML files + `frontend/assets/header.js` (comment-only) changed.
|
|
||||||
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# Task 02 — Update the tests that assert the old nav label text
|
|
||||||
|
|
||||||
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Keep the existing suites honest after the rename: update the (few) assertions and label-describing comments that reference the OLD label texts, without touching any assertion that is about a different control.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
Only these tests assert the old label text (verified by grepping `tests/` for the strings) — update exactly these:
|
|
||||||
|
|
||||||
1. `tests/e2e/test_git_sources_admin.py` — `test_admin_nav_link_on_all_five_pages_and_click_navigates`:
|
|
||||||
- the comment `pointing at the git sources page, labeled "Git sources" — and it` → `labeled "Sources"`;
|
|
||||||
- `expect(link).to_have_text("Git sources")` → `expect(link).to_have_text("Sources")`.
|
|
||||||
- This suite has no assertion on `#nav-sources`'s TEXT (it is id-based everywhere) — do not add one here; the story E2E (task 03) owns the label checks.
|
|
||||||
2. `tests/e2e/test_mobile_hamburger_nav.py` —
|
|
||||||
- `LINK_TEXTS = ("Chat", "Sources", "Git sources", "Tuning")` → `LINK_TEXTS = ("Chat", "RAG", "Sources", "Tuning")` (DOM order unchanged — only the labels swapped);
|
|
||||||
- the comment near `test_…` that reads `Chat / Sources / Git sources / Tuning — i.e. the whoami reveal` → `Chat / RAG / Sources / Tuning …`.
|
|
||||||
3. **Comments-only drift fixes** (no assertion changes) where a suite's docstring/comment quotes the old labels as the link's identity: `tests/e2e/test_shared_header.py` (module docstring line `"the 'Sources' nav link (#nav-sources)"` → `'RAG'`), `tests/e2e/test_nav_consistency.py` (the comment `the Git sources link joined in phase 35` → mention the relabeled link), `tests/e2e/test_header_consistency.py` if it quotes the labels. Where a comment also explains the phase-35 origin, keep that history and only fix the quoted label.
|
|
||||||
|
|
||||||
**Explicitly DO NOT touch** (different controls / different text — they stay green and must stay):
|
|
||||||
- `tests/e2e/test_nav_consistency.py` lines asserting the document viewer's **back button** `span` text "Sources" / "Chat" — that is the viewer's back link (`document.js`), not the nav.
|
|
||||||
- `tests/e2e/test_document_back_navigation.py` — same back button.
|
|
||||||
- `tests/unit/test_document_viewer.py` — the `'Sources' in js` check on the viewer's back-link label.
|
|
||||||
- `tests/e2e/test_nav_consistency.py` `expect(page.locator("#sync-label")).to_have_text("Sync sources")` — the Sync button label is unchanged.
|
|
||||||
- `tests/integration/test_api.py::test_html_pages_served_locally_no_cdn` — the `("/git-sources.html", "Git sources")` marker matches the page's unchanged `<h1>`; leave it.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No new logic — assertion-text updates only.
|
|
||||||
- Coverage: frontend/test-only; `app/` coverage unaffected.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `grep -rn '"Git sources"\|Git sources' tests/e2e/*.py` shows no remaining assertion expecting the old nav label (only phase-history comments, if any, are acceptable — and none should quote it as the current label).
|
|
||||||
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] The two updated E2E suites pass in isolation: `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` and `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` (DB up).
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Task 03 — Story E2E, regression suites, commit, phase move
|
|
||||||
|
|
||||||
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the renamed nav end to end with the story's dedicated Playwright suite (run in isolation per A16), confirm the surrounding header suites stay green, and land the one atomic `--no-gpg-sign` commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **Create `tests/e2e/test_nav_rename_sources.py`** (one story, one file — follow the conventions of `test_git_sources_admin.py` / `test_nav_consistency.py`: the shared `app_url` / `db_ready` fixtures, the `login(page, app_url, next=…)` helper, desktop viewport 1280×800, `expect` sync-API assertions, a module docstring stating the story + the rename). The six pages under test are `/` (chat), `/sources.html`, `/git-sources.html`, `/tuning.html`, `/login.html`, and `/document.html` (viewer — seed one document row first, reusing the seeding pattern `test_nav_consistency.py` uses for its viewer pass, and wait for `#doc-title` to settle as that file does).
|
|
||||||
|
|
||||||
Test cases:
|
|
||||||
- **admin_labels_on_all_six_pages** — signed-in admin visits each of the six pages; on each: `#nav-sources` is visible, has text exactly `RAG` and href `/sources.html`; `#nav-git-sources` is visible, has text exactly `Sources` and href `/git-sources.html`; the nav DOM order is Chat, RAG, Sources, Tuning (assert the `.app-nav a` text sequence); the current page's link carries `is-active`/`aria-current="page"` (and only it does).
|
|
||||||
- **click_navigates_with_marker** — from the chat page: click `#nav-sources` (label "RAG") → lands on `/sources.html` where `#nav-sources` is the active link; from the chat page: click `#nav-git-sources` (label "Sources") → lands on `/git-sources.html` where `#nav-git-sources` is the active link.
|
|
||||||
- **anonymous_sees_neither** — anonymous (no login): on `/` and `/login.html` both `#nav-sources` and `#nav-git-sources` are present in the DOM but hidden (ship-hidden contract unchanged), `#sign-in-link` visible.
|
|
||||||
- **untouched_controls_stay** — the rename did not leak: on `/sources.html` the Sync button still reads "Sync sources" (`#sync-label`), and on the settled viewer page the back button's span still reads "Sources" (the viewer back link is a different control — regression guard for task 02's do-not-touch list).
|
|
||||||
2. **Run the regression suites in isolation** (DB up, `--no-cov`), fixing nothing unless a test genuinely asserted a renamed nav label (if one does, update it as in task 02 and note it in the commit message):
|
|
||||||
`test_nav_consistency.py`, `test_shared_header.py`, `test_mobile_hamburger_nav.py`, `test_git_sources_admin.py`, `test_header_consistency.py`, `test_sources_midstream_bug.py`, `test_smoke.py`, `test_tuning_nav_link.py`.
|
|
||||||
3. **Full gates**: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` with TOTAL unchanged from the pre-phase baseline (frontend-only change); `uv run ruff check . && uv run pyright` clean.
|
|
||||||
4. **UI Structure Check (AGENTS.md rule 5)** — the relabeled links are still labeled `<a>`s inside the semantic `<nav aria-label="Primary">`; landmarks/contrast/focus-visible untouched (label text only); no CDN (rule 6 — the no-CDN integration test covers it).
|
|
||||||
5. **Commit** — one atomic commit staging exactly this phase's files (the six HTML files, `frontend/assets/header.js`, `tests/e2e/test_git_sources_admin.py`, `tests/e2e/test_mobile_hamburger_nav.py`, the comment-only test files from task 02, and the new `tests/e2e/test_nav_rename_sources.py`):
|
|
||||||
`feat(ui): rename nav items — "Sources" becomes "RAG", "Git sources" becomes "Sources"` with a body citing the owner request (2026-08-28) + phase 48. `git commit --no-gpg-sign` (repo also has `commit.gpgsign=false`).
|
|
||||||
6. **Move the phase directory**: `mv .agent/phases/todo/48_nav_rename_sources .agent/phases/complete/` and `git add -f .agent/phases/complete/48_nav_rename_sources` into the SAME commit as step 5 (`.agent/` is gitignored by design — AGENTS.md rule 8), plus the story file `git add -f .agent/user_stories/nav-sources-rag-rename.md`.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Story E2E: `tests/e2e/test_nav_rename_sources.py` green **in isolation**.
|
|
||||||
- Coverage: frontend-only; `app/` >90% gate unaffected (TOTAL unchanged).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_nav_rename_sources.py -v --no-cov` green in isolation (DB up).
|
|
||||||
- [ ] All eight regression suites green in isolation (commands above).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit containing the code/test files, the story file, and the moved phase directory; `.agent/phases/todo/` no longer lists 48.
|
|
||||||
- [ ] No behavior change in completed phases (the suites above are the proof).
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
# Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)
|
|
||||||
|
|
||||||
**Source:** owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.)
|
|
||||||
**Story:** `.agent/user_stories/archive-upload-sources.md`
|
|
||||||
**Context:** `35_git_sources_admin` (complete) — the `git_sources` table (`id, url, kind, path, added_at`), the admin-only `/api/git-sources` router, and the `/git-sources.html` manager page; `38_local_directory_sources` (complete) — `kind='local'` rows the Sync pipeline and `import_docs` walk directly, plus the page's "Add a local directory" form this phase removes; `32_admin_sync_button` + `41_sync_fail_fast_models` (complete) — the in-process pipeline parts this phase reuses: `check_models` fail-fast, `import_sources(sources, llm, prune=True)` (source name = folder basename, per-file transactions, per-source prune), `regenerate_overview`, and the sync-detail count keys (`files, added, updated, unchanged, pruned, errors, chunks, overview`); `16_admin_auth` (complete) — the `require_admin` router dependency the new route inherits.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Replace the local-directory form on the Sources page with an **archive upload** form: `POST /api/git-sources/upload` accepts `.tar`/`.tar.gz`/`.tgz`/`.zip`, unpacks it **safely** into `BOR_UPLOAD_DIR/<name>/` (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the `git_sources` row (`kind=local`, no duplicates), and **scans it** — single-source `import_sources(prune=True)` + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `48_nav_rename_sources` (todo — runs first) — sequential only: it relabels the nav in the same `git-sources.html` this phase edits (keeps the diffs clean).
|
|
||||||
- `35_git_sources_admin` (complete) — the table/API/page this phase extends; the `require_admin` router; the `IntegrityError → 409` backstop pattern (`_commit_new`).
|
|
||||||
- `38_local_directory_sources` (complete) — the `kind=local` rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
|
|
||||||
- `32_admin_sync_button` / `41_sync_fail_fast_models` (complete) — `check_models` + `import_sources` + `regenerate_overview` + the count-key contract the upload response mirrors.
|
|
||||||
- `16_admin_auth` (complete) — admin-only surface (A10 revision).
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_settings_and_unpack_utility.md` — `BOR_UPLOAD_DIR` + `BOR_UPLOAD_MAX_MB` settings, `.env.example`, and the new `app/rag/archive_upload.py` (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
|
|
||||||
2. `02_upload_api.md` — `python-multipart` dependency + `POST /api/git-sources/upload` (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
|
|
||||||
3. `03_admin_page_upload.md` — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
|
|
||||||
4. `04_story_e2e_docs_commit.md` — the story E2E (`test_archive_upload_sources.py`), README, regression suites in isolation, the one `--no-gpg-sign` commit, and the phase-dir move.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `tests/unit/test_archive_upload.py` — the name-derivation matrix (suffix stripping incl. double `.tar.gz`; empty/`..`/separator/control-char rejection); safe unpack (valid zip + tar; zip-slip `../`; absolute member; symlink + hardlink escape; device member; extracted-byte cap); `swap_in` (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
|
|
||||||
- Integration: `tests/integration/test_git_sources_upload.py` — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of `test_git_sources_api.py`); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row `kind=local` under `upload_dir`, docs in the KB); **re-upload same name** (one row, old folder content fully replaced, dropped file pruned, new file indexed); **failed re-upload leaves the previous folder + row + KB untouched**. The existing `test_git_sources_api.py` / `test_sync_api.py` / `test_import_docs_git.py` suites stay green through the change (the `kind=local` POST contract is untouched).
|
|
||||||
- Coverage: **>90%** on `app/` — the new module + endpoint fully covered.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_archive_upload_sources.py`, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
|
|
||||||
- [ ] Uploading `homelab.tar.gz` via the page: unpacked under `BOR_UPLOAD_DIR/homelab/`, scanned (result line shows the counts), one list row (Local badge, name `homelab`), documents visible on `/sources.html`.
|
|
||||||
- [ ] Re-uploading `homelab.tar.gz` (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
|
|
||||||
- [ ] Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
|
|
||||||
- [ ] `#local-source-form` is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
|
|
||||||
- [ ] Regression E2E suites green in isolation: `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (when 48 is complete), `test_smoke.py`.
|
|
||||||
- [ ] README + `.env.example` document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions (owner permission 2026-08-28 — the confirmed design)
|
|
||||||
- **Formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` only (422 naming the accepted set otherwise).
|
|
||||||
- **Identity & in-place replace:** the source name is the uploaded filename minus the archive suffix (`homelab.tar.gz` → `homelab`, case-sensitive — Linux FS). The name determines the folder under `BOR_UPLOAD_DIR`; re-uploading the same name unpacks to a temp sibling and **renames it over the existing folder** (no missing window; a failed upload never touches the existing folder, row, or KB). **No second folder, no second row** — the `git_sources` row is upserted by `path` (`kind='local'`, reusing the phase-38 discriminator — **no migration, no new table**; A13 honoured).
|
|
||||||
- **New settings:** `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads` — deliberately separate from the git checkouts in `BOR_SOURCES_DIR`) and `BOR_UPLOAD_MAX_MB` (default **512**) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
|
|
||||||
- **The scan is synchronous in the upload request** (owner-confirmed): fail-fast `check_models` (phase 41) → `import_sources([folder], llm, prune=True)` (single source) → `regenerate_overview` when the KB changed → **200** with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
|
|
||||||
- **Unpack safety:** absolute member paths, `..` traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
|
|
||||||
- **Page:** the "Add a local directory" form is **removed**; the `POST /api/git-sources` `kind=local` **API contract is unchanged** (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + `import_docs` keep walking them).
|
|
||||||
- **No auto-unwrap** of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
|
|
||||||
- **`python-multipart`** is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
|
|
||||||
- **Boundaries (deliberately out of scope):** page `<title>`/`<h1>` rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
|
|
||||||
- **A10 / A11 / A16 / A17 honoured** — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic `--no-gpg-sign` commit.
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
# Task 01 — Settings + safe archive unpack utility
|
|
||||||
|
|
||||||
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Add the two upload settings and the reusable, unit-tested unpack machinery in a new module `app/rag/archive_upload.py`: archive-name derivation, safe tar/zip extraction (traversal/symlink/device/size guards), and the atomic swap-in that makes re-uploads replace in place without ever exposing a missing or partial folder.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `app/config.py` — add to `Settings` (both env-overridable, documented like `sources_dir`):
|
|
||||||
- `upload_dir: str = "~/bor-sources/uploads"` (→ `BOR_UPLOAD_DIR`) — where uploaded archives are unpacked (one subdirectory per source name). Kept **separate** from `sources_dir` (git checkouts).
|
|
||||||
- `upload_max_mb: int = 512` (→ `BOR_UPLOAD_MAX_MB`) — caps both the compressed upload and the total extracted bytes (zip-bomb guard). Add a validator rejecting `<= 0` (fail loud at startup, the `agent_max_rounds` pattern).
|
|
||||||
2. `.env.example` — document both settings next to `BOR_SOURCES_DIR` (default, meaning, the cap's dual role).
|
|
||||||
3. `app/rag/archive_upload.py` (new module) — pure file-system logic, no FastAPI imports (the API layer maps its exceptions to status codes):
|
|
||||||
- `class ArchiveUploadError(Exception)` — carries a user-safe message (no paths beyond the owner's own upload dir, never secrets).
|
|
||||||
- `ARCHIVE_SUFFIXES: tuple[str, ...] = (".tar.gz", ".tgz", ".zip", ".tar")` (longest-first — `.tar.gz` must strip before `.tar` would).
|
|
||||||
- `archive_source_name(filename: str) -> str` — take the basename (defensively strip any `/` or `\` a client could send), strip ONE trailing archive suffix from `ARCHIVE_SUFFIXES`; the result must be non-empty and not `.`/`..`, contain no path separators or control characters, or raise `ArchiveUploadError` (the API maps to 422). `homelab.tar.gz` → `homelab`; `notes.tgz` → `notes`; `a.zip` → `a`; `x.tar` → `x`; bare `tar.gz` → error (empty stem).
|
|
||||||
- `unpack_archive(archive: Path, target_dir: Path, max_extract_bytes: int) -> None` — extract into `target_dir` (the caller guarantees it does not exist yet and creates it empty):
|
|
||||||
- **zip** (`zipfile`): per member — reject absolute names and any name with a `..` part; resolve the final path and require it to stay within `target_dir`; reject symlink entries (mode bits) and non-file/non-dir entries; write file members while counting bytes — exceeding `max_extract_bytes` raises `ArchiveUploadError` (name the cap, not the archive content).
|
|
||||||
- **tar** (`tarfile.open(mode="r:*")` — handles gz/bz2/xz transparently): per member — the same name/containment checks; reject char/block devices and FIFOs; for symlinks/hardlinks, resolve the link target against the member's directory and reject any target that escapes `target_dir`; write regular files with the byte-counted cap.
|
|
||||||
- Clean up partial state: on any error, remove `target_dir` (shutil.rmtree, ignore missing) so no half-unpacked tree survives.
|
|
||||||
- `swap_in(new_dir: Path, final_dir: Path) -> None` — make `new_dir` become `final_dir` with **no missing window**: if `final_dir` exists, rename it to a same-filesystem sibling `final_dir.with_name(final_dir.name + ".old-" + uuid4().hex)`, rename `new_dir` → `final_dir`, then delete the `.old-` sibling; if it does not exist, just rename. On a rename failure, best-effort restore (`.old-` back, `new_dir` cleaned) and re-raise as `ArchiveUploadError`.
|
|
||||||
4. `tests/unit/test_archive_upload.py` (new) — full coverage of the module:
|
|
||||||
- name derivation: the matrix above incl. `upper.TAR.GZ` (case-sensitive stems preserved — `upper`), `a.tar.gz` double-strip, `..tar.gz` / `..` / `a/b.tar` / `a\tb.zip` / empty-stem rejections.
|
|
||||||
- unpack: a valid zip (nested dir + file) and a valid tar.gz extract byte-identically; zip-slip (`../evil.txt`), absolute member (`/etc/x`), tar symlink escaping (`ln -s /etc/passwd link`), tar hardlink escaping, a char-device member, and the extracted-cap (e.g. cap=10 bytes, 20-byte file) each raise `ArchiveUploadError` AND leave no partial `target_dir` behind.
|
|
||||||
- `swap_in`: fresh (final absent), replace (final's previous content fully gone, new content complete — no interleave), and restore-on-failure (monkeypatch `os.rename` to fail on the second rename → previous folder intact, new dir cleaned, `ArchiveUploadError` raised).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: `uv run pytest tests/unit/test_archive_upload.py -v` green; the module is fully covered (the >90% gate applies to `app/` — this new module must not drag the TOTAL down; aim for ~100% here).
|
|
||||||
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — no drop vs. baseline (no existing behavior touched).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `Settings` exposes `upload_dir` / `upload_max_mb` (defaults as above); `BOR_UPLOAD_MAX_MB=0` or negative fails startup with the validator message.
|
|
||||||
- [ ] `.env.example` documents both settings.
|
|
||||||
- [ ] `app/rag/archive_upload.py` exists with `ArchiveUploadError`, `ARCHIVE_SUFFIXES`, `archive_source_name`, `unpack_archive`, `swap_in`; no FastAPI/DB imports in the module.
|
|
||||||
- [ ] `uv run pytest tests/unit/test_archive_upload.py -v` green; full `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] No behavior change in completed phases (no existing file edited beyond config + .env.example).
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# Task 02 — `POST /api/git-sources/upload` endpoint
|
|
||||||
|
|
||||||
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Ship the admin-only upload endpoint: stream the archive with the size cap, unpack it safely to a temp sibling, swap it in atomically, upsert the `git_sources` row (no duplicates), fail-fast check the models, scan the single source (`prune=True`) with the overview refresh, and answer 200 with the sync-style counts. One upload at a time (409).
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `pyproject.toml` — add `python-multipart` to the main `dependencies` list (FastAPI's required multipart parser; the owner-confirmed implementation detail, phase locked decisions), then `uv lock`.
|
|
||||||
2. `app/api/git_sources.py` — extend the existing admin-only router (the `require_admin` dependency covers the new route automatically):
|
|
||||||
- `@router.post("/upload", response_model=UploadOut)` (new `UploadOut` schema in `app/schemas.py`: `source: str` + the sync-detail keys `files, added, updated, unchanged, pruned, errors, chunks: int` and `overview: bool` — same names as `_run_sync`'s `detail` so the page reuses its result-line shape).
|
|
||||||
- Handler (`async def upload_archive(file: UploadFile = File(...)) -> UploadOut`):
|
|
||||||
1. **Name/format gate** — `archive_source_name(file.filename or "")`: `ArchiveUploadError` → **422** with its message (naming the accepted formats when the extension is the problem).
|
|
||||||
2. **One at a time** — module-level `asyncio.Lock` (or a `bool` flag, the phase-32 `_task` spirit): already in flight → **409** `"an upload is already in progress"`.
|
|
||||||
3. **Stream with cap** — read the upload in 1 MiB chunks into `upload_dir / f".{name}.{uuid4().hex}.upload"`; total bytes > `upload_max_mb * 1024 * 1024` → delete the temp file, **413** naming the cap. (Compute `upload_dir` once: `Path(get_settings().upload_dir).expanduser()` — create it with `mkdir(parents=True, exist_ok=True)`.)
|
|
||||||
4. **Unpack to temp sibling** — `unpack_archive(archive, upload_dir / f".{name}.{uuid4().hex}.unpack", same cap)`; `ArchiveUploadError` → delete both temps, **422** (the message is already user-safe). A **completely empty archive** (zero entries) is **422** `"the archive contains no files"`. An archive with only non-A9 files is a **valid replacement**: the swap happens, the scan indexes nothing, and prune removes that source's docs — that is the intended "replace" semantics, do not reject it.
|
|
||||||
5. **Swap in** — `swap_in(temp_unpack, upload_dir / name)`; `ArchiveUploadError` → clean temps, **422** (the previous folder/row/KB are untouched — assert this in the tests).
|
|
||||||
6. **Upsert the row** — `path = str(upload_dir / name)`: a `GitSource` row with `path == path` already exists → leave it (no new row, `added_at` preserved); otherwise create `GitSource(url=path, kind="local", path=path)` (the `url` column is the NOT-NULL location column, phase-38 convention) via the shared `_commit_new` IntegrityError → 409 backstop.
|
|
||||||
7. **Fail-fast models** (phase 41) — `llm = LLMClient()`; `await check_models(llm)`; failure → **503** with the sanitized model-unavailable message (clean up nothing else — the folder/row are already committed, and the next sync/re-upload retries idempotently).
|
|
||||||
8. **Scan** — `summary = await import_sources([upload_dir / name], llm, prune=True)`; `overview = await regenerate_overview(llm) if summary.added + summary.updated > 0 else False`.
|
|
||||||
9. **Log** (PLAN §9, AGENTS.md rule 10) — one INFO line: `upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…`.
|
|
||||||
10. **Respond 200** with `UploadOut(source=name, …counts…, overview=overview)`.
|
|
||||||
- Keep every existing route byte-identical (the `kind=local` POST contract is untouched — the page just stops offering it).
|
|
||||||
3. `tests/integration/test_git_sources_upload.py` (new) — real Postgres + TestClient, following `tests/integration/test_git_sources_api.py`'s conventions (`clean_git_sources`-style TRUNCATE autouse fixture, the monkeypatched `get_settings` pattern to point `upload_dir` at a tmp dir and shrink `upload_max_mb`, the shared `client` / `admin_client` / `db` fixtures). Build real archives in-test with `tarfile`/`zipfile` over tmp fixture files (two `.md` sentinels). Cases:
|
|
||||||
- anonymous `client` → 403 on `/api/git-sources/upload` (same body as the rest of the router).
|
|
||||||
- `admin_client` + `data={"file": ("notes.txt", b"…", "text/plain")}` → **422** (accepted formats named); `("…", …)` with a `..`/empty stem → 422.
|
|
||||||
- oversized (shrink `upload_max_mb` via the settings override, upload a bigger file) → **413**, temp file cleaned up (the upload dir holds no stray `.` files).
|
|
||||||
- zip-slip archive (a member named `../evil.txt`) and a tar-slip symlink archive → **422**; then a prior good upload's folder + row + docs are **untouched** (the no-partial-state locked decision, asserted explicitly).
|
|
||||||
- happy path: `homelab.tar.gz` (2 md files) → **200**, `source="homelab"`, `added=2`, a `git_sources` row exists (`kind=local`, `path` under the tmp `upload_dir`), the unpacked folder exists, `GET /api/docs` lists both files under source `homelab`.
|
|
||||||
- **re-upload, same name, modified archive** (drop one file, add one, change one) → 200; `git_sources` still has exactly **one** row for that path (`added_at` unchanged); the folder contains only the new archive's files; the KB: dropped file **pruned** (pruned ≥ 1), new file added, changed file updated.
|
|
||||||
- corrupt archive (truncated zip bytes) → 422, previous state intact.
|
|
||||||
- 409: hold the in-flight flag (test seam: a module-level `upload_in_progress()` helper or the lock object exposed for tests — pick the smallest seam) → second request → **409**.
|
|
||||||
- the existing `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` suites stay green (run them).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Integration: all cases above green; the endpoint's every branch (422/413/409/503/200 + both upsert branches + the empty-archive 422) is covered.
|
|
||||||
- Coverage: **>90%** on `app/` (the new handler + schema fully exercised).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `python-multipart` in `pyproject.toml` `dependencies` + lock file updated; `uv sync` clean.
|
|
||||||
- [ ] `POST /api/git-sources/upload` implements steps 1–10; `git diff app/api/git_sources.py` shows no change to the existing GET/POST/DELETE handlers' behavior.
|
|
||||||
- [ ] `uv run pytest tests/integration/test_git_sources_upload.py -v` green (DB up); `uv run pytest tests/unit tests/integration -q` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
# Task 03 — Sources page: local form out, upload form in (+ phase-38 E2E rewrite)
|
|
||||||
|
|
||||||
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Rework the manager UI: remove the "Add a local directory" form and replace it with the labeled archive upload form (never-stale button, inline error, result line), update the hint/caption, and — because the form it drives is gone — rewrite the phase-38 story E2E to add local sources via the API instead.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `frontend/git-sources.html`:
|
|
||||||
- **Remove** the whole `#local-source-form` block (the phase-38 form: label, `#local-source-path` input, `#local-source-add` button, `#local-source-error` — including its phase-38 comment).
|
|
||||||
- **Add**, in its place (same `.git-source-error` / never-stale visual language as the git form):
|
|
||||||
```html
|
|
||||||
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
|
|
||||||
replaces the phase-38 local-directory form — an uploaded
|
|
||||||
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
|
|
||||||
scanned immediately; the same filename replaces the source in
|
|
||||||
place (no new folder, no duplicate row). -->
|
|
||||||
<form id="archive-upload-form">
|
|
||||||
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
|
|
||||||
<input id="archive-upload-file" name="file" type="file"
|
|
||||||
accept=".tar,.tar.gz,.tgz,.zip" required>
|
|
||||||
<button type="submit" id="archive-upload-btn">Upload & scan</button>
|
|
||||||
<p class="git-source-error" id="archive-upload-error" role="alert" hidden></p>
|
|
||||||
<p class="git-source-result" id="archive-upload-result" role="status"
|
|
||||||
aria-live="polite" hidden></p>
|
|
||||||
</form>
|
|
||||||
```
|
|
||||||
(No `Content-Type` concerns: the JS posts a `FormData` and the browser sets the multipart boundary. The visible `<label>` satisfies the WCAG input-label rule for the file control.)
|
|
||||||
- **Update the hint** `#git-sources-hint`: uploads unpack + scan immediately and a same-name re-upload replaces in place; the Sync button still imports git checkouts and local directories together (union prune), and removing a source prunes on the next sync. Keep it one `role="note"` paragraph.
|
|
||||||
- **Update the table `<caption>`** (visually hidden): it reads "git repositories it clones and local directories it walks" — add uploaded archives (unpacked under the upload dir) to the description.
|
|
||||||
- Do **not** touch the page `<title>`/`<h1>` ("Git sources") — out of scope (flagged in phase 48); the nav link already reads "Sources" after phase 48.
|
|
||||||
2. `frontend/assets/styles.css` — one small block near the git-sources page styles: the file input (mono-ish, on-surface, ≥44px touch target, `:focus-visible` 3px outline like the other controls) and the `#archive-upload-result` success line (ink-soft on surface, ≥4.5:1 — reuse the existing palette; `prefers-reduced-motion` already global). Keep it minimal — no new layout regions.
|
|
||||||
3. `frontend/assets/git-sources.js`:
|
|
||||||
- Remove the local-form element refs (`localFormEl`, `pathInput`, `localAddBtn`, `localAddError`) and the second `wireAddForm(…)` call; update the module docstring (the page now wires the git add form + the archive upload).
|
|
||||||
- **Upload wiring** (one new `wireUploadForm`-style submit handler on `#archive-upload-form`), the §7.4 never-stale lifecycle:
|
|
||||||
- submit → `e.preventDefault()`; no file selected → inline error (the input is `required` too — browser prompt first); hide a previous result line; disable `#archive-upload-btn`, label **"Uploading…"**.
|
|
||||||
- `fetch("/api/git-sources/upload", { method: "POST", body: new FormData([["file", file]]) })` — **no** manual `Content-Type` header.
|
|
||||||
- **200**: clear the file input; hide the error; show `#archive-upload-result` with the counts in the sync-result shape (`2 added · 1 updated · 3 unchanged · 1 pruned` — omit zero parts, the `fmtSyncResult` convention from `sources.js`); announce through `#git-sources-announcer` (`"Archive uploaded: …"`); `await loadSources()` (the new/updated row lands with the Local badge; on a re-upload the row simply refreshes — no duplicate).
|
|
||||||
- **non-2xx**: inline the server detail via the existing `apiDetail(r, fallback)` (422 format/name/traversal, 413 size, 409 busy — the server messages are already user-safe); keep the file selection; restore the button (finally block, success AND failure).
|
|
||||||
- **network failure**: the `networkMessage` line, button restored.
|
|
||||||
4. **Rewrite `tests/e2e/test_local_directory_sources.py`** (phase-38 story — the form it drives is gone; its *acceptance* stands):
|
|
||||||
- Replace every form interaction (`page.fill("#local-source-path", …)` + `page.click("#local-source-add")`) with the authenticated API call the page's own JS no longer makes: `r = page.request.post(f"{app_url}/api/git-sources", json={"kind": "local", "path": str(local_dir)})` (the cookie rides the browser context — the established `page.request` pattern, cf. `test_git_sources_admin.py`).
|
|
||||||
- The "missing path → inline 422 naming the path + input kept" test becomes: the API returns **422** whose `detail` names the path (assert on the JSON body); drop the input-value assertions.
|
|
||||||
- Everything else stays: the Local badge on list rows, add/remove lifecycle, Sync importing the local dir (visible via `GET /api/docs` / the page), prune-on-file-deletion after re-sync, anonymous 403s, and the gate.
|
|
||||||
- The module docstring must note the phase-49 rewrite (form → API) so a future reader doesn't "restore" the form.
|
|
||||||
5. **UI Structure Check (AGENTS.md rule 5)** while in there: the new form is inside the existing `<main>` region, labeled, focus-visible, error `role=alert`, result `role=status`; no CDN (rule 6 — the no-CDN integration test re-proves it).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- No new unit/integration logic (frontend + one E2E rewrite). `app/` coverage unaffected.
|
|
||||||
- The no-CDN integration test still passes (markup only, same-origin).
|
|
||||||
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov` green in isolation **after** the rewrite (DB up) — this is the proof the form removal caused no regression in the phase-38 story.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `#local-source-form` / `#local-source-path` / `#local-source-add` are gone from `frontend/git-sources.html` (and `git-sources.js`); `#archive-upload-form` with file input (accept set), button, `role=alert` error and `role=status` result line is in its place.
|
|
||||||
- [ ] Hint + caption updated; page `<title>`/`<h1>` untouched.
|
|
||||||
- [ ] `tests/e2e/test_local_directory_sources.py` green in isolation, fully API-driven for local adds.
|
|
||||||
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean (no Python behavior change expected in `app/` beyond nothing).
|
|
||||||
- [ ] Manual-feel check via the story E2E in task 04 (upload → counts → row) — this task leaves the page in a usable state on its own (independent viability).
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# Task 04 — Story E2E, README, regression suites, commit, phase move
|
|
||||||
|
|
||||||
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the archive-upload story end to end with its dedicated Playwright suite (A16, in isolation), document the feature in the README, confirm no regression in the surrounding suites, and land the one atomic `--no-gpg-sign` commit.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. **Create `tests/e2e/test_archive_upload_sources.py`** (one story, one file — conventions of `test_git_sources_admin.py` / `test_local_directory_sources.py`: shared `app_url` / `db_ready` fixtures, the `login(page, app_url, next=…)` helper, `page.request` for API assertions, desktop viewport 1280×800, sync `expect`). Build real archives in-test with Python's `tarfile` over `tmp_path` fixture files (markdown sentinels, e.g. `ALPHA-…` / `BETA-…` / `GAMMA-…`), named `e2e-upload.tar.gz` (source name will be `e2e-upload`).
|
|
||||||
|
|
||||||
Test cases:
|
|
||||||
- **form_swapped** — signed-in admin on `/git-sources.html`: `#local-source-form` count is 0; `#archive-upload-form` is visible with the file input (`accept` contains the four extensions) and the "Upload & scan" button; the hint mentions unpack/scan + in-place replace.
|
|
||||||
- **upload_scans_and_lists** — `page.set_input_files("#archive-upload-file", tarball_v1)` + click: the button shows "Uploading…" while in flight, then restores; `#archive-upload-result` shows the added count (2); the list gains exactly one row for `e2e-upload` with the **Local** badge; `page.request.get("/api/docs")` lists both sentinel files under source `e2e-upload`; on `/sources.html` (the RAG catalog) the table shows them (the admin sees the content where they expect it).
|
|
||||||
- **reupload_replaces_in_place** — back on the page, upload `tarball_v2` under the **same filename** (`e2e-upload.tar.gz`; v2: `alpha` modified, `beta` removed, `gamma` added): result line shows pruned ≥ 1; the list still has exactly **one** `e2e-upload` row (no duplicate — the row count for that source is invariant); `/api/docs` now shows `gamma` + the changed `alpha` and NOT `beta`.
|
|
||||||
- **bad_file_inline_error** — upload a `.txt` via the file input: `#archive-upload-error` (role=alert) shows the 422 detail naming the accepted formats; the button restores; the list is unchanged; a subsequent good upload still works (the form isn't wedged).
|
|
||||||
- **anonymous_gate** — anonymous on `/git-sources.html`: the gate (`#git-sources-gate`) shows, `#git-sources-content` (and thus the upload form) stays hidden, and `page.request.post(f"{app_url}/api/git-sources/upload", …)` is 403.
|
|
||||||
2. **README** — in the sources section (next to the phase-35/38 admin-sources docs): the upload form (accepted formats), the naming rule (filename minus archive suffix = source/folder name), the in-place replace + prune semantics on re-upload, the unpack destination (`BOR_UPLOAD_DIR`, default `~/bor-sources/uploads`) and the size cap (`BOR_UPLOAD_MAX_MB`, default 512, compressed + extracted), and the note that the local-directory *form* is gone but `POST /api/git-sources` with `kind=local` still works and existing Local rows are unchanged.
|
|
||||||
3. **Run the regression suites in isolation** (DB up, `--no-cov`): `test_git_sources_admin.py`, `test_local_directory_sources.py` (task 03's rewrite), `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (if phase 48 is complete — otherwise note it as the 48 gate), `test_smoke.py`, `test_shared_header.py`.
|
|
||||||
4. **Full gates**: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
5. **Commit** — one atomic commit staging exactly this phase's files (config, `.env.example`, `app/rag/archive_upload.py`, `app/api/git_sources.py`, `app/schemas.py`, `pyproject.toml` + lock, the two new test files, the rewritten `tests/e2e/test_local_directory_sources.py`, `frontend/git-sources.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, README):
|
|
||||||
`feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place` with a body citing the owner request (2026-08-28) + phase 49. `git commit --no-gpg-sign`.
|
|
||||||
6. **Move the phase directory**: `mv .agent/phases/todo/49_archive_upload_sources .agent/phases/complete/` and `git add -f` the moved directory + the story file `.agent/user_stories/archive-upload-sources.md` into the SAME commit (`.agent/` is gitignored by design — AGENTS.md rule 8).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Story E2E: `tests/e2e/test_archive_upload_sources.py` green **in isolation**.
|
|
||||||
- Coverage: `app/` >90% (the phase's Testing & Quality bar, re-proven on the final pass).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
|
|
||||||
- [ ] All regression suites green in isolation (list above).
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] README + `.env.example` document the upload semantics and both new settings.
|
|
||||||
- [ ] One `--no-gpg-sign` commit containing code, tests, story file, and the moved phase directory; `.agent/phases/todo/` no longer lists 49.
|
|
||||||
- [ ] No behavior change in completed phases (the suites above are the proof — incl. the Sync button and the `kind=local` API contract).
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# Phase 40 — Tuning toggle anonymous flash
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L3 — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
|
|
||||||
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
|
|
||||||
**Context:** Phase 15/34 shared header (`frontend/assets/header.js` owns `#steering-toggle` / `#steering-panel` on all six pages; anonymous → `remove()` post-whoami). The admin-only **nav links** already ship `hidden` (phase-19 contract) — the flashing control is the **steering toggle button labeled "Tuning"**, which ships visible in all six pages and is removed only after `/api/whoami` resolves.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Kill the anonymous flash: the tuning toggle ships `hidden` in every page's markup and is revealed only when whoami says admin (the exact ship-hidden / reveal-for-admin contract the nav links use), so an anonymous user never sees the "Tuning" button — not for a single frame.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `39_configurable_brand` (complete; last existing phase) — current header state: full shared bar on all six pages.
|
|
||||||
- `19_shared_header` / `16_admin_auth` / `34_consistent_navbar` (complete) — the `fetchIsAdmin()` gate, the ship-hidden nav contract, and the module-owned steering controls this task modifies.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_toggle_ships_hidden.md` — add `hidden` to `#steering-toggle` in all six pages and reveal-for-admin in `header.js`; pin at source level.
|
|
||||||
2. `02_flash_e2e_and_regression.md` — story E2E suite (never-visible-for-anonymous, admin reveal, nav-contract regression) + regression pass + commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: new `tests/unit/test_steering_toggle_visibility.py` — `hidden` present on `#steering-toggle` in all six HTML pages; `header.js` unhides for admin (line before `refreshSteering()`) and the anonymous `remove()` path is intact; any existing source-pin test asserting the exact old markup is updated (check `tests/unit/test_shared_header.py`, `test_steering.py`).
|
|
||||||
- Coverage: frontend-only — the `app/` >90% gate is unaffected (must stay unchanged).
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_tuning_toggle_flash.py`, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] Anonymous load of every page: the toggle is never visible (MutationObserver records zero visible frames) and is absent from the DOM after load.
|
|
||||||
- [ ] Admin load: toggle visible, panel opens, count badge correct — admin behavior unchanged.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_tuning_toggle_flash.py -v --no-cov` green in isolation.
|
|
||||||
- [ ] Regression E2E suites green in isolation: `test_shared_header.py`, `test_global_tuning.py`, `test_steering.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] UI Structure Check (AGENTS.md rule 5): no new focus targets; landmarks/contrast unchanged; no CDN.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase dir moved `.agent/phases/todo/` → `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A10 untouched** — no API change; the fix is pure UI visibility off the existing `/api/whoami` gate.
|
|
||||||
- **A11 untouched** — no new assets, no CDN.
|
|
||||||
- **Phase-16 contract preserved** — anonymous still gets "absent, not hidden" (remove-from-DOM); this phase only removes the pre-whoami flash window.
|
|
||||||
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
# Task 02 — Flash E2E + regression + commit
|
|
||||||
|
|
||||||
**Phase:** `40_tuning_toggle_flash` · **Source:** `TODO.md:3` — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
|
|
||||||
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
Prove the flash is gone at the browser level (never visible, not even for a frame) and that the shared-header contract is intact; commit the phase.
|
|
||||||
|
|
||||||
## Work
|
|
||||||
1. `tests/e2e/test_tuning_toggle_flash.py` (new) — mock-only suite (DB up), per the story's Playwright Mapping Rule:
|
|
||||||
- a helper `install_visibility_observer(page)`: `page.add_init_script` a MutationObserver on `document.documentElement` that appends to `window.__tuningVisibleFrames` every time `#steering-toggle` is added/attribute-changed and is both in the DOM **and** not `[hidden]` (check `el.offsetParent !== null` or `!el.hidden`);
|
|
||||||
- `test_anonymous_never_sees_toggle` — load `/` anonymously, wait for network idle + header settle (whoami resolved), assert `window.__tuningVisibleFrames` is empty and `#steering-toggle` is absent from the DOM;
|
|
||||||
- `test_anonymous_other_pages_never_flash` — same on `/sources.html`, `/tuning.html`, `/login.html`;
|
|
||||||
- `test_admin_toggle_revealed_and_working` — `login()` (e2e.auth_helpers), reload `/`, toggle visible + clickable (opens `#steering-panel`, `aria-expanded="true"`), count badge matches the list;
|
|
||||||
- `test_nav_contract_regression` — anonymous: `#nav-sources` / `#nav-git-sources` / `#nav-tuning` stay hidden; admin: revealed.
|
|
||||||
2. Regression pass (isolation runs, per A16): `test_shared_header.py`, `test_global_tuning.py`, `test_steering.py`, `test_tuning_nav_link.py`, `test_smoke.py` — all green; fix only true regressions.
|
|
||||||
3. `uv run pytest` (unit+integration) green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
|
|
||||||
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `fix(header): ship the tuning toggle hidden — no anonymous flash`, staging this phase's changed files; move `.agent/phases/todo/40_tuning_toggle_flash/` → `.agent/phases/complete/` (force-add per AGENTS.md rule 8 if the history tracks the tree).
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- E2E: `uv run pytest tests/e2e/test_tuning_toggle_flash.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`).
|
|
||||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] The story E2E file passes in isolation; the four regression suites pass in isolation.
|
|
||||||
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
# Phase 41 — Sync fails fast + modal when a model is down
|
|
||||||
|
|
||||||
**Source:** `TODO.md` L4 — "If the embedding or lite model is not accessible the sync button should fail fast and there should be a modal error popup explaining that the model isn't available."
|
|
||||||
**Story:** `.agent/user_stories/sync-model-fail-fast.md`
|
|
||||||
**Context:** `app/api/sync.py::_run_sync` (phase 32/35/38) runs source resolution → git clones → `import_sources` (embeds) → overview (lite) — with a dead LLM endpoint the run discovers it only mid-import, after slow clones. The sync state machine is module-owned by `frontend/assets/header.js` (`applySyncFailure` → button title/aria + `.is-error` + `bor:sync-status` event; the Sources page renders `#sync-error-banner`). No dialog component exists yet.
|
|
||||||
|
|
||||||
## Objective
|
|
||||||
When `embed` or `lite` is unreachable, the sync fails **before any expensive work** with a message naming the model, and the failure is readable in a **modal dialog** on every page that carries `#sync-btn`.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
- `40_tuning_toggle_flash` (todo) — current shared-header state (sequential; no code overlap, but both touch `header.js` — keep this phase's changes confined to the sync section).
|
|
||||||
- `32_admin_sync_button` / `35_git_sources_admin` / `38_local_directory_sources` (complete) — the pipeline, the status contract, and the module-owned button lifecycle this phase extends.
|
|
||||||
|
|
||||||
## Tasks
|
|
||||||
1. `01_model_probe_fail_fast.md` — `check_models()` probe in `app/rag/llm.py`, called first in `_run_sync`; unit + integration tests.
|
|
||||||
2. `02_sync_error_modal.md` — `header.js` modal (built in JS, all pages) + CSS; source pins.
|
|
||||||
3. `03_model_down_e2e_and_commit.md` — dedicated E2E suite (dead-LLM module app) + phase-32 regressions + commit.
|
|
||||||
|
|
||||||
## Testing & Quality
|
|
||||||
- Unit: probe success/failure paths with a fake LLM client (embed-down, lite-down, both up); the sync task's fail-fast ordering (probe before source resolution — assert no clone call happens).
|
|
||||||
- Integration: `POST /api/sync` with a stubbed failing client → `GET /api/sync/status` reaches `failed` with the model-naming error; healthy path regression.
|
|
||||||
- Coverage: **>90%** on `app/` including the new probe code.
|
|
||||||
- E2E (mandatory, A16): `tests/e2e/test_sync_model_down.py`, run in isolation.
|
|
||||||
|
|
||||||
## Completion Criteria
|
|
||||||
- [ ] With a dead LLM endpoint: sync fails within seconds, **before** any clone, error names the unavailable model; the modal shows it; button settles retry-ready.
|
|
||||||
- [ ] Modal contract: `role="alertdialog"`, `aria-modal`, text via `textContent`, close via button / `Esc` / backdrop, focus in-and-out.
|
|
||||||
- [ ] Healthy sync pipeline (clone → import → overview) unchanged — phase-32 suite green in isolation.
|
|
||||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
|
|
||||||
- [ ] `uv run pytest tests/e2e/test_sync_model_down.py -v --no-cov` green in isolation (DB up).
|
|
||||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
|
||||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
|
|
||||||
|
|
||||||
## Locked decisions
|
|
||||||
- **A12 untouched** — still in-process, no queue, no new service; the probe is two cheap model calls.
|
|
||||||
- **A10 untouched** — no new endpoint; `/api/sync` + `/api/sync/status` keep their shapes (a model failure is just another `failed` state).
|
|
||||||
- **Phase-32 contract kept** — 2 s poll, 202/409, no client timeout, `bor:sync-status` event, button title/aria affordance, Sources banner (the modal is additive).
|
|
||||||
- **Owner-locked (2026-08-27, roadmap A4):** probe runs **before** git clones; the modal is the primary failure surface on every page.
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user