Compare commits
47
Commits
d4f38ad3ce
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bef24e05e2 | ||
|
|
a19d78d284 | ||
|
|
0f77e9a876 | ||
|
|
3a0fc3db05 | ||
|
|
0ff1f8c4d6 | ||
|
|
a5b63f83ad | ||
|
|
795fb56425 | ||
|
|
9820c361b0 | ||
|
|
21aad84a6d | ||
|
|
c851d1a1c5 | ||
|
|
2ac3fc89c2 | ||
|
|
adf31a4a35 | ||
|
|
990c8adf13 | ||
|
|
3846f26a58 | ||
|
|
97d663d16d | ||
|
|
1374faf136 | ||
|
|
2683128876 | ||
|
|
f37c517590 | ||
|
|
2b75f3cc85 | ||
|
|
9ce13b6290
|
||
|
|
3a4035fc96 | ||
|
|
35d65d2f25 | ||
|
|
3a81793565 | ||
|
|
fbbd98d734 | ||
|
|
6bcee831ec | ||
|
|
addbd4ca08 | ||
|
|
e2d08a95a9 | ||
|
|
ee3efb28c9 | ||
|
|
cec819743d | ||
|
|
8476dc1e07 | ||
|
|
f665a83b1a | ||
|
|
909c96c7bc
|
||
|
|
d731169b8b | ||
|
|
ecc921098a | ||
|
|
1f1c01c9f7 | ||
|
|
3b2dea5685 | ||
|
|
4dbac1660a | ||
|
|
58e9d94cff | ||
|
|
a2ca2f905f | ||
|
|
ffa6bda3e5 | ||
|
|
17dd3bfac1 | ||
|
|
800bea769a | ||
|
|
ad7585d474 | ||
|
|
a49be80b8e | ||
|
|
bcaef800c5 | ||
|
|
d4943b4822 | ||
|
|
9188be259b |
+42
-51
@@ -1,7 +1,7 @@
|
|||||||
# Brain of Reese — Master Plan (minimal)
|
# Brain of Reese — Master Plan (minimal)
|
||||||
|
|
||||||
> **Status:** Minimal re-land (2026-09-10). Phases 01–92 shipped;
|
> **Status:** Minimal re-land (2026-09-10). See §12 for current state.
|
||||||
> `todo/` holds 93–95 (see §12). Section numbers and anchor IDs match the
|
> Section numbers and anchor IDs match the
|
||||||
> previous full plan (in git history, commit `dac4a3e`) so the codebase's
|
> previous full plan (in git history, commit `dac4a3e`) so the codebase's
|
||||||
> `PLAN §…` / anchor comments stay valid — consult that revision or the
|
> `PLAN §…` / anchor comments stay valid — consult that revision or the
|
||||||
> phase records in `.agents/phases/complete/` for full detail and the
|
> phase records in `.agents/phases/complete/` for full detail and the
|
||||||
@@ -43,17 +43,17 @@ accounts (one admin + hand-out tokens is the model).
|
|||||||
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) |
|
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) |
|
||||||
| A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors |
|
| A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors |
|
||||||
| A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) |
|
| A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) |
|
||||||
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview; phase 94 adds sync-time folder summaries) |
|
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) |
|
||||||
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
|
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
|
||||||
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path — phase 95) |
|
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) |
|
||||||
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
|
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** (zero FTS hits **or** best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` (0.35)); an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, owner-confirmed, TODO L2a: lexical-only hits without vector support deflect). LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
|
||||||
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
|
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
|
||||||
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
|
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
|
||||||
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links, phase 76) |
|
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) |
|
||||||
| A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted |
|
| A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted |
|
||||||
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) |
|
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) |
|
||||||
| A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) |
|
| A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) |
|
||||||
| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error` (+ `tool_result`, owner-permitted extension — phase 95, not yet landed); no proxy buffering, no client caching on the stream |
|
| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error`, `tool_result`; no proxy buffering, no client caching on the stream |
|
||||||
| A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) |
|
| A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) |
|
||||||
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase |
|
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase |
|
||||||
| A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset |
|
| A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset |
|
||||||
@@ -65,17 +65,17 @@ accounts (one admin + hand-out tokens is the model).
|
|||||||
| # | Decision |
|
| # | Decision |
|
||||||
|---|----------|
|
|---|----------|
|
||||||
| B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) |
|
| B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) |
|
||||||
| B3 | **Revised (owner 2026-09-10, phase 93):** the semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted |
|
| B3 | The semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted |
|
||||||
| B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only |
|
| B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only |
|
||||||
| B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") |
|
| B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") |
|
||||||
|
|
||||||
**Key recent revisions (full log in the previous plan revision / phase
|
**Key recent revisions (full log in the previous plan revision / phase
|
||||||
records):** A10 — single-admin cookie auth (16), saved chats + shares +
|
records):** A10 — single-admin cookie auth, saved chats + shares +
|
||||||
staleness (50/51/53), client history budgets (74), access tokens (79).
|
staleness, client history budgets, access tokens.
|
||||||
A15 — `thinking` (17), `tool` (37), unlimited tool rounds under
|
A15 — `thinking`, `tool`, unlimited tool rounds under
|
||||||
`BOR_AGENT_MAX_ROUNDS` (45), `retry` (67), harness-aligned
|
`BOR_AGENT_MAX_ROUNDS`, `retry`, harness-aligned
|
||||||
`ls`/`read`/`grep` surface (70). A7 — the **retrieval** path's
|
`ls`/`read`/`grep` surface. A7 — the **retrieval** path's
|
||||||
never-truncated contract is unchanged; the phase-95 `read`-tool cap is
|
never-truncated contract is unchanged; the `read`-tool cap is
|
||||||
the only exception, owner-permitted 2026-09-10.
|
the only exception, owner-permitted 2026-09-10.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -111,7 +111,7 @@ BOR_GIT_SOURCES is the empty-table git-only fallback):
|
|||||||
| API routers | `app/api/` |
|
| API routers | `app/api/` |
|
||||||
| Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` |
|
| Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` |
|
||||||
| Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) |
|
| Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) |
|
||||||
| Migrations (0001–0015, all reversible) | `alembic/versions/` |
|
| Migrations (all reversible) | `alembic/versions/` |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -119,7 +119,7 @@ BOR_GIT_SOURCES is the empty-table git-only fallback):
|
|||||||
|
|
||||||
```
|
```
|
||||||
POST /api/chat {message, history?} (auth: require_user)
|
POST /api/chat {message, history?} (auth: require_user)
|
||||||
→ embed(question) [retried, phase 67 / A15 ext.]
|
→ embed(question) [retried, A15 ext.]
|
||||||
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
|
||||||
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
|
||||||
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
|
||||||
@@ -136,9 +136,9 @@ restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
|
|||||||
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
|
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
|
||||||
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
|
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
|
||||||
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
|
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
|
||||||
still counted). LLM retries (phase 67, A15 extension): `BOR_LLM_RETRIES`=3 × flat
|
still counted). LLM retries (A15 extension): retry count and delay are
|
||||||
`BOR_LLM_RETRY_DELAY`=5 s, **only before a request has streamed its
|
configurable (defaults: 3 retries, 5 s flat delay), **only before a
|
||||||
first output frame**.
|
request has streamed its first output frame**.
|
||||||
|
|
||||||
Retrieval details (A7/A8) and the persona/`<tools>` prompt contract: see
|
Retrieval details (A7/A8) and the persona/`<tools>` prompt contract: see
|
||||||
the previous plan revision §6 or `app/rag/retriever.py` /
|
the previous plan revision §6 or `app/rag/retriever.py` /
|
||||||
@@ -153,7 +153,7 @@ the full contracts. **Persona text changes through the plan, not in code
|
|||||||
## 5. Data Model & Chunking (§5 in older comments)
|
## 5. Data Model & Chunking (§5 in older comments)
|
||||||
|
|
||||||
Tables (full column detail: `app/models.py` — it is the living doc;
|
Tables (full column detail: `app/models.py` — it is the living doc;
|
||||||
migrations 0001–0015, each with a tested downgrade):
|
each migration has a tested downgrade):
|
||||||
|
|
||||||
| Table | Purpose |
|
| Table | Purpose |
|
||||||
|-------|---------|
|
|-------|---------|
|
||||||
@@ -162,12 +162,12 @@ migrations 0001–0015, each with a tested downgrade):
|
|||||||
| `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) |
|
| `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) |
|
||||||
| `steering_notes` | owner tuning notes → `<tuning>` section of every turn (char-budgeted) |
|
| `steering_notes` | owner tuning notes → `<tuning>` section of every turn (char-budgeted) |
|
||||||
| `kb_overview` | single row `id=1`: `lite`-generated KB outline → `<knowledge_base>` section (regenerated on KB change) |
|
| `kb_overview` | single row `id=1`: `lite`-generated KB outline → `<knowledge_base>` section (regenerated on KB change) |
|
||||||
| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB (phase 89) |
|
| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB |
|
||||||
| `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/<token>`); `sources_version` (stale when < current) |
|
| `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/<token>`); `sources_version` (stale when < current) |
|
||||||
| `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync |
|
| `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync |
|
||||||
| `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) |
|
| `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) |
|
||||||
| `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead |
|
| `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead |
|
||||||
| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 9 identity colors; NULL = default, B1 — grows to 17 vars in phase 93) |
|
| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 17 color variables; NULL = default, B1) |
|
||||||
|
|
||||||
Single-row tables use `id = 1` (the `kb_overview` precedent).
|
Single-row tables use `id = 1` (the `kb_overview` precedent).
|
||||||
|
|
||||||
@@ -195,13 +195,13 @@ teaching refusals, and scaffolding filter in `app/rag/agent.py` /
|
|||||||
`app/rag/scaffolding.py`. Empty prompt sections omit themselves — a
|
`app/rag/scaffolding.py`. Empty prompt sections omit themselves — a
|
||||||
no-notes/no-overview prompt is byte-identical to the pre-steering text.)
|
no-notes/no-overview prompt is byte-identical to the pre-steering text.)
|
||||||
|
|
||||||
Tool surface (harness-aligned, phase 70): **`ls`** (lists indexed docs
|
Tool surface (harness-aligned): **`ls`** (lists indexed docs
|
||||||
`source: X | path: Y | title: Z`; `path` = a source name — becomes a
|
`source: X | path: Y | title: Z`; `path` = a source name — drill-down
|
||||||
drill-down tree in phase 94), **`read`** (combined `source/path`
|
tree); **`read`** (combined `source/path` including the source name;
|
||||||
including the source name; appends the full document — gets a
|
appends the full document — with a truncation cap); **`grep`** (case-
|
||||||
truncation cap in phase 95), **`grep`** (case-insensitive **fixed
|
insensitive **fixed substring**, ≤20 `source/path:line: text` matches,
|
||||||
substring**, ≤20 `source/path:line: text` matches, optional one-doc
|
optional one-doc scope; a locator that adds no source/context — never a
|
||||||
scope; a locator that adds no source/context — never a regex).
|
regex).
|
||||||
Rejected calls get deterministic teaching refusals and consume a round;
|
Rejected calls get deterministic teaching refusals and consume a round;
|
||||||
`holder.tool_calls` counts executed calls only.
|
`holder.tool_calls` counts executed calls only.
|
||||||
|
|
||||||
@@ -235,8 +235,8 @@ Rejected calls get deterministic teaching refusals and consume a round;
|
|||||||
deflected) / error / KB-offline / stopped — has a defined UI, every
|
deflected) / error / KB-offline / stopped — has a defined UI, every
|
||||||
failure path re-enables its controls, a 300 s pre-token guard
|
failure path re-enables its controls, a 300 s pre-token guard
|
||||||
(`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts
|
(`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts
|
||||||
only visible time — phase 73), and **no auto-follow during a turn**
|
only visible time), and **no auto-follow during a turn**
|
||||||
(viewport moves only on user intent, phase 42). Pinned by unit tests
|
(viewport moves only on user intent). Pinned by unit tests
|
||||||
on the `app.js` state machine + the story E2E suites.
|
on the `app.js` state machine + the story E2E suites.
|
||||||
- Frontend house rules: `app.js` and siblings **never build HTML
|
- Frontend house rules: `app.js` and siblings **never build HTML
|
||||||
strings** (createElement + textContent); asset paths carry
|
strings** (createElement + textContent); asset paths carry
|
||||||
@@ -307,30 +307,21 @@ uv run python -m scripts.llm_probe # models + embedding dim
|
|||||||
|
|
||||||
The loop for git sources is *commit → re-run*. Uploads:
|
The loop for git sources is *commit → re-run*. Uploads:
|
||||||
`POST /api/git-sources/upload` unpacks + registers (202); the scan is
|
`POST /api/git-sources/upload` unpacks + registers (202); the scan is
|
||||||
deferred to Sync (phase 90). Source removal prunes on the next sync
|
deferred to Sync. Source removal prunes on the next sync.
|
||||||
(phase 69).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 12. Current State & Roadmap
|
## 12. Current State & Roadmap
|
||||||
|
|
||||||
- **Shipped: phases 01–92** (`.agents/phases/complete/` — read-only
|
- **Shipped phases:** `ls .agents/phases/complete/` — read-only
|
||||||
history; the shipped-features recap of the previous plan revision and
|
history; each phase's `00_phase.md` has the full detail. The shipped-
|
||||||
the README cover it). Migrations through 0015.
|
features recap of the previous plan revision and the README cover it.
|
||||||
- **Next up (`todo/`, in order — already authored via
|
- **Migrations:** `ls alembic/versions/` — all reversible.
|
||||||
`phase-authoring`):**
|
- **Next up (`todo/`):** Already authored via `phase-authoring`.
|
||||||
1. `93_theme_semantic_completion` — 8 semantic state colors join the
|
Read the directory listing in `.agents/phases/todo/` for the ordered
|
||||||
Theme-tab palette (17 total, B3 revised) + surface panels behind
|
task list; each file's `00_phase.md` has the full description.
|
||||||
every page head.
|
- **Next free phase number:** `max(completed phase numbers) + 1` —
|
||||||
2. `94_ls_tree_drilldown` — `ls` becomes a filesystem-style drill-down
|
computed from `.agents/phases/complete/`.
|
||||||
tree with sync-time `lite` folder summaries (`folder_summaries`
|
|
||||||
table, migration 0017) + real-model battery.
|
|
||||||
3. `95_read_truncation_cap` — capped `read` (default 128 000 chars
|
|
||||||
≈ 32k tokens, `BOR_READ_MAX_CHARS`), honest truncation notice to
|
|
||||||
the LLM + visible "(truncated — showing N of M chars)" marker
|
|
||||||
(new optional `tool_result` SSE frame — A15 extended to seven
|
|
||||||
event types).
|
|
||||||
- **Next free phase number: 96.**
|
|
||||||
- **Post-v1 hooks (deliberately not built):** HNSW index at scale;
|
- **Post-v1 hooks (deliberately not built):** HNSW index at scale;
|
||||||
inotify auto-import; more providers (the OpenAI-compatible client is
|
inotify auto-import; more providers (the OpenAI-compatible client is
|
||||||
the seam); multi-user accounts (the `require_user` split is the seam);
|
the seam); multi-user accounts (the `require_user` split is the seam);
|
||||||
@@ -357,7 +348,7 @@ phase dir moved to `complete/`.
|
|||||||
(`app/rag/retriever.py`) is the only overflow marker; char budgets are
|
(`app/rag/retriever.py`) is the only overflow marker; char budgets are
|
||||||
the pattern for anything sent to `lite`.
|
the pattern for anything sent to `lite`.
|
||||||
- **Byte-identical contracts are load-bearing:** deflected turns
|
- **Byte-identical contracts are load-bearing:** deflected turns
|
||||||
(A8), empty prompt sections, the B4 theme no-op, and the phase-33/54
|
(A8), empty prompt sections, the B4 theme no-op, and the
|
||||||
rewrite-only caching all have tests that assert byte-identity.
|
rewrite-only caching all have tests that assert byte-identity.
|
||||||
- **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib
|
- **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib
|
||||||
subprocess); everything else talks Postgres.
|
subprocess); everything else talks Postgres.
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
# VULNS.md — Known & Historically Found Vulnerabilities
|
||||||
|
|
||||||
|
Registry of security findings for Brain of Reese. **Full details, proofs of
|
||||||
|
concept, and the priority-ordered remediation task list live in
|
||||||
|
[`.agents/remediation_plan.md`](./remediation_plan.md)** (this file is the
|
||||||
|
quick-reference index; when a finding is fixed, update its status here and
|
||||||
|
append to the fixed log at the bottom).
|
||||||
|
|
||||||
|
Conventions:
|
||||||
|
- Statuses: `open` → `mitigated` (control in place) → `fixed` (remediated +
|
||||||
|
verified) → `accepted` (owner decision, risk consciously kept) →
|
||||||
|
`false positive` (intended behavior — never re-raise or "fix" without an
|
||||||
|
explicit owner decision).
|
||||||
|
- Prior-audit IDs keep their original numbering (SEC-0x = audit 2026-09-07,
|
||||||
|
SEC-14-xx = audit 2026-09-14).
|
||||||
|
|
||||||
|
## Open findings
|
||||||
|
|
||||||
|
| ID | Sev | Type (CWE) | Location | Status |
|
||||||
|
|----|-----|-----------|----------|--------|
|
||||||
|
| SEC-14-04 | Medium | In-flight SSE chat stream pins a DB connection for the whole turn → pool-exhaustion DoS (~15 streams, one token user) (CWE-770/400) | `app/api/chat.py` (request-scoped `get_db` across the `StreamingResponse`), `app/db.py` (default pool 5+10) | open |
|
||||||
|
|
||||||
|
**SEC-14-04 — one-line essence:** 16 slow `curl -N` streams on `/api/chat`
|
||||||
|
with any token hang every DB-backed route (default pool = 15 connections).
|
||||||
|
Remediation: short-lived sessions per DB step inside the stream (the
|
||||||
|
sync/upload house pattern), explicit pool kwargs, optional chat concurrency
|
||||||
|
cap — see `.agents/remediation_plan.md` §SEC-14-04.
|
||||||
|
|
||||||
|
## False positives — 2026-09-14 audit (owner: intended behavior, well known)
|
||||||
|
|
||||||
|
Reclassified by the owner on 2026-09-14: every 2026-09-14 finding **except
|
||||||
|
SEC-14-04** is intended, well-known behavior of a single-owner homelab app —
|
||||||
|
not a vulnerability to remediate. Kept here for the record so a future audit
|
||||||
|
(or a future agent) does not re-raise them. Do NOT "fix" these without an
|
||||||
|
explicit owner decision; the trade-offs below are deliberate.
|
||||||
|
|
||||||
|
| ID | Sev (as audited) | Type (CWE) | Location | Why it's intended |
|
||||||
|
|----|-----|-----------|----------|-------------------|
|
||||||
|
| SEC-14-01 | Medium | Unthrottled anonymous write surface → storage-exhaustion DoS (CWE-770/307) | `app/api/chats.py` (public POST/PUT `/api/chats`, POST `/{id}/share`); `app/core/rate_limit.py` covers login routes only | Public chat-save is the owner-locked phase-55 A1 design (the visitor saves their own conversation; auto-save is a normal traffic path). Homelab LAN trust model. |
|
||||||
|
| SEC-14-02 | Medium | Guest IDOR — chat-id holder can force-share / overwrite a private chat (CWE-639) | `app/api/chats.py` `share_chat`, `update_chat` | Owner-locked trust model: the unguessable uuid4 row id IS the credential (phase 51/55). Sharing/overwriting by an id holder is intended. |
|
||||||
|
| SEC-14-03 | Medium | Postgres on `0.0.0.0:5432` with weak documented creds `reese:reese` (CWE-790/668) | `compose.yaml` (db `ports`, `POSTGRES_PASSWORD`), `app/config.py` default `database_url`, `.env.example` | Deliberate homelab dev convenience; the trusted LAN is the security boundary and the creds are documented on purpose. |
|
||||||
|
| SEC-14-05 | Low | Cleartext transport + plaintext admin password (CWE-319/256) | `app/main.py` (`https_only=False`), `app/config.py` | Owner decision 2026-08-22: plaintext password and homelab HTTP are deliberate (TLS is a documented non-goal). |
|
||||||
|
| SEC-14-06 | Low | `debugpy` binds `0.0.0.0:5678` when `DEBUGPY=1` → LAN RCE if left on (CWE-1188) | `app/core/debugging.py` | Explicit opt-in dev feature (AGENTS.md rule 7); the wide bind is intentional for remote IDE attach. |
|
||||||
|
| SEC-14-07 | Low | No throttle on `/api/chat` for token users — leaked token = unlimited LLM turns (CWE-770) | `app/api/chat.py`, `app/rag/llm.py` | Tokens are capability credentials by design (phase 79); holders are trusted to use the models. |
|
||||||
|
| SEC-14-08 | Low | Importer reads source files whole, no per-file size cap (git/local sources; uploads are capped) (CWE-770) | `app/rag/importer.py:390` | Source dirs are admin-chosen trusted content; the never-truncated A7 contract implies whole-file reads. |
|
||||||
|
| SEC-14-09 | Low | Full question text + tool args in stdout logs and `query_log` (CWE-532) | `app/api/chat.py` per-turn line, `app/rag/agent.py` tool line | The full per-turn log line is a locked PLAN §9 requirement (AGENTS.md rule 10: "amply log"). |
|
||||||
|
| SEC-14-10 | Info | Admin git-source registry reaches arbitrary https/ssh/git@ hosts (admin-gated SSRF) (CWE-918) | `app/api/git_sources.py` `URL_RE` → `scripts/git_sync.py` | Admin-only surface; the admin already controls the host. Arbitrary remotes are the feature (phase 59/35). |
|
||||||
|
| SEC-14-11 | Info | No `Referrer-Policy` / `Permissions-Policy` (owner decision A4, phase 82) (CWE-538) | `app/core/security_headers.py` | Phase-82 decision A4 deliberately scoped headers to CSP + XFO + nosniff; no external requests exist (No-CDN). |
|
||||||
|
| SEC-14-12 | Info | `BOR_SESSION_SECRET` only checked non-empty — weak secret allows cookie forgery (CWE-340) | `app/core/auth.py` `ensure_admin_configured` | Operator-provisioned secret (generator command documented in `.env.example`); strength is the operator's choice. |
|
||||||
|
|
||||||
|
Note: `.agents/remediation_plan.md` (the audit report) still lists these with
|
||||||
|
remediation tasks — its task list items P1 (parts), P2, P4, P5, P6, P7
|
||||||
|
address findings now reclassified as false positives and are **dropped**
|
||||||
|
unless the owner reopens them. Only task P3 (SEC-14-04) remains actionable.
|
||||||
|
|
||||||
|
## Fixed (verified present in the 2026-09-14 audit)
|
||||||
|
|
||||||
|
| ID | Sev | What it was | Fixed by | Fixed phase |
|
||||||
|
|----|-----|-------------|----------|-------------|
|
||||||
|
| SEC-03 | Medium | No rate limit on failed sign-ins (`/api/login`, `/api/token-auth`) | Per-IP sliding window (10 fails / 15 min, shared counter, fail-open) — `app/core/rate_limit.py` | 81 |
|
||||||
|
| SEC-04 | Medium | No CSP / X-Frame-Options / nosniff → clickjacking of the admin UI | `SecurityHeadersMiddleware` — CSP `default-src 'self'; base-uri 'none'; frame-ancestors 'none'` + `X-Frame-Options: DENY` + nosniff on every response | 82 |
|
||||||
|
| SEC-05 | Medium | Unbounded anonymous payload on the public chat-save surface | Schema caps: ≤200 messages, 32 KB text/thinking, nested list caps (~13 MB worst-case) — `app/schemas.py` | 83 |
|
||||||
|
| SEC-08 | Medium | Docs-push 502 leaked git stderr with `user:pass@` credentials | `app/core/errors.sanitize_error` userinfo masker applied to all error surfaces | 84 |
|
||||||
|
| SEC-13 | Low | LLM error strings interpolated the raw base URL (possible embedded creds) | Same sanitizer on every LLM error f-string | 84 |
|
||||||
|
|
||||||
|
## Controls verified sound (2026-09-14 audit — no finding)
|
||||||
|
|
||||||
|
- SQL injection: all ORM / parameter-bound `text()`; FTS tsquery regex-constrained to `[a-z0-9.]` **and** bound as a parameter.
|
||||||
|
- XSS: escape-first `renderMarkdown()`, `textContent` for all data, strict CSP, no `unsafe-inline` (theme tag uses a sha256-hashed style-src exemption), all `innerHTML` sites audited static-or-escaped.
|
||||||
|
- Command injection: single list-argv subprocess funnel (`scripts/git_sync.run_git`), no `shell=True`; branch tokens shape-validated at boot; draft paths refuse absolute/`..`.
|
||||||
|
- Archive upload: traversal / absolute / drive-letter / symlink-hardlink escape / device-FIFO rejection, compressed + extracted byte caps (zip-bomb), atomic swap-in with rollback — `app/rag/archive_upload.py`.
|
||||||
|
- API tokens: 128-bit CSPRNG, SHA-256 hash stored (plaintext one-shot in the 201 body only), per-request live row check = revocation, generic 401s (no enumeration).
|
||||||
|
- Open redirect: login `?next=` restricted to same-origin relative paths.
|
||||||
|
- Agent tools: model-supplied `ls/read/grep` resolve against Postgres rows only (no FS/network), round cap bounds loops, fixed-substring grep (no ReDoS).
|
||||||
|
- Dependencies: `pip-audit` clean on the locked set (2026-09-14); bandit hits are false positives / dev scripts.
|
||||||
|
- Git history: no committed secrets across all 161 commits (2026-09-14 scan); `.env` gitignored.
|
||||||
|
|
||||||
|
## Re-audit triggers
|
||||||
|
|
||||||
|
Re-run the audit (or at least the relevant lens) when any of these land:
|
||||||
|
a reverse proxy or TLS in front of the app, a multi-user/auth rework,
|
||||||
|
changes to `app/api/chats.py` write gating, the compose port bindings, the
|
||||||
|
subprocess/git surface, the upload pipeline, or any dependency with a
|
||||||
|
published CVE (`uvx pip-audit`).
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
- 2026-09-14 (owner reclassification) — all 2026-09-14 findings except
|
||||||
|
SEC-14-04 reclassified as **false positives** (intended behavior, well
|
||||||
|
known). Open count: 12 → 1.
|
||||||
|
- 2026-09-14 — second audit: 12 new findings (4 Medium / 4 Low / 4 Info),
|
||||||
|
5 prior findings verified fixed. Full report: `.agents/remediation_plan.md`.
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Phase 100 — Every page matches the RAG page's width (the 72rem container, at every viewport)
|
||||||
|
|
||||||
|
**Source:** Owner request (chat, 2026-09-12) — "The theme, tuning, and chat pages are still pretty narrow, I want you to match the width of the RAG page for all other pages to keep things consistent."
|
||||||
|
**Story:** n/a (owner request — supersedes the PLAN §7.1 "centered 46rem chat column (2× = 92rem at ≥1500px)" rule and the phase-58/27/91 form-column caps; the width lineage: `58_wide_desktop_column`, `86_history_page_width`).
|
||||||
|
**Context:** `frontend/assets/styles.css`: `.container` is the shared 72rem centered column (`width: 100%; max-width: 72rem; padding-inline: 1.25rem;`). The RAG view's `.sources-shell` (and `.git-sources-shell` / `.history-shell` / `.tokens-shell`) carry NO max-width — their content is the full 72rem container: **that is the width the owner wants everywhere**. The narrow pages: `:root { --chat-column: 46rem; }` caps `.chat-shell` (the chat view — the div IS `.container.chat-shell`, the later rule overriding the 72rem cap), `.shared-shell` (shared.html), `.doc-md` (the document viewer's reading column) and `.doc-summary:has(+ .doc-md)` (exactly four selectors use the token); `@media (min-width: 1500px) { :root { --chat-column: 92rem; } .doc-shell { max-width: calc(var(--chat-column) + 2 * 1.25rem); } }` (phase 58 — the wide-desktop doubling + the standalone document page's cap lift); and three hard-coded `max-width: 46rem` form columns: `.tuning-shell`, `.theme-shell` (phase 91), `.doc-edit-shell` (phase 59). Width pins live in: `tests/unit/test_wide_column_css.py` (the phase-58 contract file — 46rem base, the 92rem media block, the four-token count, the three-hard-coded-46rem negative pin, the provenance comments), `tests/unit/test_markdown_tables.py`, `test_shared_page.py`, `test_pinned_composer.py`, `test_doc_edit_screen.py`, `test_remove_confirm_modal.py`, `test_source_ignore_paths.py` (grep `46rem|92rem|chat-column` under `tests/` for the authoritative list — some hits are fixture text, judge per pin), and E2E measured-width suites: `tests/e2e/test_wide_desktop_column.py` (the measured 1472px/736px contract), `test_document_viewer.py`, `test_save_share_ux.py`, `test_sticky_navbar.py`, `test_markdown_tables.py`, `test_responsive_polish.py` (mobile no-overflow).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every view and page renders its content at the RAG page's width — the full 72rem container at every viewport (centered, with the container's 1.25rem gutters): the chat column, the Tuning and Theme form columns, the doc-edit screen, the shared page, and the document viewer all widen to 72rem; the phase-58 ≥1500px 92rem doubling and the 46rem form caps are retired; the mobile (≤640px) and tablet layouts are unchanged (everything was already full-width below the caps).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `86_history_page_width` / `58_wide_desktop_column` (complete) — the width lineage being revised (their suites are updated IN THIS PHASE, the phase-97 task-07/08 precedent — asserted behavior preserved, only the width contract changes).
|
||||||
|
- No code dependencies beyond `frontend/assets/styles.css` (CSS-only — `app/`, `frontend/*.html`, and every JS module are untouched).
|
||||||
|
|
||||||
|
## Decisions recorded here (owner instruction 2026-09-12 — PLAN.md is being redone by the owner)
|
||||||
|
- **D1 — one width for everything:** the reading/form columns ALL ride the 72rem container. `:root { --chat-column: 72rem; }` (the token stays — the four reading-column selectors keep `max-width: var(--chat-column)`, so `.doc-md`/`.shared-shell`/`.chat-shell`/the doc-summary panel follow automatically); the `@media (min-width: 1500px)` block is DELETED entirely (the 92rem doubling AND the `.doc-shell` cap lift — with the token at 72rem the standalone document page's `.container.doc-shell` simply keeps the 72rem cap); the three hard-coded `max-width: 46rem` caps (`.tuning-shell`, `.theme-shell`, `.doc-edit-shell`) are REMOVED (plus their `margin-inline: auto` centering — the `.container` ancestor centers; the shells keep their flex-column + gap rules, matching `.sources-shell` exactly). Net effect: ZERO literal `max-width: 46rem` rules remain in the file (the negative pin flips). This SUPERSEDES PLAN §7.1's 46rem/92rem rule and the phase-58 owner instruction of 2026-08-31 (recorded here per the phase-94/96/97 convention — the owner's new instruction wins; the PLAN.md anchor table gets the owner's sign-off separately).
|
||||||
|
- **D2 — the document viewer's modal is untouched:** the same-page doc modal's 1100px panel stays its effective ceiling (`.doc-md` at 72rem is wider than the panel's inner width, so the panel binds — no panel resize, no modal change). The standalone `document.html` page rides the 72rem container like every other page.
|
||||||
|
- **D3 — CSS-only, contracts intact:** no HTML change (the B4 byte-identical theme contract is unaffected — the served HTML is unchanged; only `styles.css` bytes change), no JS change, no endpoint change; A19's `?v=<deploy-token>` versioning busts the old CSS automatically. The `≤640px` mobile block is UNCHANGED (at 360px the container is already 100% — the caps never bound there; the mobile squeeze rules for chat/shared/tokens/theme keep working).
|
||||||
|
- **D4 — the comment sweep:** every stale "46rem base, 92rem at >=1500px" / "46rem column contract" / phase-58-provenance comment in `styles.css` is updated to the new contract (the owner instruction 2026-09-12 as provenance at the token + the note that the wide-desktop doubling was retired) — the house rule that comments carry the contract (and `test_wide_column_css.py` pins the comment states).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_width_css.md` — the CSS change: the token to 72rem, the media block deleted, the three form caps removed, the comment sweep
|
||||||
|
2. `02_width_unit_pins.md` — the unit pin updates: `test_wide_column_css.py` rewritten to the new contract + the grep-sweep of the other unit files
|
||||||
|
3. `03_e2e_width_sweep.md` — `test_wide_desktop_column.py` rewritten to the measured 72rem contract + the other E2E width pins + the regression sweep + the atomic commit
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_wide_column_css.py` REWRITTEN to the new contract (the pins below); every other unit file whose pins reference the old widths updated (the grep sweep — `46rem|92rem|chat-column` under `tests/unit/`, judged per pin: a width assertion updates, fixture text does not).
|
||||||
|
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_wide_desktop_column.py -v --no-cov` in isolation — the phase's dedicated width suite, REWRITTEN to the measured 72rem contract (the phase-97 task-07/08 precedent for updating a completed phase's suite when its contract changes); the other measured-width suites updated in place.
|
||||||
|
- Coverage: **>90%** on `app/` (untouched — the full-suite gate still runs).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] at any viewport ≥ ~1200px, the chat, shared, document-viewer, Tuning, Theme, and doc-edit content columns measure the 72rem container width (E2E-measured, ±4 px) — identical to the RAG page's; below that, everything is full-width as today; 360px has no horizontal overflow
|
||||||
|
- [ ] `styles.css` has ZERO literal `max-width: 46rem` rules and NO `@media (min-width: 1500px)` block; the four reading-column selectors still cap with `var(--chat-column)` (now 72rem)
|
||||||
|
- [ ] the B4 byte-identical theme contract holds (no HTML change — the theme no-op test suite green); the mobile squeeze layouts green
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; ruff + pyright clean
|
||||||
|
- [ ] one atomic Conventional Commit, `--no-gpg-sign` (e.g. `feat(ui): match every page to the RAG page's 72rem width`)
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 01 — The CSS: one 72rem column everywhere (token, media-block deletion, form caps, comment sweep)
|
||||||
|
|
||||||
|
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`styles.css` carries the new width contract (D1–D4 of `00_phase.md`): `--chat-column` is 72rem, the phase-58 ≥1500px block is gone, the three 46rem form caps are removed, and every stale width comment tells the new story.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css`:
|
||||||
|
- `:root` — `--chat-column: 72rem;` (was 46rem). The token's provenance comment updates: the owner instruction 2026-09-12 ("match the width of the RAG page for all other pages") + the note that it now EQUALS the `.container`'s 72rem cap (the reading columns ride the same width as every other page; the phase-58 2026-08-31 instruction is superseded).
|
||||||
|
- DELETE the entire `@media (min-width: 1500px) { … }` block at the bottom of the file (the `:root { --chat-column: 92rem; }` doubling + the `.doc-shell { max-width: calc(var(--chat-column) + 2 * 1.25rem); }` lift — D1). Its phase-58 block comment goes with it; the deletion is total (the `prefers-reduced-motion` block that follows it stays).
|
||||||
|
- `.tuning-shell` — remove `max-width: 46rem;` and `margin-inline: auto;` (keep `display: flex; flex-direction: column; gap: 1.25rem; flex: 1;` — the shell now matches `.sources-shell` exactly); update its phase-27 comment (the "Same width discipline as the chat column — a centered, capped column" wording) to the full-container-width contract (owner instruction 2026-09-12).
|
||||||
|
- `.theme-shell` — the same removal (phase-91's "form column" rationale is superseded — the owner wants it wide); comment updated.
|
||||||
|
- `.doc-edit-shell` — the same removal (phase-59's form-column cap is superseded); comment updated.
|
||||||
|
- The comment sweep (D4): every remaining reference to "46rem base", "92rem at >=1500px", "the 46rem chat-column width", "the >=1500px 92rem override", and the stale `≤640rem`/`≤640px`-era "46rem column contract" wording in the reading-column regions (`.chat-shell`'s block comment, `.shared-shell`'s inline comment + the shared-page region comment, `.doc-md`'s comment, the doc-summary panel comment, the `≤640px` shared-page note "the shell keeps its base 46rem column (the >=1500px 92rem override never applies here…)", and any others the grep `46rem|92rem|1500px` finds) — rewrite each to the new contract (72rem at every width; the wide-desktop doubling retired 2026-09-12). The `.shared-shell` comment must keep the "the PLAN §7 centered chat column" lineage sentence only if it still reads true — it does NOT (PLAN §7.1's 46rem rule is superseded): reword to "the shared page reads exactly like the chat page — both at the 72rem container width".
|
||||||
|
- Do NOT touch: the `.container` rule itself, the `≤640px` mobile block (the squeeze rules), the doc modal's 1100px panel, any non-width CSS, any `app/` or HTML/JS file (D3 — CSS-only).
|
||||||
|
2. Verify with a grep: `grep -n "46rem\|92rem\|1500px" frontend/assets/styles.css` returns NOTHING (or only the new provenance comment's mention of the retired 2026-08-31 instruction, if phrased without the literal values — prefer phrasing it without the literals so the grep is clean).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pin updates land in task 02 — this task's own check is the grep verification above + a visual sanity pass (`uv run uvicorn app.main:app --reload`, compare the chat/tuning/theme pages to the RAG page at 1280px and 1920px — the content columns match; the doc modal looks unchanged).
|
||||||
|
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `styles.css`: `--chat-column: 72rem` in `:root`; no `@media (min-width: 1500px)` block; zero literal `max-width: 46rem` rules; the three shells (tuning/theme/doc-edit) structurally match `.sources-shell` (flex column, no cap)
|
||||||
|
- [ ] the width grep is clean (task 02's pins will enforce the comment states — do not leave a stale "46rem base / 92rem" claim anywhere)
|
||||||
|
- [ ] no HTML/JS/app change (D3)
|
||||||
|
- [ ] the full unit suite is green EXCEPT the width pins task 02 rewrites (run it to enumerate the breakage — the list must be exactly the files named in task 02)
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — The unit pins: `test_wide_column_css.py` rewritten + the grep-sweep of the other width pins
|
||||||
|
|
||||||
|
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The unit layer pins the NEW width contract — `test_wide_column_css.py` becomes the phase-100 contract file, and every other unit pin that asserted the old widths is updated (fixture text that merely mentions a width is left alone).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_wide_column_css.py` — REWRITE (keep the file's balanced-brace `_rule_block` helpers + its module purpose — "pin the styles.css markers the width contract depends on"):
|
||||||
|
- **The token:** `:root` declares `--chat-column: 72rem` with the provenance comment citing the owner instruction 2026-09-12 (replaces the 2026-08-31 pin).
|
||||||
|
- **The retired doubling:** NO `@media (min-width: 1500px)` block anywhere in the file (`css.count("@media (min-width: 1500px)") == 0`), and no `--chat-column: 92rem` assignment (`css.count("--chat-column: 92rem") == 0`).
|
||||||
|
- **The four reading columns:** `.chat-shell`, `.shared-shell`, `.doc-md`, `.doc-summary:has(+ .doc-md)` each cap with `max-width: var(--chat-column)` — and exactly those four rules use the token (`css.count("max-width: var(--chat-column)") == 4` — the token count pin survives, the VALUE changes).
|
||||||
|
- **The flipped negative pin:** ZERO literal `max-width: 46rem` rules remain (`css.count("max-width: 46rem") == 0`) — the three former form columns (`.tuning-shell`, `.theme-shell`, `.doc-edit-shell`) must NOT carry a max-width at all (each `_rule_block` free of `max-width`), and they carry no `margin-inline: auto` (the `.container` centers).
|
||||||
|
- **The comment pins:** the stale claims are gone (`"≤46rem" not in css`, `"92rem at >=1500px" not in css`, `"46rem base" not in css`); the provenance appears at the token (owner instruction 2026-09-12); the chat-shell / shared-shell comments name the 72rem-everywhere contract (pin the key phrases, not whole paragraphs — the house "pin the contract words" style).
|
||||||
|
- Update the module docstring to describe the new pins.
|
||||||
|
2. The grep sweep — `grep -rln "46rem\|92rem\|chat-column" tests/unit/` (the authoritative list; `test_wide_column_css.py` is done above). For each hit, judge per pin: a pin that ASSERTS the old width (a literal in an `assert`, a `_rule_block` expectation, a counted occurrence) is updated to the new contract (72rem / no cap / the new comment wording); a hit that is fixture text, a docstring example, or an unrelated measurement (e.g. a fixture document mentioning "46rem", or a pin about an UNRELATED rule's max-width) is left untouched. Expected candidates (from the phase-100 `00_phase.md` context): `test_markdown_tables.py`, `test_shared_page.py`, `test_pinned_composer.py`, `test_doc_edit_screen.py`, `test_remove_confirm_modal.py`, `test_source_ignore_paths.py` — the actual set is whatever the grep + the run reveal.
|
||||||
|
3. Run `uv run pytest tests/unit/ -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the rewritten `test_wide_column_css.py` + the swept pins ARE this task's test layer (they guard the CSS bytes — the house source-pin pattern).
|
||||||
|
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `test_wide_column_css.py` pins the phase-100 contract (72rem token + 2026-09-12 provenance, no 1500px block, no 92rem, the four token selectors, ZERO literal 46rem rules, the three shells cap-free, the comment states)
|
||||||
|
- [ ] the grep sweep is complete — every remaining `46rem|92rem|chat-column` reference under `tests/unit/` is either the new contract's pin or non-width fixture text (list the judgment in the commit's test-run notes / the final summary)
|
||||||
|
- [ ] `uv run pytest tests/unit/ -q` green
|
||||||
|
- [ ] no behavior change in completed work (the non-width pins in the swept files still pass unchanged)
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 03 — The E2E width sweep: measured 72rem everywhere + updated suites + regressions + commit
|
||||||
|
|
||||||
|
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The browser proves the owner's ask: at every desktop viewport the chat, shared, document-viewer, Tuning, Theme, and doc-edit content columns measure the SAME width as the RAG page (the 72rem container) — the phase's dedicated width suite is rewritten to that contract and the other measured-width suites are updated in place.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_wide_desktop_column.py` — REWRITE (the phase-58 measured-width suite becomes the phase-100 one — the phase-97 task-07/08 precedent: a completed phase's suite updates in place when its contract changes; keep its seeding helpers — the fixture markdown doc for the viewer, the shared-chat helper):
|
||||||
|
- **`test_all_columns_match_the_rag_page`** (the core pin): viewport 1280×800 — measure the `.chat-shell` bounding-box width on `/`, the `.tuning-shell` width on `/tuning.html` (admin), the `.theme-shell` width on `/theme.html` (admin), and the RAG page's `.sources-shell` width on `/sources.html` (admin): all four EQUAL (± 4 px) — the owner's "match the width of the RAG page" as one assertion. Then viewport 1920×1080: the same four still EQUAL each other (± 4 px) and each ≈ 1152 px (72rem at the 16px root, ± 4 px — the box includes the container's 2×1.25rem padding, border-box).
|
||||||
|
- **`test_reader_columns_wide`** (the token's consumers): viewport 1920×1080 — `.doc-md` in the opened fixture doc ≈ 1058 px (the 1100px border-box modal panel minus its 1px borders and `.doc-modal-content`'s 2×1.25rem padding is the binding ceiling there — 1100 − 2 − 40; D2: pin that the modal is UNCHANGED by asserting the modal panel's own width is still ~1100 px, not that `.doc-md` is 1152); `/shared/<token>`'s `.shared-shell` ≈ 1152 px; the standalone `document.html?source=…&path=…` page's `.doc-md` ≈ 1112 px (its `.container.doc-shell` is the 72rem container → inner content = 1152 − 40 px padding).
|
||||||
|
- **`test_narrow_unchanged`** (the no-regression leg): viewports 360×800 and 900×600 — `.chat-shell` width == viewport width (100%, no overflow: `document.documentElement.scrollWidth <= viewport width` — the `test_responsive_polish.py` assertion style), and at 900px the chat/tuning/theme shells are all 900px wide (equal to each other — the cap never bound below 72rem anyway).
|
||||||
|
2. The other measured-width E2E suites — update in place (each in isolation, `--no-cov`, DB up): `test_document_viewer.py` (its `≤ 736 px` standalone `.doc-md` pin → the ≈ 1112 px contract; keep its non-width assertions byte-identical), `test_save_share_ux.py` (any 92rem/46rem measured or source-pinned width references), `test_sticky_navbar.py` (its width-related pins only), `test_markdown_tables.py` (the e2e table-width pins against the reading column), `test_responsive_polish.py` (the mobile pins — expected UNCHANGED; run to confirm). Judge per assertion: a width measurement updates, a layout/behavior assertion never does.
|
||||||
|
3. The theme no-op contract (D3): run `test_ui_customization.py` + `test_admin_theme_tab.py` in isolation — no HTML changed, so the B4 byte-identical pins must pass UNCHANGED (if one fails, the CSS change touched something it should not — fix the cause, not the pin).
|
||||||
|
4. Regression sweep (each in isolation, `--no-cov`): the five suites from step 2 (updated) + `test_chat_rag.py` (the chat layout smoke) + `test_theme_semantic_completion.py` (the phase-93 page-head panels at the new width).
|
||||||
|
5. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `100_page_width_consistency` → `.agents/phases/complete/`; one atomic commit:
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(ui): match every page to the RAG page's 72rem width"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_wide_desktop_column.py -v --no-cov` green in isolation — the phase's dedicated suite.
|
||||||
|
- Coverage: **>90%** on `app/` (the full-suite gate).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the rewritten width suite passes in isolation: chat == tuning == theme == RAG at 1280 AND 1920 (± 4 px, ≈ 1152 px at 1920); the doc modal unchanged (~1100 px); shared + standalone document pages ≈ 1152/1112 px; 360px overflow-free
|
||||||
|
- [ ] the other measured-width suites pass in isolation (updated in place); the theme no-op suites pass UNCHANGED
|
||||||
|
- [ ] the regression sweep passes in isolation
|
||||||
|
- [ ] full suite green, coverage >90%, ruff + pyright clean
|
||||||
|
- [ ] phase dir moved to `complete/`, one atomic `--no-gpg-sign` Conventional Commit
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Phase 101 — Tokens page: active/revoked table split, per-table search, required name, one-click regenerate
|
||||||
|
|
||||||
|
**Source:** Owner request (chat, 2026-09-12) — "The tokens page should move revoked tokens to a separate table below the active ones. Generating a token should not be possible without giving it a name. Both active and revoked token tables should be searchable. I should be able to regenerate active tokens with the click of a button."
|
||||||
|
**Story:** n/a (owner request — extends `79_api_tokens`: the model/service/API on phase 79 tasks 01–03, the single-table admin view on task 06, the once-block + two-step revoke UI, the E2E `test_api_tokens.py`).
|
||||||
|
**Context:** `api_tokens` (migration 0012): `id` UUID, `label` (1–120, NOT unique), `token_hash` (sha256 of the full `bor_…` string, unique), `created_at`, `last_used_at`, `revoked_at` (NULL = active). `app/core/tokens.py` service: `generate_token`, `hash_token`, `create_token(db, label) -> (row, plaintext)`, `find_active_by_token`, `mark_used`, `revoke(db, token_id) -> bool` (stamps `revoked_at` only when unset; False when the row is missing). Admin API (`app/api/tokens.py`, router-wide `require_admin`): `POST /api/tokens` (201 `TokenCreated{id, label, token, created_at}` — the ONLY response carrying the plaintext, owner-locked A4), `GET /api/tokens` (200 `TokenList{tokens: [TokenListItem{id, label, created_at, last_used_at, revoked: bool}]}` newest-first — `revoked_at` is NOT returned today), `POST /api/tokens/{id}/revoke` (204 idempotent, 404 unknown). The view (`frontend/index.html` `#view-tokens` + `frontend/assets/tokens.js`, phase-76 fold): the create row (`#token-label` + `#token-generate` — a BLANK label currently sends the fallback `"token"`), the shown-once block (`#token-once` — plaintext in the read-only field's value ONLY, hidden + wiped on the next `loadTokens()` / re-show), ONE full-width table (`#tokens-table` / `#tokens-tbody`, columns Label | Created | Last used | Status (active em-dash vs the rose `.stale-pill` Revoked) | Actions (the two-step inline Revoke — the `history-confirm-*` pattern; revoked rows carry no action)), `#tokens-empty-row`, the `#tokens-status` live region, the `bor:view-refresh` re-show contract (re-entrant `loadTokens()`), the whoami gate (anonymous → the gate, NO fetch). The phase-79 E2E `tests/e2e/test_api_tokens.py` pins the single-table layout — including REVOKED rows living in `#tokens-tbody` (e.g. `test_revocation_closes_the_door` asserts the `.stale-pill` there) — and `auth_helpers.login_with_token(page, app_url, token)` drives the real gate (the regenerate proof reuses it).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The Tokens view splits its list into an ACTIVE table and a REVOKED table below it (each independently searchable), a token can no longer be generated without a name, and every active token carries a one-click **Regenerate** (a house two-step confirm, then the new plaintext shown once) that rotates the credential — the old token is revoked and lands in the revoked table, the new one (same label) takes its place in the active table.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `79_api_tokens` (complete) — the model, the service, the admin API, the A4 plaintext-once contract, the view module, the E2E helper (`login_with_token`).
|
||||||
|
- Queue order only: `98_sync_summary_visibility`, `99_kb_tree_table_and_back_nav`, `100_page_width_consistency` (todo) — no code dependency (different views; the tokens table sits in the full-width `.container` already — phase 100 changes nothing here).
|
||||||
|
|
||||||
|
## Decisions recorded here (owner review — PLAN.md is being redone by the owner)
|
||||||
|
- **D1 — the split + the column contracts:** the active table keeps the phase-79 shape MINUS the Status column (a table that is all-active needs no status) — **Label | Created | Last used | Actions** (Actions visually-hidden header, as today). The revoked table (below, its own `.table-wrap` + a visible `Revoked tokens` heading + its own search input) is **Label | Created | Last used | Revoked** (the `revoked_at` date — locale date+time, full ISO on hover, the house `tokens-date-cell` language). The table IS the status — the em-dash / `.stale-pill` column disappears from BOTH tables (the pill's meaning is now the table's position). The revoked section (heading + search + table) is HIDDEN when there are no revoked tokens (an empty table is noise); the active table keeps its empty-state row. Newest-first in both (the server order).
|
||||||
|
- **D2 — regenerate = rotation, atomic:** new endpoint `POST /api/tokens/{token_id}/regenerate` (admin, 201 `TokenCreated`): in ONE transaction — stamp `revoked_at` on the old row (the `revoke` service primitive) and `create_token(db, old.label)` (same label — the hand-out name persists) — returning the NEW row + the new plaintext exactly once (A4 preserved: the plaintext is per-token, shown once, never re-shown; the old plaintext was already gone). Unknown id → 404 `token not found`; the row is ALREADY revoked → 409 `token already revoked` (a dead token cannot be rotated — the button never offers it, the API stays honest). UI: the active row's Actions cell gains **Regenerate** (before Revoke) — the house two-step inline confirm (first click swaps to `Regenerate? The current token is revoked. [Yes] [No]`, focus to Yes — a destructive rotation deserves the same confirm weight as Revoke; ONE button starts it, per the owner's "click of a button"). On 201: the re-entrant `loadTokens()` runs (the old row moves to the revoked table, the new row lands in the active one), the once-block reveals with the new plaintext + Copy, the live region reads `Regenerated "<label>" — copy the new token now; it won't be shown again.` Failure (409 → the row re-renders in the revoked table + a line; other → neutral retry copy, the button restores — the phase-55 convention).
|
||||||
|
- **D3 — a name is required:** `generateToken()` trims the label and, when empty, does NOT send — the live region reads `Give the token a name first.`, the label input re-focuses, the Generate button's §7.4 lifecycle is untouched (it was never disabled — the request simply doesn't happen). The `|| "token"` fallback is DELETED (the create row's comment + placeholder update: the name is the hand-out identity, not an optional decoration). The server's 422 on blank/over-long labels stands unchanged (defense in depth — the UI just stops volunteering the fallback).
|
||||||
|
- **D4 — search = client-side, per table, label-scoped:** one `type="search"` input per table (`#token-search-active` above the active table, `#token-search-revoked` above the revoked table — inside the revoked section, so it hides with it), placeholder `Search active tokens…` / `Search revoked tokens…` (labeled via `aria-label` — the house visually-hidden-label language for single-purpose inputs). Filtering: case-insensitive SUBSTRING over the label, applied on the `input` event to the CURRENTLY rendered rows (no fetch — toggle each row's `hidden`), and RE-APPLIED after every `loadTokens()` (the query state survives re-renders / re-shows — the phase-77 contract). Zero visible matches with a non-empty query → a per-table no-match row (`No tokens match "<query>".` — the query text is textContent-filled; distinct from the active table's true empty-state row `No tokens yet — …`). Empty query → all rows visible, no-match row hidden.
|
||||||
|
- **D5 — the list API gains `revoked_at`:** `TokenListItem` adds `revoked_at: datetime | None = None` (wire-additive — null for active tokens; the revoked table's Revoked column needs the actual timestamp, and nothing else changes on the wire). The existing `revoked: bool` stays (the client's table split key).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### The API (task 01)
|
||||||
|
- `app/core/tokens.py` — `regenerate_token(db, token_id) -> tuple[ApiToken, str] | None` (the service surface, unit-testable without FastAPI): fetch the row (`db.get`) — missing → return `None`; `revoked_at` set → raise `TokenAlreadyRevoked` (a new module-level exception, one sentence docstring: a dead token cannot be rotated); else `revoke(db, row.id)` (stamp) + `create_token(db, row.label)` (new row + plaintext) + `db.flush()` (the caller commits — the `create_token` convention: the service flushes, the endpoint owns the commit). ONE commit covers both writes (the rotation is atomic — a create failure rolls the revoke back with it).
|
||||||
|
- `app/api/tokens.py` — `POST /{token_id}/regenerate` (201, `response_model=TokenCreated`, the router-wide `require_admin` covers it): `None` → 404 `token not found` (the revoke endpoint's exact message); `TokenAlreadyRevoked` → 409 `token already revoked`; success → commit + refresh + `TokenCreated(id, label, token, created_at)` of the NEW row. The list endpoint passes `revoked_at=row.revoked_at` into `TokenListItem` (D5). Docstrings: the rotate-once semantics (the docstring-carries-the-contract rule).
|
||||||
|
- `app/schemas.py` — `TokenListItem.revoked_at: datetime | None = None` (the docstring notes D5).
|
||||||
|
- Unit (`tests/unit/test_tokens.py`): the `regenerate_token` matrix — active row → (new row, new plaintext) with the SAME label, the old row stamped (its original `revoked_at` semantics: first stamp), the new plaintext well-formed (`bor_` + 32 hex, ≠ the old hash); missing id → None; revoked id → raises; the new row's hash is the sha256 of the new plaintext (the lookup round-trips via `find_active_by_token`); the old token no longer authenticates (`find_active_by_token(old_plaintext)` → None).
|
||||||
|
- Integration (`tests/integration/test_tokens_api.py`): the endpoint matrix — 201 (the 201 body carries the NEW plaintext exactly once; the old row `revoked: true` + `revoked_at` non-null in the next list; the new row active, same label, newer `created_at`); 404 unknown id; 409 already-revoked; 403 anonymous + 403 token-user (the router-wide gate — the existing pattern); the list shape now carries `revoked_at` (null active / timestamp revoked, ISO-8601).
|
||||||
|
|
||||||
|
### The view (tasks 02 + 03)
|
||||||
|
`frontend/index.html` `#view-tokens` (the static skeleton — tokens.js fills the tbodys):
|
||||||
|
- The create row: the input's `aria-label` → `Token name`, the placeholder → `e.g. alice — required` (D3); the comment updated (no fallback — the name is required).
|
||||||
|
- A search row above the active table: `<input type="search" id="token-search-active" aria-label="Search active tokens" placeholder="Search active tokens…">` (the house input language — a new `.token-search` class in the CSS, full width, the `#tune-note`/archive-upload input styling family).
|
||||||
|
- The active table: thead drops the Status `<th>` (four columns now: Label | Created | Last used | the visually-hidden Actions); `#tokens-empty-row` → `colspan="4"`, copy unchanged; a NEW hidden `#tokens-no-match-row` (`colspan="4"`, its text JS-filled — ships empty).
|
||||||
|
- A NEW revoked section BELOW the active table's wrap (ships `hidden`): `<h2 id="tokens-revoked-heading" class="tokens-revoked-heading">Revoked tokens</h2>` (a visible sub-heading — house typography: `font-family: var(--mono); font-size: 1rem; color: var(--brand-ink);` on the surface-less page background, AA pair, matching the phase-97 `.kb-level h2` voice) + the search input `#token-search-revoked` + `<div class="table-wrap tokens-table-wrap" id="tokens-revoked-wrap" role="region" aria-label="Revoked tokens" tabindex="0">` + `<table class="tokens-table" id="tokens-revoked-table">` (visually-hidden caption `Revoked tokens — newest first`; thead Label | Created | Last used | Revoked; `tbody#tokens-revoked-tbody` + the hidden `#tokens-revoked-no-match-row` `colspan="4"`).
|
||||||
|
- `frontend/assets/tokens.js` (task 02 — the split + search + required name):
|
||||||
|
- `loadTokens()` — the fetched list splits: `revoked === false` → `#tokens-tbody`, `true` → `#tokens-revoked-tbody` (the server's newest-first order kept per table); the revoked SECTION (`#tokens-revoked-heading` + search input + wrap) is shown iff the revoked table has ≥ 1 row; the active empty-state row logic stands (0 active rows → the empty row — a 0-row fetch with revoked tokens shows the empty active table + the populated revoked section: honest states); AFTER rendering, both search filters re-apply (D4 — the queries persist in module state `activeQuery` / `revokedQuery`, initialized `""`, never reset by a load).
|
||||||
|
- `makeRow(tok, { table })` — the active variant: Label | Created | Last used | Actions (Regenerate — task 03 — + Revoke, the existing two-step, unchanged); the revoked variant: Label | Created | Last used | Revoked (the `revoked_at` locale date+time, full ISO in `title` — the `tokens-date-cell` class) + NO actions. The `title` hover attributes carry over (label, created, last-used, revoked).
|
||||||
|
- The search: per-table `applyFilter(tbody, noMatchRow, query, tableLabel)` — case-insensitive `label.toLowerCase().includes(query.toLowerCase())` over the data rows (the no-match/empty rows excluded); rows toggle `hidden`; `noMatchRow` visible ⟺ query non-empty AND zero data rows visible (its `<td>` textContent = `No tokens match "<query>".` — the query inside the quotes is textContent, never HTML). The `input` listeners set the module query + apply (no fetch). The active search input is revealed with the create row (admin branch); the revoked one with the revoked section.
|
||||||
|
- `generateToken()` (D3): `const label = (labelInput ? labelInput.value : "").trim();` — `if (!label) { announce("Give the token a name first."); if (labelInput) labelInput.focus(); return; }` — the `|| "token"` fallback deleted; the rest of the function (the 201 once-block reveal, the list re-load FIRST, the label clear, the error lines) unchanged.
|
||||||
|
- `frontend/assets/tokens.js` (task 03 — regenerate): `makeRegenerateControl(tok, row)` mirroring `makeRevokeControl` (the `history-confirm-*` swap, focus to Yes) — Yes → `POST /api/tokens/<id>/regenerate` → on 201: `loadTokens()` (the re-entrant load — the old row relocates, the new row lands) → the once-block reveal (`onceValue.value = created.token`, `onceBlock.hidden = false` — the same once-block, the A4 value-only contract) → `announce("Regenerated \"<label>\" — copy the new token now; it won't be shown again.")`; 404 → the row re-renders revoked-in-place + `That token was already revoked.` (the revoke control's existing 404 handling, mirrored); 409 → the row relocates via `loadTokens()` + the same line; other failure/network → neutral retry copy + the confirm restores (the `restoreRegenerate` pattern — focus returns). The active Actions cell order: **Regenerate, then Revoke** (the rotation is the primary lifecycle action).
|
||||||
|
- `frontend/assets/styles.css` — `.token-search` (the input row: `width: 100%;` + the house input surface — reuse the `#tune-note` font/padding/border family, `:focus-visible` via the global rule, ≥ 44px target height), `.tokens-revoked-heading` (the sub-heading, the AA pair noted above), the no-match rows' `<td>` (the `tokens-empty-row` styling family — `color: var(--ink-soft)`), the Regenerate button (the `.token-revoke` family's neutral sibling — or a new `.token-regenerate` class styled identically to `.token-revoke` minus the hover-error: keep it simple — REUSE `.token-revoke`'s classes for the confirm machinery, one extra class for the label if needed; NO new hue — the phase-92 monochrome invariant). The `≤640px` block: the search inputs + the two tables squeeze as today (verify the `tokens-table-wrap` mobile rules cover the second table — they are class-based, so they do).
|
||||||
|
- Source pins: extend the existing tokens.js full-file source-pin unit file (the phase-79 task-06 pin — locate it under `tests/unit/`, likely `test_tokens_ui.py` or the `test_api_tokens_model.py` sibling; if no dedicated file exists, CREATE `tests/unit/test_tokens_ui.py`): the two tbodys + the section show/hide, the split in `loadTokens`, the filter logic (the case-insensitivity, the re-apply-after-load, the no-match row text), the required-name block (the exact announce copy + NO `|| "token"` anywhere in the file), the regenerate wiring (the POST path, the once-block reveal, the 404/409 branches, the confirm copy) + the styles.css class pins.
|
||||||
|
|
||||||
|
### The E2E (task 04)
|
||||||
|
`tests/e2e/test_tokens_page.py` (new; `app_server` + `mock_llm` + `db_ready`; admin via `auth_helpers.login`; a token-user context via a FRESH browser context + `auth_helpers.login_with_token` — the phase-79 pattern; the e2e-`-labeled` token cleanup autouse fixture from `test_api_tokens.py`):
|
||||||
|
1. **`test_revoked_tokens_move_to_their_own_table`** — create two tokens (the UI create row, labels `e2e-act` / `e2e-rev`); Revoke `e2e-rev` (the two-step): the active table shows ONLY `e2e-act` (the active thead has NO Status column — four `<th>`); the revoked section is visible BELOW the active table: its heading `Revoked tokens`, the `e2e-rev` row with a Revoked date cell (non-empty, ≠ created) and NO action buttons.
|
||||||
|
2. **`test_both_tables_are_searchable`** — with ≥ 2 active tokens (`e2e-a1` / `e2e-a2` — labels chosen so one is a substring of the other's neighbor: type `e2e-a1` → only that row visible; type `zzz` → the no-match row reads `No tokens match "zzz".`; clear → both rows back). Same on the revoked search (with the revoked row from test 1's flow or a fresh revoke).
|
||||||
|
3. **`test_a_token_cannot_be_generated_without_a_name`** — clear the label, click Generate: NO once-block, the live region reads `Give the token a name first.`, the active table's row count is unchanged (and `GET /api/tokens` via the admin cookie shows no new row — the request never happened).
|
||||||
|
4. **`test_regenerate_rotates_the_token`** — create `e2e-rot`; Regenerate → Yes: the once-block shows a NEW plaintext (≠ the original, `bor_` + 32 hex); the active table has the `e2e-rot` row (the NEW one — its created timestamp is newer); the revoked table now holds the ORIGINAL `e2e-rot` (same label, Revoked date set); the live region carries the D2 line; a fresh token-user context signs in with the NEW token (the gate closes → the app opens — `login_with_token`), and the ORIGINAL token is refused at the gate (the `#auth-gate-error` alert — it was revoked by the rotation).
|
||||||
|
5. Update `tests/e2e/test_api_tokens.py` in place (the phase-97 task-07/08 precedent): the revoked-row assertions relocate to `#tokens-revoked-tbody` (`test_admin_generates_token_in_ui`'s post-revoke state, `test_revocation_closes_the_door`'s `.stale-pill` → the revoked table's row + its Revoked date cell); the Status-column assertions (if any) drop; every non-layout assertion (the auth flows, the cached token, the 403 walls, the wrong-token error) stays byte-identical.
|
||||||
|
6. Regression sweep (each in isolation, `--no-cov`, DB up): the updated `test_api_tokens.py` + `test_admin_auth.py` (the gate contract) + `test_shared_header.py` (the header/whoami plumbing — expected unchanged).
|
||||||
|
7. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `101_tokens_page_overhaul` → `.agents/phases/complete/`; one atomic commit:
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(tokens): split active and revoked tables with search, require a name, and add one-click regenerate"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_regenerate_api.md` — the service `regenerate_token` + `POST /api/tokens/{id}/regenerate` (rotation, atomic) + `TokenListItem.revoked_at` + unit/integration
|
||||||
|
2. `02_two_tables_search_required_name.md` — the view skeleton (two tables + the revoked section + the search inputs) + the client split/filter + the required-name block
|
||||||
|
3. `03_regenerate_ui.md` — the Regenerate control (two-step confirm, the once-block reveal, the 404/409 branches) + the CSS + the source pins
|
||||||
|
4. `04_e2e_tokens_page.md` — `tests/e2e/test_tokens_page.py` + the `test_api_tokens.py` update + the regression sweep + the atomic commit
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the `regenerate_token` matrix (`tests/unit/test_tokens.py`); the source pins (task 03's file) for the view logic.
|
||||||
|
- Integration: the endpoint matrix + the list shape (`tests/integration/test_tokens_api.py`); the existing phase-79 auth-enforcement pins stay green (the new route rides the router-wide gate).
|
||||||
|
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` in isolation; the updated `test_api_tokens.py` + `test_admin_auth.py` + `test_shared_header.py` green in isolation.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the Tokens view (admin) shows the active tokens in the top table (Label | Created | Last used | Actions) and the revoked ones in a separate table below (Label | Created | Last used | Revoked) — the section hidden while no token is revoked; both tables filter live by label with a per-table no-match row that survives re-renders
|
||||||
|
- [ ] a blank/whitespace name generates NOTHING (the live-region line, no request, no row); the `token` fallback is gone
|
||||||
|
- [ ] Regenerate (one button + the house confirm) rotates: the old token is revoked (lands in the revoked table, refuses new sign-ins immediately) and the new token (same label) is active with its plaintext shown exactly once; 404/409 are pinned at the API level
|
||||||
|
- [ ] the A4 contract holds end to end (the list never carries plaintext/hashes; the once-block re-shows nothing — the phase-79 pins green)
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; ruff + pyright clean
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`); the regression suites green in isolation
|
||||||
|
- [ ] one atomic Conventional Commit, `--no-gpg-sign`
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 01 — `POST /api/tokens/{id}/regenerate`: the atomic rotation + `TokenListItem.revoked_at`
|
||||||
|
|
||||||
|
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The admin can rotate an active token server-side: one atomic call revokes the old row and creates its successor (same label, new plaintext returned exactly once) — plus the list API exposes `revoked_at` so the revoked table can show the revocation date.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/core/tokens.py`:
|
||||||
|
- `class TokenAlreadyRevoked(Exception)` — module-level, one-sentence docstring (a dead token cannot be rotated — the 409's reason).
|
||||||
|
- `regenerate_token(db, token_id: uuid.UUID) -> tuple[ApiToken, str] | None`:
|
||||||
|
- `row = db.get(ApiToken, token_id)` — `None` → return `None` (the endpoint maps it to the 404).
|
||||||
|
- `row.revoked_at is not None` → `raise TokenAlreadyRevoked`.
|
||||||
|
- `revoke(db, row.id)` (stamps the original revocation time — the existing primitive; it returns True here by construction) + `new_row, plaintext = create_token(db, row.label)` (the SAME label — D2).
|
||||||
|
- `db.flush()` and return `(new_row, plaintext)` — the CALLER commits (the `create_token` convention: the service flushes, the endpoint owns the commit). ONE commit covers stamp + create (atomicity: a create failure rolls the revoke back with it — note this in the docstring).
|
||||||
|
- Module docstring: the rotation's contract (the old plaintext was already one-shot; the new one is too — A4 never weakens).
|
||||||
|
2. `app/api/tokens.py` — `@router.post("/{token_id}/regenerate", response_model=TokenCreated, status_code=201)`:
|
||||||
|
- `result = token_service.regenerate_token(db, token_id)`; `None` → 404 `token not found` (the revoke endpoint's exact message); `except TokenAlreadyRevoked` → 409 `token already revoked`; success → `db.commit()` + `db.refresh(new_row)` (the server-default `created_at`) + `TokenCreated(id, label, token, created_at)`.
|
||||||
|
- `list_tokens` — pass `revoked_at=row.revoked_at` into each `TokenListItem` (D5).
|
||||||
|
- The new route's docstring: rotate-once semantics + the A4 note (this 201 is the new token's ONLY plaintext moment).
|
||||||
|
3. `app/schemas.py` — `TokenListItem` gains `revoked_at: datetime | None = None` (the docstring: null while active; the ISO-8601 timestamp the revoked table renders — the D5 note).
|
||||||
|
4. Unit (`tests/unit/test_tokens.py` extensions) — the `regenerate_token` matrix (the existing file's DB-session pattern): active row → a NEW row (different id) with the SAME label + a well-formed new plaintext (`bor_` + 32 hex) whose sha256 is the stored hash; the OLD row stamped with a `revoked_at`; the new plaintext round-trips through `find_active_by_token`; the old plaintext no longer authenticates (`find_active_by_token(old)` → `None`); missing id → `None`; an already-revoked id → `TokenAlreadyRevoked`; the rotation is atomic (a create failure — e.g. simulate by exhausting… if the existing harness cannot fail `create_token`, pin the flush-only/commit-caller contract by asserting the session is NOT committed by the service: the caller's rollback undoes both writes).
|
||||||
|
5. Integration (`tests/integration/test_tokens_api.py` extensions): 201 (body carries the new plaintext — `bor_`-prefixed, ≠ the original; the follow-up list shows the old row `revoked: true` + non-null `revoked_at` and the new row active, same label, newer `created_at`); 404 unknown id (one message); 409 an already-revoked id (revoked via the endpoint first); 403 anonymous + 403 a token-user session (the router-wide gate — the existing 403 pattern in the file); the list response shape now includes `revoked_at` (null for active, timestamp for revoked — the phase-79 list pins updated in place).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the matrix above (`tests/unit/test_tokens.py`).
|
||||||
|
- Integration: the endpoint matrix + the list shape (`tests/integration/test_tokens_api.py`).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `POST /api/tokens/{id}/regenerate` → 201 with the new token's plaintext (same label); the old row is revoked in the SAME transaction; 404 / 409 pinned; the admin gate pins hold (403 anonymous + token-user)
|
||||||
|
- [ ] `GET /api/tokens` items carry `revoked_at` (null active / timestamp revoked); the existing phase-79 list assertions (no plaintext, no hashes, newest-first) still pass
|
||||||
|
- [ ] the A4 contract: the list never gains a credential field (unit + integration pinned)
|
||||||
|
- [ ] full test suite green, coverage >90%
|
||||||
|
- [ ] no behavior change in completed work (the phase-79 auth-enforcement + token-gate suites green)
|
||||||
+58
@@ -0,0 +1,58 @@
|
|||||||
|
# Task 02 — The view: active/revoked tables, per-table search, the required name
|
||||||
|
|
||||||
|
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The Tokens view splits its list into an active table and a revoked table below it (each with its own live label search), and a blank name can no longer generate a token — the `token` fallback is gone.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` `#view-tokens` (the static skeleton — the D1/D4 shapes from `00_phase.md`; keep every existing id that survives):
|
||||||
|
- The create row: the input's `aria-label` → `Token name`, the `placeholder` → `e.g. alice — required`; the row's comment updated (a blank name is REFUSED client-side — D3 — the server's 422 is defense in depth).
|
||||||
|
- A search row between the create row/once-block area and the active table's wrap: `<input type="search" id="token-search-active" aria-label="Search active tokens" placeholder="Search active tokens…" class="token-search">` (ships hidden — revealed in the admin branch with the create row).
|
||||||
|
- The active table (`#tokens-table`): thead loses the Status `<th>` — four columns now: Label | Created | Last used | `<th scope="col"><span class="visually-hidden">Actions</span></th>` (unchanged); `#tokens-empty-row` → `colspan="4"` (copy unchanged); a NEW `<tr class="tokens-empty-row" id="tokens-no-match-row" hidden><td colspan="4"></td></tr>` (its text JS-filled — the no-match copy, D4).
|
||||||
|
- The NEW revoked section AFTER the active table's wrap (ships `hidden` — tokens.js shows it when ≥ 1 revoked token):
|
||||||
|
```html
|
||||||
|
<h2 id="tokens-revoked-heading" class="tokens-revoked-heading">Revoked tokens</h2>
|
||||||
|
<input type="search" id="token-search-revoked" aria-label="Search revoked tokens" placeholder="Search revoked tokens…" class="token-search">
|
||||||
|
<div class="table-wrap tokens-table-wrap" id="tokens-revoked-wrap" role="region" aria-label="Revoked tokens" tabindex="0">
|
||||||
|
<table class="tokens-table" id="tokens-revoked-table">
|
||||||
|
<caption class="visually-hidden">Revoked tokens — newest first</caption>
|
||||||
|
<thead><tr>
|
||||||
|
<th scope="col">Label</th><th scope="col">Created</th>
|
||||||
|
<th scope="col">Last used</th><th scope="col">Revoked</th>
|
||||||
|
</tr></thead>
|
||||||
|
<tbody id="tokens-revoked-tbody">
|
||||||
|
<tr class="tokens-empty-row" id="tokens-revoked-no-match-row" hidden><td colspan="4"></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
```
|
||||||
|
(the view's HTML comments carry the D1/D2/D4 provenance — the house comment style; the section's ids are all new, nothing collides).
|
||||||
|
2. `frontend/assets/tokens.js` (the split + search + required name):
|
||||||
|
- Module lookups: the revoked section's heading / search input / wrap / tbody / no-match row + the active no-match row (scoped to `root`, the phase-76 contract).
|
||||||
|
- Module state: `activeQuery = ""` / `revokedQuery = ""` (the D4 persistent queries — initialized once, NEVER reset by a load).
|
||||||
|
- `makeRow(tok, table)` — `table` is `"active"` | `"revoked"`: shared Label/Created/Last-used cells (the `title` hovers carry over); the active variant's Actions cell (Regenerate — task 03 — + the existing two-step Revoke, UNCHANGED in this task); the revoked variant's last cell = the `revoked_at` date (locale date+time via the existing `fmtDate`, full ISO in `title`, class `tokens-date-cell`) and NO actions.
|
||||||
|
- `loadTokens()` — the re-entrant core is preserved (once-block hidden + field wiped, data rows dropped, the fetch, the error lines, the return value — all UNCHANGED); the render step now SPLITS the fetched list by `tok.revoked` (server order kept per table: active rows → `#tokens-tbody`, revoked rows → `#tokens-revoked-tbody`); the active empty-state row shows iff there are zero ACTIVE rows (a 0-active fetch with revoked rows shows the empty-state row AND the populated revoked section — both honest); the revoked SECTION (heading + search input + wrap — a small `setRevokedSectionVisible(n)` helper) shows iff n ≥ 1; FINALLY both filters re-apply (`applyFilter` below — D4: a re-render never loses the queries).
|
||||||
|
- `applyFilter(tbody, noMatchRow, query, )` — pure DOM: trim + lowercase the query; over the tbody's data rows (everything EXCEPT the no-match row), `row.hidden = query && !label.toLowerCase().includes(query.toLowerCase())` (read the label from the row's label cell textContent — the rows are the data source of truth after a render); `noMatchRow` hidden unless `query && zeroDataRowsVisible` — its `<td>` textContent = `No tokens match "${query}".` (the ORIGINAL, untrimmed user query inside the quotes — textContent only).
|
||||||
|
- The `input` listeners (armed in the admin branch, after the whoami gate — the anonymous branch arms nothing): set the module query + `applyFilter` (NO fetch — D4).
|
||||||
|
- `generateToken()` (D3): `const label = (labelInput ? labelInput.value : "").trim();` — `if (!label) { announce("Give the token a name first."); if (labelInput) labelInput.focus(); return; }` — the `|| "token"` DELETED; the placeholder/comment (step 1) and the function's comment updated; the rest of the flow (the disabled/"Generating…" lifecycle, the 201 reveal, the list re-load FIRST, the label clear, the error lines) byte-identical in behavior.
|
||||||
|
- Module docstring: the phase-101 section (the split, the search contract, the required name, the revoked column).
|
||||||
|
- The `bor:view-refresh` listener is UNCHANGED (it calls `loadTokens()`, which now splits + re-applies the filters — the queries survive a re-show for free).
|
||||||
|
3. `frontend/assets/styles.css` (the token region):
|
||||||
|
- `.token-search { width: 100%; min-height: 44px; padding: 0.55rem 0.8rem; border: 1px solid var(--line); border-radius: var(--radius-sm); background: var(--surface); color: var(--ink); font: inherit; font-size: 0.9rem; }` + `:focus-visible` via the global 3px outline rule (no custom focus rule — the house pattern) + a comment (the search inputs' surface — the `#tune-note`/archive-upload input family, AA pairs, no new hue).
|
||||||
|
- `.tokens-revoked-heading { margin: 1.5rem 0 0.5rem; font-family: var(--mono); font-size: 1rem; color: var(--brand-ink); }` (the sub-heading — the phase-97 `.kb-level h2` voice, AA on `--bg`).
|
||||||
|
- The no-match rows reuse the existing `.tokens-empty-row` styling (the class is shared — verify it covers both tables; if the empty-row rule is `#tokens-empty-row`-specific, generalize it to `.tokens-empty-row` — the styling is already class-based, so likely a no-op).
|
||||||
|
- The `≤640px` block: verify the existing `.token-create` / `.tokens-table-wrap` squeeze rules cover the new inputs + the second table (class-based — expected no change; add a search-input width rule only if the mobile layout needs it).
|
||||||
|
4. Source pins: locate the phase-79 tokens.js full-file source-pin unit file under `tests/unit/` (grep `tokens.js` in `tests/unit/` — if a dedicated file exists, EXTEND it; if the pins live inside `test_api_tokens_model.py` or similar, extend there; if no dedicated pin file exists, CREATE `tests/unit/test_tokens_ui.py`): the two-tbody split in `loadTokens`, the section show/hide helper, `applyFilter` (case-insensitivity, the re-apply-after-load call, the no-match text with the original query, the data-row exclusion), the required-name block (the exact announce copy `Give the token a name first.`, the early return BEFORE any fetch, and a negative pin: the string `"token"` fallback pattern `|| "token"` is GONE from the file), the search inputs' armed-in-admin-branch wiring + the styles.css class pins.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source pins above are this task's test layer for the JS (`app/` untouched this task — task 01's API pins already cover the server side).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (frontend pins cover the JS; the browser proof is task 04's E2E).
|
||||||
|
- Note: do NOT add an E2E file here — the split/search/name proofs land in task 04.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the view skeleton carries the four-column active table, the revoked section (heading + search + the four-column table) shipping hidden, and both search inputs (ids per the D1/D4 contract)
|
||||||
|
- [ ] `loadTokens` splits by `revoked`, hides the section when empty, keeps the once-block / error / return-value contracts byte-identical, and re-applies BOTH persistent queries after every render (source-pinned)
|
||||||
|
- [ ] a blank/whitespace name announces `Give the token a name first.`, re-focuses, and sends NO request (source-pinned; the `|| "token"` pattern is gone from the file)
|
||||||
|
- [ ] full test suite green, coverage >90%
|
||||||
|
- [ ] no behavior change in completed work (the phase-79 gate / whoami / once-block pins green; the Regenerate button is NOT built yet — task 03 adds it)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 03 — The Regenerate control: one button, the house confirm, the once-block reveal
|
||||||
|
|
||||||
|
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every active token row carries a **Regenerate** button (before its Revoke): one click opens the house two-step confirm, Yes rotates the token server-side, and the new plaintext appears in the shown-once block — the old token relocates to the revoked table.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/tokens.js`:
|
||||||
|
- `makeRegenerateControl(tok, row)` — a structural mirror of `makeRevokeControl` (the D2 contract from `00_phase.md`): a `.token-regenerate` button labeled `Regenerate` (`aria-label: "Regenerate token: <label>"`); first click swaps the cell's contents to the confirm pair — `span.history-confirm-text` `Regenerate? The current token is revoked.` + Yes (`.history-confirm-yes`) + No (`.history-confirm-no`), focus to Yes; No / failure → `restoreRegenerate()` (the button back, focus restored — the revoke control's restore pattern, copied).
|
||||||
|
- `confirmRegenerate(tok, row, yesBtn, restoreRegenerate)`: `yesBtn.disabled = true`; `POST /api/tokens/<id>/regenerate` (JSON — no body):
|
||||||
|
- **201** → parse `{ id, label, token, created_at }` → `await loadTokens()` FIRST (the re-entrant load: the old row moves to the revoked table, the new row lands in the active one — the D2 relocation) → the once-block reveal (`if (onceValue) onceValue.value = data.token;` + `if (onceBlock) onceBlock.hidden = false;` — the SAME once-block, the A4 value-only contract: the plaintext lives in the field's value, never a data attribute) → `announce(`Regenerated "${tok.label}" — copy the new token now; it won't be shown again.`)`.
|
||||||
|
- **404** → the row is gone (revoked elsewhere / regenerated elsewhere): `row.replaceWith(makeRow({ ...tok, revoked: true }, "revoked"))` — wait: the row is an ACTIVE-table row; a 404 means the id no longer exists at all. Mirror the revoke control's 404 handling: re-render the row in the REVOKED table's state is wrong if the row is gone — instead `row.remove()` + `announce("That token no longer exists.")`… **Decision (pinned here):** a 404 on regenerate means the row vanished (revoked AND deleted by another admin, or a stale render) — `row.remove()` + `await loadTokens()` (the re-fetch reconciles both tables) + `announce("That token was already revoked.")` (the revoke control's existing 404 line — one house message for the one common case).
|
||||||
|
- **409** → the row was revoked between render and click: `await loadTokens()` (reconciles) + `announce("That token was already revoked.")`.
|
||||||
|
- **other non-2xx / network** → `announce("Couldn't regenerate \"<label>\" — try again.")` (network: the `is the app reachable?` variant — the house two-line convention) + `restoreRegenerate()` (retryable, the button back).
|
||||||
|
- The active `makeRow`'s Actions cell: `actionsTd.append(makeRegenerateControl(tok, tr), makeRevokeControl(tok, tr))` — Regenerate FIRST (the primary lifecycle action, D2); the two controls each own their own `<span class="tokens-actions">` cell-content swap (verify the two swap-scopes don't interfere — the revoke control swaps `cell`'s children; give each control its OWN wrapper span (the regenerate control gets its own, the revoke control keeps its existing one) so a confirm in one never clobbers the other).
|
||||||
|
- The `#tokens-status` live-region contract is unchanged (every outcome lands a line — D2's copy is pinned).
|
||||||
|
- Module docstring: the regenerate section (the rotation semantics + the 404/409 copy).
|
||||||
|
2. `frontend/assets/styles.css` (the token region): `.token-regenerate` — the `.token-revoke` button's visual language (same size/border/radius/focus; color `var(--ink-soft)` on transparent, hover `var(--brand-soft)` / `var(--brand-ink)` — the neutral action's hover, NOT the revoke's error hover; a `.token-regenerate:hover:disabled` state consistent with the revoke button's disabled rule) + the comment (the rotation button — no new hue, phase-92 invariant; the confirm pair reuses the `history-confirm-*` classes unchanged).
|
||||||
|
3. Source pins (the file established in task 02): `makeRegenerateControl` (the button label + aria-label, the confirm text EXACTLY `Regenerate? The current token is revoked.`, the focus-to-Yes, the restore path), `confirmRegenerate` (the POST path `/api/tokens/` + `regenerate`, the 201 sequence — load FIRST then once-block reveal then announce (the D2 line, exact), the 404/409 copy, the neutral retry copy, the disabled-while-in-flight), the Actions cell order (Regenerate before Revoke), the per-control wrapper spans, + the `.token-regenerate` CSS pin.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source pins above are this task's test layer (`app/` untouched — task 01's API pins cover the endpoint).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (frontend pins cover the JS; the browser proof is task 04's E2E).
|
||||||
|
- Note: do NOT add an E2E file here — the rotation proof lands in task 04.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] an active row shows Regenerate (before Revoke); the confirm copy/focus/restore are pinned; Yes → 201 → the old row is in the revoked table, the new row active (same label), the once-block shows the new plaintext, the D2 line is announced (source-pinned; task 04 proves it in a browser)
|
||||||
|
- [ ] the 404 / 409 / failure branches are pinned (the house copy, the retryable restore, the load-based reconciliation)
|
||||||
|
- [ ] the revoke control's behavior is byte-identical (its pins green — the shared cell hosts two independent confirm scopes)
|
||||||
|
- [ ] full test suite green, coverage >90%
|
||||||
|
- [ ] no behavior change in completed work (the phase-79 once-block / copy / gate pins green)
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 04 — The dedicated E2E: the split tables, the search, the required name, the rotation + regressions + commit
|
||||||
|
|
||||||
|
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Pin the owner-visible contract in a browser: revoked tokens live in their own table below the active ones, both tables search, a nameless token cannot be created, and Regenerate rotates the credential end to end (the new token signs in, the old one is refused) — then the phase-79 suite moves its revoked-row pins to the new table.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_tokens_page.py` (new — the phase's dedicated A16 suite, run in isolation). Fixtures: `app_server` + `mock_llm` + `db_ready` (the `conftest` pattern); admin via `tests/e2e/auth_helpers.login(page, app_url, next="/tokens.html")`; a token-user context via a FRESH browser context + `auth_helpers.login_with_token(page, app_url, token)` (the phase-79 helper — drives the real gate). An autouse fixture deletes `e2e-`-labeled tokens (the `test_api_tokens.py` cleanup pattern — `DELETE FROM api_tokens WHERE label LIKE 'e2e-%'`).
|
||||||
|
- **`test_revoked_tokens_move_to_their_own_table`** — create two tokens through the UI create row (`e2e-act`, `e2e-rev` — label → Generate → the once-block appears each time); Revoke `e2e-rev` (the existing two-step: click Revoke → click Yes): `#tokens-tbody` shows ONLY `e2e-act`; the active table's `<thead>` has exactly four `<th>` (NO Status column); `#tokens-revoked-heading` reads `Revoked tokens` and is visible; `#tokens-revoked-tbody` shows the `e2e-rev` row with a non-empty Revoked date cell (locale format, ≠ the created date) and NO buttons in its row; `#tokens-status` reads `Revoked "e2e-rev".`.
|
||||||
|
- **`test_both_tables_are_searchable`** — with `e2e-a1` + `e2e-a2` active and one revoked token present: type `e2e-a1` in `#token-search-active` → only the `e2e-a1` row visible (`e2e-a2` hidden); type `zzz` → the no-match row visible reading `No tokens match "zzz".`; clear the input → both rows back, no-match hidden. The same three-beat on `#token-search-revoked` (filter the revoked row in/out). The queries SURVIVE a re-show: type a query, leave via the Tuning nav link, come back (the phase-77 refresh re-renders) → the filter is still applied (the visible set unchanged).
|
||||||
|
- **`test_a_token_cannot_be_generated_without_a_name`** — with the label empty, click Generate: `#token-once` stays hidden, `#tokens-status` reads `Give the token a name first.`, the active table's data-row count is unchanged, and `GET /api/tokens` (admin cookie via `page.request.get`) shows no new row (the request never happened). Whitespace-only label → the same refusal.
|
||||||
|
- **`test_regenerate_rotates_the_token`** — create `e2e-rot` (capture its plaintext from `#token-once-value`); on its row click Regenerate → the confirm pair appears (`Regenerate? The current token is revoked.`, focus on Yes) → click Yes: `#token-once` re-appears with a NEW plaintext (≠ the original, matches `bor_[0-9a-f]{32}`); `#tokens-status` reads `Regenerated "e2e-rot" — copy the new token now; it won't be shown again.`; the active table has exactly ONE `e2e-rot` row (the new one — its created cell is newer than the old row's); the revoked table now holds an `e2e-rot` row (the original — its Revoked date is set); a fresh token-user context signs in with the NEW token (the gate closes, the chat view usable — `login_with_token` + an assertion on the signed-in state) and the ORIGINAL token is refused in another fresh context (the `#auth-gate-error` role=alert visible after submitting it — the rotation killed it, the phase-79 revocation semantics).
|
||||||
|
2. `tests/e2e/test_api_tokens.py` — UPDATE IN PLACE (the phase-97 task-07/08 precedent — the layout changed, the auth contracts did not):
|
||||||
|
- The revoked-row assertions relocate to `#tokens-revoked-tbody`: `test_admin_generates_token_in_ui` (its post-revoke expectations — the row's NEW home; the `.stale-pill` assertion drops with the Status column — assert the revoked row + its Revoked date cell instead) and `test_revocation_closes_the_door` (the two-step revoke now lands the row in the revoked table — update the locator + the pill assertion the same way; the `Revoked "e2e-revoke".` live-region line is UNCHANGED).
|
||||||
|
- Any Status-column / 5-`<th>` / `colspan="5"` references update to the four-column contract; `#tokens-empty-row`'s colspan is 4.
|
||||||
|
- EVERYTHING else — the anonymous lockout, the shared-chats-stay-open, the token-user app usage, the cached-token reload, the 403 walls, the sign-out clears, the wrong-token generic error — stays byte-identical (this suite is the phase-79 auth contract; only its layout locators move).
|
||||||
|
3. Regression sweep (each in isolation, `--no-cov`, DB up): the updated `test_api_tokens.py` + `test_admin_auth.py` (the gate contract — expected untouched) + `test_shared_header.py` (the whoami/header plumbing — expected untouched).
|
||||||
|
4. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `101_tokens_page_overhaul` → `.agents/phases/complete/`; one atomic commit:
|
||||||
|
```bash
|
||||||
|
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(tokens): split active and revoked tables with search, require a name, and add one-click regenerate"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` green in isolation.
|
||||||
|
- Coverage: **>90%** on `app/` (the full-suite gate).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the four E2E tests pass in isolation (the split + the column contract, both searches incl. the re-show persistence, the nameless refusal, the full rotation with the new-token-signs-in / old-token-refused proof)
|
||||||
|
- [ ] the updated `test_api_tokens.py` passes in isolation (the phase-79 auth contract intact, the layout pins relocated)
|
||||||
|
- [ ] the regression suites pass in isolation
|
||||||
|
- [ ] full suite green, coverage >90%, ruff + pyright clean
|
||||||
|
- [ ] phase dir moved to `complete/`, one atomic `--no-gpg-sign` Conventional Commit
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# Phase 102 — Extensionless filenames: sync `Dockerfile` / `Containerfile` when their name is in `BOR_IMPORT_EXTENSIONS`
|
||||||
|
|
||||||
|
**Source:** owner report (chat): "files without extensions never get synced, so 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
|
||||||
|
**Story:** n/a (defect — the import pipeline of `38_local_directory_sources` / `56_import_extensions_env`; the badge display rides on the phase-10 viewer).
|
||||||
|
**Context:** the importer's format filter is `path.suffix.lower() in extensions` (`app/rag/importer.py`, `iter_importable_files`) and `Settings.import_extension_set` (the dotted set, `app/config.py`) feeds it. `Path("Dockerfile").suffix` is `""`, so **extensionless files never match — no value of `BOR_IMPORT_EXTENSIONS` can import them**. The validator already accepts `dockerfile` as a well-formed token (lowercase alphanumerics, 1-16 chars, phase 56 — "any extension"), so today the user can *type* the format and it still silently imports nothing. The viewer's format badge (`doc_format`, `app/api/docs.py`, rendered by `frontend/assets/document.js`) falls back to `text` for any suffix-less path; the import `formats=` counter (`ImportSummary.formats`) keys extensionless files under `unknown`.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An extensionless file is imported **iff its lowercased full filename is a token in `BOR_IMPORT_EXTENSIONS`** — exact name, case-insensitive. With `dockerfile,containerfile` in the env (the owner's current config), `Dockerfile` and `Containerfile` sync like any other A9 file: chunks, `lite` summary, Sources tree, viewer with a truthful format badge.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `56_import_extensions_env` (the env-driven extension scope this phase extends) — complete
|
||||||
|
- `38_local_directory_sources` / `97_kb_tree_catalog` (import pipeline + drill-down tree the E2E asserts on) — complete
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### D1 — the matching rule (task 01)
|
||||||
|
A file under a source root is in scope **iff**:
|
||||||
|
1. its lowercased dotted suffix is a member of `import_extension_set` (the existing rule — `kubernetes.md` → `.md`), **or**
|
||||||
|
2. it has **no suffix** and its lowercased **full filename** equals a bare token of `import_extension_set` (`Dockerfile` → `dockerfile`).
|
||||||
|
|
||||||
|
Consequences (pinned by tests): `DOCKERFILE` matches (case-insensitive name); `mydockerfile` and `dockerfile`-suffixed lookalikes do **not** (exact name only — the list is user-controlled and exact is the only predictable rule); `Dockerfile.dev` is governed by its `.dev` suffix as today; dot-prefixed files (`.dockerfile`) stay skipped by the pre-existing hidden-component rule; the rule applies at any depth (`services/api/Dockerfile` matches).
|
||||||
|
|
||||||
|
### D2 — one choke point (task 01)
|
||||||
|
`app/rag/importer.py` gains `match_extension(path: Path, extensions: frozenset[str]) -> str | None` — returns the matched **bare lowercased token** (``md``, ``dockerfile``) or `None` when out of scope; `extensions` keeps the existing dotted form (``{'.md'}``), bare tokens are derived by stripping the leading dot. `iter_importable_files` filters with `match_extension(path, extensions) is not None` (the walk and the phase-64 progress pre-walk both call this function — one change covers both, `total` stays honest). The `summary.formats` counter in `import_sources` uses `match_extension(path, …) or "unknown"`, so an imported `Dockerfile` counts under `dockerfile:1` in the PLAN §9 line, not `unknown:1`.
|
||||||
|
|
||||||
|
### D3 — the badge (task 02)
|
||||||
|
`doc_format(path, extensions: frozenset[str] = frozenset())` in `app/api/docs.py`: a **non-empty suffix still returns the suffix token unconditionally** (display must never depend on the import list — an out-of-scope `readme.rst` still badges `rst`); a suffix-less path returns the name token when it matches per D1, else `text` (today's fallback, byte-identical for every existing row). The content endpoint (`GET /api/documents/content`) passes `get_settings().import_extension_set` (`app.config`, lru-cached). `frontend/assets/document.js` needs no change: only `md`/`markdown` trigger markdown rendering, so a `dockerfile` badge renders the raw `<pre>` view — correct for a build file.
|
||||||
|
|
||||||
|
### D4 — no changes (pinned)
|
||||||
|
Title stays `full_path.stem` (`Dockerfile` — already correct); the phase-30 `lite` summary applies (non-markdown); prune semantics unchanged — a file that stops matching (file renamed, or token removed from the env) leaves `seen` and is deleted by the next `prune=True` run (the A9 junk precedent).
|
||||||
|
|
||||||
|
### D5 — docs
|
||||||
|
`app/config.py` `import_extensions` docstring + the `.env.example` "Import scope" comment gain the extensionless rule, with `dockerfile,containerfile` as the example (no default-list change — the built-in A9 default stays exactly as phase 56 locked it).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_name_token_matching.md` — `match_extension` in the importer + the walk + the `formats` counter (unit-pinned).
|
||||||
|
2. `02_format_badge_and_docs.md` — the viewer badge via `doc_format` + config/`.env.example` docs.
|
||||||
|
3. `03_integration_and_e2e.md` — the dedicated integration suite + the Playwright story suite + regressions + atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_importer.py` — the matching matrix (D1), walk inclusion/exclusion, the `formats` counter; `tests/unit/test_document_viewer.py` — `doc_format` (with/without the token set, suffix precedence, `text` fallback).
|
||||||
|
- Integration: `tests/integration/test_import_extensionless.py` (new) — in-process `import_sources` against a story-dedicated fixture dir with the deterministic mock LLM (the phase-02/56 seeding pattern): positive import (counts, per-format `dockerfile:1,containerfile:1,md:1`, lite summary generated for the extensionless files), negative import (no token → not walked), prune-on-token-removal; `tests/integration/test_document_content.py` — the `format` field carries `dockerfile` for an extensionless row and `text` for an unknown-name row.
|
||||||
|
- E2E (mandatory, house rule 4): `tests/e2e/test_extensionless_import.py`, run in isolation (DB up) — admin Signs in, the drill-down tree lists `Dockerfile`/`Containerfile`, the viewer modal shows badge `dockerfile` + title + summary, `Makefile` (no token) is absent, rows cleaned up in `finally`.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the phase is "Complete" only when the full suite passes and the gate holds.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `BOR_IMPORT_EXTENSIONS` containing `dockerfile,containerfile` (plus the usual formats) makes a `Dockerfile` and a `Containerfile` under a source sync end-to-end: imported, chunked, `lite`-summarized, listed in the Sources drill-down tree, openable in the viewer.
|
||||||
|
- [ ] Case-insensitivity: `DOCKERFILE` imports under the `dockerfile` token; `mydockerfile` does **not**; `Dockerfile.dev` follows its `.dev` suffix; `.dockerfile` stays hidden-skipped.
|
||||||
|
- [ ] The import `formats=` counter reports `dockerfile:1` (not `unknown:1`) for an imported `Dockerfile`.
|
||||||
|
- [ ] The viewer format badge shows `dockerfile`/`containerfile`; a suffix-less file whose name is not in the env still badges `text`; every suffixed row's badge is unchanged.
|
||||||
|
- [ ] Removing a token from the env prunes the previously-imported extensionless documents on the next `prune=True` run.
|
||||||
|
- [ ] The existing suites stay green in isolation: `test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
|
||||||
|
- [ ] `uv run pytest` green; `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` green in isolation (DB up).
|
||||||
|
- [ ] No behavior change to completed phases for any suffixed path (the existing walk rule is untouched — only the suffix-less branch is new).
|
||||||
|
- [ ] One atomic Conventional Commit, `--no-gpg-sign` (e.g. `fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS`), `.agents/` phase files moved to `complete/` with the phase.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
|
||||||
|
```
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Task 01 — The name-token match: `match_extension`, the walk, the `formats` counter
|
||||||
|
|
||||||
|
**Phase:** `102_extensionless_filenames` · **Source:** owner report — "files without extensions never get synced … 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
|
||||||
|
**Story:** n/a (defect).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One pure helper decides import scope; the walk and the per-format counter use it; extensionless files whose lowercased full name is a `BOR_IMPORT_EXTENSIONS` token are finally imported.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — new module-level pure function, placed near `iter_importable_files`:
|
||||||
|
```python
|
||||||
|
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
|
||||||
|
"""The bare lowercased token *path* imports under, or ``None``.
|
||||||
|
|
||||||
|
1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
|
||||||
|
``kubernetes.md`` → ``md``).
|
||||||
|
2. No suffix: the lowercased FULL filename equals a bare token of
|
||||||
|
*extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
|
||||||
|
extensionless rule. Exact name only: ``mydockerfile`` never
|
||||||
|
matches the ``dockerfile`` token.
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
*extensions* keeps the existing dotted form (``{'.md', '.dockerfile'}``); derive the bare token set by stripping the leading dot. Rule 1 first; rule 2 only when ``path.suffix`` is empty.
|
||||||
|
2. `app/rag/importer.py` — `iter_importable_files`: replace the `path.suffix.lower() not in extensions` skip with `match_extension(path, extensions) is None`. The hidden-component check and the phase-89 ignore-prefix check stay before it, byte-identical.
|
||||||
|
3. `app/rag/importer.py` — the `summary.formats` line in `import_sources` (`ext = path.suffix.lower().lstrip(".") or "unknown"`): use `match_extension(path, llm.settings.import_extension_set) or "unknown"` so an imported `Dockerfile` counts under `dockerfile`, never `unknown`.
|
||||||
|
4. `app/rag/importer.py` — module docstring: one sentence for the extensionless rule in the scope paragraph (tokens may also name extensionless files by exact lowercased name — phase 102).
|
||||||
|
|
||||||
|
ASSUMPTIONS (pinned in `00_phase.md` D1/D2): exact-name, case-insensitive matching; no partial name matching; dotted set kept as the parameter form so no `app/config.py` change is needed in this task.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (`tests/unit/test_importer.py`, new tests next to the existing `iter_importable_files` ones):
|
||||||
|
- `match_extension` matrix: `kubernetes.md` + `{'.md'}` → `md`; `Kubernetes.MD` → `md` (case-insensitive suffix, as today); `Dockerfile` + `{'.md', '.dockerfile'}` → `dockerfile`; `DOCKERFILE` → `dockerfile`; `Dockerfile` without the token → `None`; `mydockerfile` with the token → `None`; `Dockerfile.dev` with only `dockerfile` → `None` (with `.dev` it → `dev`); `readme.rst` + `{'.md'}` → `None`.
|
||||||
|
- walk: a fixture tree with `Dockerfile`, `Containerfile`, `mydockerfile`, `Dockerfile.dev`, `.dockerfile`, `notes.md` — with tokens `md,dockerfile,containerfile` the walk yields exactly `Dockerfile`, `Containerfile`, `notes.md` (sorted); the hidden `.dockerfile` stays skipped.
|
||||||
|
- counter: in-process `import_sources` (the file's existing `db` + fake-LLM pattern) over a tree with one `Dockerfile` + one `.md` → `summary.formats == {"dockerfile": 1, "md": 1}` and the PLAN §9 log line carries `dockerfile:1` (not `unknown`).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `match_extension` exists, pure, dotted-set parameter, rule order pinned by the matrix above
|
||||||
|
- [ ] `iter_importable_files` + the progress pre-walk import extensionless name-token files (both go through the helper)
|
||||||
|
- [ ] the `formats` counter reports the matched token, never `unknown`, for in-scope files
|
||||||
|
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] no behavior change for any suffixed path (existing `test_importer.py` cases byte-identical)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — The truthful badge: `doc_format` learns the name-token rule + config/env docs
|
||||||
|
|
||||||
|
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect, display half).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The viewer's format badge shows `dockerfile`/`containerfile` for extensionless documents that matched by name (instead of the generic `text`), and the config/`.env.example` docs tell the user the extensionless rule exists.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/docs.py` — `doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str`:
|
||||||
|
- non-empty suffix → the suffix token, **unconditionally** (display never depends on the import list — an out-of-scope `readme.rst` still badges `rst`);
|
||||||
|
- no suffix → reuse the task-01 helper semantics from `app/rag/importer.py` (import `match_extension` — `app.api.docs` already imports from `app.rag`): the lowercased full name returns the token when it is one of `extensions` (dotted form, as passed), else `text`.
|
||||||
|
- Default `extensions=frozenset()` ⇒ today's exact behavior for every caller that passes nothing (byte-identical).
|
||||||
|
- Docstring: update the fallback line — ``text`` when the path has no suffix **and its name is not a configured token (phase 102)**.
|
||||||
|
2. `app/api/docs.py` — the content endpoint (`GET /api/documents/content`, `format=doc_format(doc.path)` at ~L143): pass `get_settings().import_extension_set` (`from app.config import get_settings` — the lru-cached accessor; no per-request Settings construction).
|
||||||
|
3. `app/config.py` — `import_extensions` docstring: one sentence — a token also matches **extensionless files whose lowercased full filename equals it exactly** (`dockerfile` → `Dockerfile`), case-insensitive, no partial names (phase 102).
|
||||||
|
4. `.env.example` — "Import scope" comment: add the same rule with the example `dockerfile,containerfile` (e.g. `add your own (e.g. md,sh,toml,dockerfile,containerfile)`); the default `BOR_IMPORT_EXTENSIONS` line is unchanged.
|
||||||
|
5. No frontend change: `frontend/assets/document.js` renders `doc.format` into the `format-badge` span and only `md`/`markdown` get markdown rendering — a `dockerfile` badge already lands in the raw `<pre>` view, which is the correct treatment for a build file. Verify, don't edit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration (`tests/integration/test_document_content.py`, extend the `test_content_format_from_suffix` neighborhood — seed via the file's existing `_seed_doc` + TRUNCATE-in-`finally` pattern):
|
||||||
|
- seeded `path="services/api/Dockerfile"` (source `Homelab`) → `body["format"] == "dockerfile"` under the default settings (the endpoint reads the real `get_settings()` — if the test needs the token, set `BOR_IMPORT_EXTENSIONS` for the client/session via the house env-override pattern used by other API tests; the default list does **not** contain `dockerfile`, so the assertion must run with a settings carrying it);
|
||||||
|
- seeded `path="README"` → `format == "text"` (unchanged — `readme` is not a token; the existing line at ~L554 stays green);
|
||||||
|
- seeded `path="notes/README.dev"` → `format == "dev"` (suffix precedence — display ignores the import list);
|
||||||
|
- `doc_format` directly: default-args call on an extensionless path → `text` (the no-arg contract pinned).
|
||||||
|
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] the content endpoint's `format` field carries the name token for an extensionless document whose name is in `BOR_IMPORT_EXTENSIONS`, `text` otherwise, and every suffixed value is unchanged
|
||||||
|
- [ ] `doc_format` with no second argument behaves byte-identically to today
|
||||||
|
- [ ] `app/config.py` docstring + `.env.example` document the extensionless rule with the `dockerfile,containerfile` example
|
||||||
|
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] `frontend/assets/document.js` verified untouched (raw `<pre>` rendering is correct for the new badge values)
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Task 03 — The proof: dedicated integration suite, the Playwright story suite, regressions, commit
|
||||||
|
|
||||||
|
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect — phase-level E2E per house rule 4).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
End-to-end proof that `Dockerfile`/`Containerfile` sync when their names are in `BOR_IMPORT_EXTENSIONS` — import → chunks → `lite` summary → drill-down tree → viewer badge — plus the negative and prune proofs, regressions, and the atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. Fixture `tests/fixtures/extensionless_kb/` (new, story-dedicated, distinctive source name = the dir name, the phase-56 pattern):
|
||||||
|
- `Dockerfile` — a few lines of plausible compose/build content carrying a unique sentinel (e.g. `DOCKERFILE-PROBE-SENTINEL-7a3e`);
|
||||||
|
- `Containerfile` — same shape, its own sentinel;
|
||||||
|
- `Makefile` — a few lines, **no token in the env under test** (the negative control);
|
||||||
|
- `notes.md` — one short markdown doc (the control row + the `md` count).
|
||||||
|
2. Integration `tests/integration/test_import_extensionless.py` (new; the `tests/integration/test_import_extensions_env.py` pattern — in-process `import_sources` against a fixture/tmp tree, `Settings(_env_file=None, import_extensions=…)` + the house mock LLM, `db` fixture, TRUNCATE/delete rows in `finally`):
|
||||||
|
- positive: extensions `md,dockerfile,containerfile` over the fixture → `added == 3`, `errors == 0`, `formats == {"dockerfile": 1, "containerfile": 1, "md": 1}` (NO `unknown` key), the `Dockerfile` row has chunks and a `lite` summary (non-markdown → phase-30 path);
|
||||||
|
- case: a `DOCKERFILE` file + token `dockerfile` imports (its row's path is `DOCKERFILE`);
|
||||||
|
- negative: extensions `md` only → `notes.md` imported, `Dockerfile`/`Containerfile`/`Makefile` absent from `documents`;
|
||||||
|
- prune: import with the tokens, then re-run `prune=True` with extensions `md` → the two extensionless rows are deleted, the `md` row survives (mirrors `test_prune_removes_files_now_excluded_by_format_filter`).
|
||||||
|
3. E2E `tests/e2e/test_extensionless_import.py` (new; **copy the skeleton of `tests/e2e/test_import_extensions_env.py`** — the module-docstring conventions, the in-thread seeding pattern, `_drill`/`_go_top`, the distinctive-source row cleanup in `finally`):
|
||||||
|
- boot/app with the house `mock_llm` + `db_ready` fixtures; seed via `import_sources([FIXTURES], LLMClient(settings))` with `import_extensions="md,dockerfile,containerfile"` (`Settings(_env_file=None, llm_base_url=…, import_extensions=…)`);
|
||||||
|
- test 1 (admin): summary assertions as in the integration positive case, then `login(page, app_url)`, `_drill(page, SOURCE)` — rows for `Dockerfile`, `Containerfile`, `notes.md` each count 1; `Makefile` count 0 (negative control); click the `Dockerfile` path link → same-page modal: `#doc-modal-meta .doc-source-badge` = source, `.format-badge` = `dockerfile` (the D3 badge), title `Dockerfile`, `pre.doc-raw` contains the sentinel, the phase-30 summary line renders (house assertion style — `test_summary_in_viewer.py`); still on `/sources.html`;
|
||||||
|
- test 2 (anonymous): the sign-in gate renders and `GET /api/docs` 403s (the phase-16 contract, same shape as the phase-56 suite's second test).
|
||||||
|
4. Regressions (isolated runs, per AGENTS.md rule 9): `tests/e2e/test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
|
||||||
|
5. Commit: full suite + `--cov=app` >90% + `uv run ruff check . && uv run pyright` clean, then move this phase dir `102_extensionless_filenames/` to `.agents/phases/complete/` and commit **everything together** (AGENTS.md rule 8 — `.agents/` is tracked), one atomic commit:
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The suites above ARE this task's tests; coverage gate **>90%** on `app/` (the fixture files and the E2E file need no coverage, `app/` is the measured package).
|
||||||
|
- E2E runs in isolation: `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` (DB up: `podman compose up -d db`).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/fixtures/extensionless_kb/` exists with `Dockerfile`, `Containerfile`, `Makefile`, `notes.md` (sentinels unique to this suite)
|
||||||
|
- [ ] integration suite green: positive (counts + `dockerfile:1,containerfile:1,md:1` + summary), case, negative, prune
|
||||||
|
- [ ] E2E green in isolation: tree rows (incl. `Makefile` absent), the `dockerfile` format badge in the modal, the raw-content sentinel, the anonymous gate
|
||||||
|
- [ ] all five regression suites green in isolation
|
||||||
|
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] one atomic Conventional Commit with `--no-gpg-sign` (message above); phase dir moved to `.agents/phases/complete/` in the same commit
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Phase 103 — Onboarding chips suggest only session-opening questions, never follow-ups
|
||||||
|
|
||||||
|
**Source:** Owner request (chat, 2026-09-12) — "The suggested questions on the chat page should not include follow-up questions from a session. The problem is this: Users will ask 'What are the correct arguments for qwen 3.8 27b on llama.cpp?' and BOR will answer. Then, users will ask 'What about qwen 3.6 35b?'. That second question … shows up as a suggested question to *start* a conversation but that question makes no sense without the context behind it. The only questions that should show up as a suggested question are those at the very beginning of a session."
|
||||||
|
**Story:** n/a (owner request — tightens the onboarding-chips contract of `80_history_suggestion_chips`; the story file `suggestion-chips.md` cited by phases 05/80 no longer exists in `.agents/user_stories/`).
|
||||||
|
**Context:** `app/api/suggestions.py` — `last_questions(db, limit=3)` walks each saved chat's `messages` in REVERSE (newest-first) and collects every user question (the contract this phase replaces); `GET /api/suggestions` (the `require_user`-gated endpoint, phase 79) with the seed fallback (`get_settings().suggestions` — `BOR_SUGGESTIONS` / built-in) when the walk yields nothing. `app/models.py` — `SavedChat.messages` is the raw `bor.chat.v1` JSONB list in CONVERSATIONAL order (oldest→newest); `SavedChat.title` is the first user question truncated to 120 chars + whitespace-collapsed at save time (`app/api/chats.py` `_auto_title`, `_AUTO_TITLE_MAX = 120`) and user-editable on re-Save. `tests/integration/test_suggestions_api.py` — the phase-80 state matrix (REWRITTEN in task 01). `tests/e2e/test_suggestion_chips.py` — the phase-80 story suite (REWRITTEN in task 03, the phase-76/80 precedent). Docs carrying the "last 3 questions asked" wording: `app/config.py` ~L373-377 (the `suggestions` seed docstring), `.env.example` ~L60 (the `BOR_SUGGESTIONS` comment), `README.md` ~L74.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A suggested question must make sense on its own: the onboarding chips are the **first user question of each saved chat** (the session's opening question) — follow-up questions ("What about …?") can never appear, because they are unanswerable without the session behind them. Walk order, dedup, the cap of 3, and the seed fallback are unchanged.
|
||||||
|
|
||||||
|
## Owner decisions (chat, 2026-09-12 — recorded per AGENTS.md rule 3)
|
||||||
|
- **A1 — openers only:** "The only questions that should show up as a suggested question are those at the very beginning of a session." Each saved chat contributes AT MOST ONE chip: its first user question. In the owner's example, "What are the correct arguments for qwen 3.8 27b on llama.cpp?" (the opener) may chip; "What about qwen 3.6 35b?" (the follow-up) may not.
|
||||||
|
- **A2 — everything else unchanged:** chats still walked newest-`updated_at` first (`created_at` tiebreak), EXACT (case-sensitive) de-dup, cap 3 applied AFTER dedup, seed fallback when the walk yields zero openers (`BOR_SUGGESTIONS` override or built-in default) — all phase-80 contracts survive. The deflection "Maybe try" chips (`app/rag/suggestions.py` `derive_suggestions`, carried in the chat response) are a separate contract and untouched. The FRONTEND is untouched — the chip row renders whatever the endpoint returns (chip sizing/truncation is phase 104's job).
|
||||||
|
- **A3 — the defensive opener rule:** a chat's opener is its first `who == "user"` message whose trimmed `text` is non-blank. A LEADING blank user entry (the UI cannot produce one — `handleSend` trims and guards `!text`) does not disqualify the chat; a record with no non-blank user message (brain-only, or blank-user-only) contributes nothing.
|
||||||
|
- **A4 — read `messages`, not `title`:** `SavedChat.title` is truncated to 120 chars + whitespace-collapsed at save time and is user-editable on re-Save — the chips must carry the EXACT full opener text from the raw `bor.chat.v1` record (the phase-80 precedent: no SQL JSON ops, the deserialized list).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **`app/api/suggestions.py` — the ONLY file changed in `app/`:**
|
||||||
|
- `last_questions` is RENAMED `opening_questions` (the old name would lie about the semantics; the helper is module-internal — its only caller is the endpoint, the tests hit the endpoint). Signature unchanged: `(db: Session, limit: int = 3) -> list[str]`.
|
||||||
|
- **The new walk:** for each chat in `updated_at DESC, created_at DESC` order, walk `chat.messages or []` FORWARD (oldest→newest — `bor.chat.v1` conversational order), take the first entry with `who == "user"` whose trimmed `text` is non-blank (the opener, per A3); if found and not already `seen` (exact, case-sensitive), append it; stop once `limit` UNIQUE openers are collected. Result in encounter order (newest chat first). No other endpoint change: `qs = opening_questions(db)` → `SuggestionList(suggestions=qs if qs else get_settings().suggestions)`.
|
||||||
|
- **Docstrings (the house dense-docstring style):** the module docstring's phase-80 paragraph becomes the opener contract — WHY follow-ups are excluded (a follow-up like "What about X?" is meaningless as a conversation starter — the owner's llama.cpp/qwen example); the function docstring documents the forward walk, the A3 rule, the A4 why-not-title note, and that dedup/cap/order are the phase-80 contracts; the endpoint docstring says "the opening questions of the 3 most recent saved chats — or, before any question has ever been saved, the seed list".
|
||||||
|
- **Not touched:** schemas, models, migrations, `app/rag/suggestions.py` (deflection), all of `frontend/` (the chips render the endpoint's list — truncation of long chips is phase 104), the auth gate.
|
||||||
|
- **Docs (task 02):** `app/config.py` seed docstring, `.env.example` `BOR_SUGGESTIONS` comment, `README.md` chat-features line — "the last 3 questions asked" → "the opening questions of the 3 most recent saved chats (the session openers, newest first)".
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `80_history_suggestion_chips` (complete) — the endpoint, the seed fallback, the dedup/cap/order contracts, the story E2E suite this phase rewrites.
|
||||||
|
- `79_api_tokens` (complete) — the `require_user` gate; the tests sign in first (unchanged).
|
||||||
|
- `102_extensionless_filenames` (todo) — queue order only (numeric); no code dependency (different subsystem).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_opener_extraction.md` — the `opening_questions` rewrite (rename + forward walk + docstrings) + the integration matrix rewrite.
|
||||||
|
2. `02_openers_docs.md` — the "last 3 questions" → "session openers" wording in config / `.env.example` / README.
|
||||||
|
3. `03_e2e_suite_commit.md` — the story-suite rewrite to the opener semantics + regression E2Es + full gate + atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration — REWRITTEN `tests/integration/test_suggestions_api.py`: the full opener matrix (one chat's follow-ups never surface; the cap now binds ACROSS chats; opener dedup; case variants; the A3 leading-blank rule; brain-only → seed; auth 401 — full detail in task 01).
|
||||||
|
- E2E — REWRITTEN `tests/e2e/test_suggestion_chips.py` (the phase-76/80 precedent: a semantic change rewrites the story suite in place), run in isolation: the opener-only core state (a 3-turn chat yields EXACTLY its opener as the single chip), the three-openers state, partial, seed, refetch-on-New-chat, plus the carried-over phase-05 behavior (one-tap submit, keyboard walk, the mobile single horizontal-scroll row).
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the changed file is `app/api/suggestions.py`, every branch of the new walk covered by the matrix.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] One saved chat with 4 user questions → the chip row holds EXACTLY its first question; none of the 3 follow-ups appears (integration + E2E pins).
|
||||||
|
- [ ] Four saved chats (each multi-turn) → exactly the 3 newest chats' OPENERS; the oldest opener is dropped by the cap; no follow-up text anywhere.
|
||||||
|
- [ ] Seed fallback, dedup, case-variant, partial (2 chips), brain-only/blank (A3), and 401 pins all green.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` green in isolation; `test_responsive_polish.py` + `test_chat_persistence.py` green in isolation (regressions).
|
||||||
|
- [ ] The deflection "Maybe try" chips are UNCHANGED (`derive_suggestions` + its suites green).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `fix(chat): onboarding chips are the session-opening questions, never follow-ups`); phase dir moved to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 01 — The `opening_questions` extraction + the integration matrix rewrite
|
||||||
|
|
||||||
|
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The endpoint's helper collects each saved chat's OPENING question (the first non-blank user message) instead of every user question — and the integration matrix pins the new semantics end to end.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/suggestions.py`:
|
||||||
|
- Rename `last_questions` → `opening_questions` (update the endpoint's call site; the helper is module-internal — no other importers; verify with `grep -rn "last_questions" app/ tests/`).
|
||||||
|
- Replace the REVERSE walk with the FORWARD opener walk (the 00_phase.md Design): for each chat in `updated_at DESC, created_at DESC` order, scan `chat.messages or []` oldest→newest; the first entry with `who == "user"` and a non-blank trimmed `text` is the chat's opener (A3 — a leading blank user entry does NOT disqualify the chat; keep scanning until a non-blank user message is found or the list is exhausted). Append the opener if not in `seen` (EXACT, case-sensitive — the phase-80 contract, the docstring keeps the why: case-insensitive dedup would drop a legitimately differently-cased re-ask). Stop once `limit` UNIQUE openers are collected; result in encounter order (newest chat first).
|
||||||
|
- Docstrings (house dense style): the module docstring's phase-80 paragraph → the opener contract + WHY (a follow-up like "What about X?" is meaningless as a conversation starter — cite the owner's qwen/llama.cpp example); the function docstring → the forward walk, the A3 rule, the A4 why-not-`SavedChat.title` note (120-char truncation + whitespace collapse at save, user-editable on re-Save — `app/api/chats.py` `_auto_title`), dedup/cap/order unchanged from phase 80; the endpoint docstring → "the opening questions of the 3 most recent saved chats — or, before any question has ever been saved, the seed list".
|
||||||
|
2. `tests/integration/test_suggestions_api.py` — REWRITE the matrix (keep the module's fixture/helper/auth scaffolding: the `clean_chats` autouse truncate, `_user`/`_brain`/`_add_chat`/`_chips`, the fixed `Q_*` texts — extend with a follow-up-flavored text such as `FOLLOW_UP = "What about qwen 3.6 35b?"` for the core pin):
|
||||||
|
- Module docstring → the new contract (openers only; everything else phase 80).
|
||||||
|
- `test_empty_db_returns_seed` — unchanged.
|
||||||
|
- REPLACE `test_cap_three_and_newest_first_within_a_chat` → `test_a_chats_follow_ups_never_surface`: ONE chat with 4 user questions (brain replies between, the 4th a follow-up-flavored text) → the chips are EXACTLY `[Q_ONE]` — the chat's opener only; Q_TWO/Q_THREE/Q_FOUR (the follow-ups) absent.
|
||||||
|
- NEW `test_cap_three_across_chats`: FOUR chats, each a multi-turn conversation (opener + at least one follow-up), DISTINCT explicit `updated_at` stamps → the chips are exactly the 3 NEWEST chats' openers, newest first; the oldest chat's opener is dropped (the cap now binds across chats); none of the four chats' FOLLOW-UPS appears anywhere.
|
||||||
|
- REWRITE `test_newer_chat_walked_first` → two multi-turn chats with DISTINCT `updated_at` → `[newer chat's opener, older chat's opener]`; the older chat's LAST (newest-looking) question is NOT in the chips.
|
||||||
|
- REWRITE the dedup pair → opener-flavored: the SAME opener text as the first question of two chats → exactly once (a verbatim re-ask as a FOLLOW-UP in the newer chat stays deduped too); a differently-cased OPENER variant → both kept (exact dedup).
|
||||||
|
- Keep the partial pins (2 chats → 2 openers; 1 chat → 1 chip — the follow-ups in those chats do not pad the row).
|
||||||
|
- Keep `test_brain_and_blank_user_texts_contribute_nothing` (all-brain + blank user → nothing).
|
||||||
|
- NEW (A3) `test_leading_blank_user_entry_does_not_disqualify`: a chat whose messages start `[user " ", user Q_TWO, …]` → contributes `Q_TWO` (the first NON-BLANK user message is the opener).
|
||||||
|
- Keep `test_all_brain_deployment_returns_seed` and `test_anonymous_is_401` unchanged.
|
||||||
|
3. Run `uv run pytest tests/integration/test_suggestions_api.py -v` (DB up: `podman compose up -d db`) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the rewritten matrix IS this task's test layer — every branch of the new walk is exercised: zero chats (seed via the empty test), opener found / no non-blank user message (brain-only), leading-blank skip (A3), dedup hit, case variant kept, cap stop (across chats), encounter order.
|
||||||
|
- Coverage: **>90%** on this task's modified code — `app/api/suggestions.py` is fully covered by the matrix (the helper's every branch; the full-suite gate runs at the phase's end).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `opening_questions` walks FORWARD, takes ONE opener per chat (first non-blank user message), keeps the phase-80 order/dedup/cap/seed contracts; the old `last_questions` name is gone from `app/` and `tests/`
|
||||||
|
- [ ] The docstrings state the WHY (follow-ups are context-free) and the A3/A4 rules
|
||||||
|
- [ ] `uv run pytest tests/integration/test_suggestions_api.py -v` green (DB up)
|
||||||
|
- [ ] No other `app/` or `tests/` file changed (the E2E rewrite is task 03; docs are task 02)
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Task 02 — The "session openers" wording: config / `.env.example` / README
|
||||||
|
|
||||||
|
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The three doc surfaces that still say "the last 3 questions asked" (the phase-80 wording) describe the NEW contract — the chips are the session-OPENING questions; the seed's meaning ("shown only before any question has ever been saved") is unchanged.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` (~L373-377, the `suggestions` field comment): the seed docstring currently reads "shown ONLY while no saved chat has ever asked a question — after that, ``GET /api/suggestions`` serves the last 3 questions asked (deployment-wide, newest first)" → "serves the opening questions of the 3 most recent saved chats (the session openers — a chat's first user question; follow-ups never chip — phase 103; deployment-wide, newest first)". Keep the `BOR_SUGGESTIONS` override sentence.
|
||||||
|
2. `.env.example` (~L60, the `BOR_SUGGESTIONS` comment): "…# JSON seed chips — shown only before any question has been saved (phase 80)" → append "; after that the chips are the 3 newest chats' opening questions (phase 103)".
|
||||||
|
3. `README.md` (~L74, the chat-features line): "follow the last 3 questions asked — on a fresh deployment they seed from …" → "follow the opening questions of the 3 most recent saved chats (each chat's first question — a follow-up never chips, phase 103) — on a fresh deployment they seed from …" (keep the surrounding sentence intact; match the README's voice).
|
||||||
|
4. Grep sweep: `grep -rn "last 3 questions" app/ .env.example README.md` → ZERO hits (the phase-80 wording is fully retired; the phase records in `.agents/phases/complete/` may keep the historical phrasing — they are read-only and out of scope).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- No new logic — doc/comment changes only; `uv run pytest tests/unit/test_config.py -q` green (the config docstring change is inert; `tests/unit/test_config.py` pins the seed list shape, untouched).
|
||||||
|
- Coverage: **>90%** on `app/` (untouched — the gate runs at the phase's end).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All three surfaces say "opening questions / session openers" (the WHY — follow-ups are meaningless without their session — appears at least in the `config.py` docstring)
|
||||||
|
- [ ] `grep -rn "last 3 questions" app/ .env.example README.md` → zero hits
|
||||||
|
- [ ] `uv run pytest tests/unit/test_config.py -q` green
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 03 — The story-suite rewrite to the opener semantics + full gate + atomic commit
|
||||||
|
|
||||||
|
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The dedicated Playwright suite proves the new contract in the browser (a multi-turn chat yields EXACTLY its opener as the single chip — the owner's "What about …?" follow-up can never surface), then the phase closes with every gate green and one atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_suggestion_chips.py` — REWRITE in place (the phase-76/80 precedent: a semantic change rewrites the story suite in place). Keep the module scaffolding (the `SEED` literal pin, the `auth_helpers.login` sign-in, the autouse `saved_chats` TRUNCATE fixture, the fixture-KB import, the mock-LLM marker constant, the run-in-isolation header). New module docstring: the opener contract + the four states. The states:
|
||||||
|
- **seed** (unchanged): fresh DB (no saved chats) → the chip texts equal the built-in default seed list EXACTLY, rendered as accessible buttons in the `role="list"` group (the phase-05 component contract).
|
||||||
|
- **opener-only** (the NEW core state): ONE saved chat with a 3-turn conversation — Q1 (the opener, e.g. "What are the correct arguments for qwen 3.8 27b on llama.cpp?") → brain → Q2 (a follow-up, e.g. "What about qwen 3.6 35b?") → brain → Q3 → brain — saved via the API (`POST /api/chats`) → a fresh page load shows EXACTLY ONE chip: Q1. Assert the chip count == 1 AND the exact text Q1; Q2/Q3 are absent (the owner's scenario, pinned).
|
||||||
|
- **three-openers** (replaces the old "last-3" state): THREE saved chats, each multi-turn (opener + at least one follow-up), DISTINCT `updated_at` (the API stamps them on save — save oldest→newest) → exactly 3 chips = the three openers, newest `updated_at` first; none of the chats' FOLLOW-UPS appears.
|
||||||
|
- **partial** (kept, re-scoped): exactly 2 saved chats → exactly 2 chips (the two openers — NO seed top-up; the follow-ups in those chats do not pad the row).
|
||||||
|
- **refetch** (kept): boot with the seed chips → save a multi-turn chat (opener Q) via the API → click New chat (`#new-chat-btn`) → the chips now are exactly Q, and the request log shows a SECOND `GET /api/suggestions` (the boot fetch was the first).
|
||||||
|
- Carried-over story behavior (unchanged semantics from the phase-05/80 suites): one-tap submit (chip click → composer filled → submitted → the mock-LLM brain bubble with the `MOCK_ANSWER_MARKER`), Tab+Enter keyboard reachability of the chips (the keyboard-walk assertion), and the mobile single horizontal-scroll row (the ≤640px viewport assertion).
|
||||||
|
2. Regression E2Es — run EACH in isolation (`--no-cov`, DB up), must stay green WITHOUT edits:
|
||||||
|
- `tests/e2e/test_responsive_polish.py` (chip visibility + the chip AA-contrast pair — it waits on `#suggestions .suggestion-chip` and measures colors; the seed chips on a truncated DB are still rendered),
|
||||||
|
- `tests/e2e/test_chat_persistence.py` (the deflection "Maybe try" chips + the empty-state chips after New chat — the separate contract, untouched).
|
||||||
|
If one of them asserts the OLD chip CONTENT semantics (not visibility/contrast/flow), update ONLY that assertion to the opener contract and note it in the commit message.
|
||||||
|
3. Full gate: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
4. `git diff --stat` — limited to `app/api/suggestions.py`, `tests/integration/test_suggestions_api.py`, `tests/e2e/test_suggestion_chips.py`, `app/config.py`, `.env.example`, `README.md`, and the phase files (no migrations, no `frontend/` diff, no other `app/` diff).
|
||||||
|
5. Move the phase dir to `.agents/phases/complete/` and make ONE atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `fix(chat): onboarding chips are the session-opening questions, never follow-ups`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the rewritten `tests/e2e/test_suggestion_chips.py` IS the phase's story suite — the opener-only state is the load-bearing pin (the owner's exact scenario); run in isolation per AGENTS.md rule 9.
|
||||||
|
- Coverage: **>90%** on `app/` — `app/api/suggestions.py` is the only `app/` delta and every branch of the new walk is covered (the integration matrix + the endpoint's seed fallback).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` green in isolation (DB up) — including the opener-only state (1 chip for a 3-turn chat) and the three-openers state
|
||||||
|
- [ ] `test_responsive_polish.py` + `test_chat_persistence.py` green in isolation (with edits only if their assertions measured the old content semantics — noted in the commit)
|
||||||
|
- [ ] The deflection "Maybe try" chips are UNCHANGED (`app/rag/suggestions.py` untouched — `git diff` shows no delta there)
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] `git diff --stat` scoped per Work step 4; phase dir moved to `.agents/phases/complete/`; one atomic `--no-gpg-sign` commit
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# Phase 104 — Suggestion chips stay single-line (hover reveals the full text) + the composer enforces the 4,000-char question cap
|
||||||
|
|
||||||
|
**Source:** Owner request (chat, 2026-09-12) — "…tweak the size of the suggestion chips on the chat page. Some users submit truly massive queries and the 'chips' become more like 'chonks'. Hovering over the chips should still show the full message so users still have a way to see it. In fact, I would suggest you put a character cap on the chat submission box, to prevent ultra-long context overflowing queries. Users should still be able to paste code example of a few dozen lines, but nothing much longer than that."
|
||||||
|
**Story:** n/a (owner request — extends the suggestion-chips component contract of `05_story_suggestion_chips` / `80_history_suggestion_chips`; the question-cap backstop pattern of `83_chat_save_payload_limits`).
|
||||||
|
**Context:** `frontend/assets/styles.css` — the base `.suggestion-chip` rule (~L1225: the pill, `min-height: 44px`, text WRAPS inside today — a 400-char question becomes a tall multi-line block), the `.maybe-try .suggestion-chip` override (~L702: `min-width: 0; max-width: 100%` — the phase-07 overflow fix), the mobile row (~L4331: `.suggestions { flex-wrap: nowrap; overflow-x: auto; }` + `.suggestion-chip { flex: 0 0 auto; }`). `frontend/assets/app.js` — `renderChips` (~L1003-1040: the shared chip component, onboarding row + "Maybe try" row; NO `title`/`aria-label` today), the source-chip `title` + `aria-label`-when-truncated pattern (~L1331-1338 — the house precedent to copy), `autoGrow` (~L1258, 192px max), the four `input.value` mutation sites (the `input` listener ~L2484, `submitSuggestion` ~L1013, the `handleSend` clear ~L2118, the `startNewChat` clear ~L2015), the out-of-turn `showErrorBanner` precedent (`saveAsDoc` ~L731). `frontend/index.html` — the composer (~L271-285: `#message-input` textarea with NO maxlength; the `.chat-bottom` sticky unit wraps the chat-actions row + the form, ~L205-215); the theme inputs' `maxlength`-mirrors-the-server comment precedent (~L1054). `app/schemas.py` — `ChatRequest.message: Field(min_length=1, max_length=4000)` (L75 — the server backstop ALREADY in force: a >4000-char question 422s today with zero UI feedback — the banner just shows "Brain's API answered with HTTP 422."); `HistoryTurn.text` stays 32_000 (HISTORY turns may be long answers — only the CURRENT question is capped at 4000). `tests/unit/test_schemas.py` — no 4000/4001 `message` boundary pin today. `tests/unit/test_pinned_composer.py` — pins `.chat-bottom` as the LAST CHILD of `.chat-shell` and the `.composer` sticky CSS (a new child INSIDE `.chat-bottom` breaks neither).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Two coupled fixes on the chat page. (1) Long questions no longer balloon the suggestion chips into multi-line "chonks": every chip is single-line, ellipsized at the row edge, and the FULL text is one hover away (native `title` tooltip) plus the accessible name. (2) The composer makes the question-length cap VISIBLE — the server already hard-caps the current question at 4,000 chars (`ChatRequest.message`); the textarea gains `maxlength="4000"`, a counter appears near the cap, and a guard covers the one path that bypasses `maxlength` (the chip one-tap fill) — so no user ever meets the 422 blind.
|
||||||
|
|
||||||
|
## Owner decisions (chat, 2026-09-12 — recorded per AGENTS.md rule 3)
|
||||||
|
- **A1 — single-line chips, never chonks:** chips never wrap — `white-space: nowrap`, ellipsized at the row edge, never taller than the one-line 44px pill, at every viewport width.
|
||||||
|
- **A2 — hover shows the full text:** "Hovering over the chips should still show the full message so users still have a way to see it" — a native `title` tooltip carrying the FULL text on every chip (the house source-chip pattern), plus `aria-label` = full text when the chip is visually truncated (screen readers).
|
||||||
|
- **A3 — the question cap is 4,000 chars, mirroring the existing server cap:** the cap must allow "code example of a few dozen lines, but nothing much longer" — 4,000 chars ≈ a 50-line block at 80 chars/line, and the server ALREADY rejects >4,000 (`ChatRequest.message max_length=4000`, pre-existing, untouched). **NO schema change:** the server cap stays the backstop (this phase pins it at the boundary, task 03); the UI becomes the visible contract (`maxlength` + counter + guard).
|
||||||
|
- **A4 — counter behavior:** hidden while the RAW length < 3,200 (80% of the cap — no noise on normal use); shows `len/4000` from 3,200; at/over the cap shows `len/4000 — character limit` in the `--err-*` semantic family (B3: the COPY change carries the state — text + color, never color alone; the executor verifies + records the AA ratio of the chosen `--err-*` pairing in the CSS comment). Count the RAW value (no trim): raw ≤ 4,000 ⟹ trimmed ≤ 4,000, so a raw count is a safe superset of what the server validates.
|
||||||
|
- **A5 — the over-cap guard:** `maxlength` constrains typing + pastes, but a programmatic `input.value = …` bypasses it — the one reachable path is `submitSuggestion` (a chip >4,000 chars; possible only via an admin-authored `BOR_SUGGESTIONS` seed — history chips are ≤4,000 by construction, having passed the same cap when asked). `handleSend` guards: trimmed text > the cap → the out-of-turn error banner (the `saveAsDoc` precedent), NO turn, the input KEEPS the text (the user trims it) — never stale (PLAN §7.4).
|
||||||
|
- **A6 — shared page:** `frontend/assets/shared.js` renders "Maybe try" chips as plain non-interactive spans (`pointer-events: none`, scoped `.shared-shell` — owner-locked phase 51: a guest tapping a chip has nowhere to go; a tooltip could never show) — UNTOUCHED; the CSS sizing applies to those pills automatically (shared stylesheet).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### Chip sizing (task 01) — `frontend/assets/styles.css`
|
||||||
|
- The base `.suggestion-chip` rule gains: `white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: 100%; min-width: 0;`
|
||||||
|
- `overflow: hidden` (≠ visible) zeroes the flex item's automatic minimum size, so `max-width: 100%` actually binds: in the desktop wrap row (`.suggestions`, the 46rem column) a long chip clips at the column edge; in the ≤640px row (`flex-wrap: nowrap; overflow-x: auto`) it clips at the VISIBLE width and the row scrolls (the phase-07 contract).
|
||||||
|
- One line per chip at every width — the 44px `min-height` pill stays a pill.
|
||||||
|
- The `.maybe-try .suggestion-chip` override (`min-width: 0; max-width: 100%`) is fully subsumed by the new base rule → **delete it**, folding its phase-07 overflow-fix provenance into the base rule's comment (house: provenance lives with the contract).
|
||||||
|
- The mobile rule `.suggestion-chip { flex: 0 0 auto; }` (≤640px block) stays.
|
||||||
|
- No other CSS in this task.
|
||||||
|
|
||||||
|
### Tooltip + a11y (task 02) — `frontend/assets/app.js` `renderChips`
|
||||||
|
- After `btn.textContent = text;`: `btn.title = text;` — the FULL text, always (A2; the house source-chip pattern, ~L1331).
|
||||||
|
- After `container.appendChild(btn)`: `if (btn.scrollWidth > btn.clientWidth) btn.setAttribute("aria-label", text);` — the source-chip truncation pattern (~L1336-1338): the screen-reader name is the full text when the visible text is clipped; attribute absent when not clipped (textContent already carries the full text).
|
||||||
|
- No other `renderChips` change (one-tap submit, `role="listitem"`, the container contract).
|
||||||
|
|
||||||
|
### The composer cap (task 03) — `frontend/index.html` + `frontend/assets/app.js`
|
||||||
|
- `index.html`:
|
||||||
|
- `#message-input` gains `maxlength="4000"`, with the provenance comment (house pattern — the theme inputs' "maxlength=300 mirrors the server's 300-char"): "maxlength=4000 mirrors ChatRequest.message max_length=4000 (app/schemas.py) — the server 422s beyond; the #char-count line makes the cap visible (app.js updateCharCount)".
|
||||||
|
- The counter element, INSIDE `.chat-bottom` between the chat-actions row and `<form class="composer">` (a new child of the sticky unit — the sticky contract + `test_pinned_composer.py`'s last-child-of-`.chat-shell` pin are untouched):
|
||||||
|
```html
|
||||||
|
<p class="char-count" id="char-count" hidden></p>
|
||||||
|
```
|
||||||
|
with the phase-104 comment: hidden until 80% of the cap (app.js); the `.is-max` state uses the `--err-*` family + a copy change (B3); NOT a live region — per-keystroke feedback is decorative, the over-cap failure announces through the error banner (`role=alert`).
|
||||||
|
- `app.js`:
|
||||||
|
- Constants next to `autoGrow` (provenance comment: mirrors the schema cap; the threshold is 80% — owner 2026-09-12 A4):
|
||||||
|
```js
|
||||||
|
const MAX_QUESTION_CHARS = 4000;
|
||||||
|
const CHAR_COUNT_SHOW_AT = 3200;
|
||||||
|
```
|
||||||
|
- Element grabber alongside the existing ones (`~L301`): `const charCountEl = document.querySelector("#char-count");`
|
||||||
|
- `updateCharCount()` (next to `autoGrow`): `len = input.value.length` (RAW); `len < CHAR_COUNT_SHOW_AT` → `hidden = true` + drop `.is-max`; else unhide, `textContent = "${len}/${MAX_QUESTION_CHARS}"` and, when `len >= MAX_QUESTION_CHARS`, append ` — character limit` + add `.is-max`. Over-cap (the chip path) shows the honest `len` (e.g. `5123/4000 — character limit`).
|
||||||
|
- Call `updateCharCount()` at the EXACT four `input.value` mutation sites (each already calls `autoGrow()` there): the `input` listener, `submitSuggestion`, the `handleSend` clear, the `startNewChat` clear.
|
||||||
|
- `handleSend` guard, immediately AFTER `if (!text || sendBtn.disabled) return;` (BEFORE the clear — the input keeps the text):
|
||||||
|
```js
|
||||||
|
if (text.length > MAX_QUESTION_CHARS) {
|
||||||
|
showErrorBanner("Questions are limited to 4,000 characters — trim the question and try again.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
with the A5 comment (maxlength caps typing + pastes; the programmatic chip fill bypasses it — this guard is the never-stale backstop).
|
||||||
|
- `styles.css` (this task): `.char-count` — `margin: 0; text-align: right; font-size: 0.75rem; line-height: 1.2; color: var(--ink-soft);` (the counter sits on the app background behind `.chat-bottom` — the executor verifies the chosen pairing is ≥4.5:1 there and records the ratio in the comment, house style); `.char-count.is-max { color: var(--err-ink); }` (verify + record the ratio on the same background; the copy change already carries the state — B3).
|
||||||
|
- `tests/unit/test_schemas.py`: the `ChatRequest.message` boundary pin — exactly 4,000 chars validates; 4,001 → a pydantic `ValidationError` naming `message` (the A2/A3 backstop — the cap the UI now mirrors).
|
||||||
|
|
||||||
|
### Unit source pins (house pattern — one new file, `tests/unit/test_chip_sizing_question_cap.py`, extended per task)
|
||||||
|
- Task 01 (CSS): the `.suggestion-chip` rule block contains `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`, `max-width: 100%`, `min-width: 0`; `css.count(".maybe-try .suggestion-chip") == 0` (the subsumed override is gone); the ≤640px block keeps `.suggestion-chip { flex: 0 0 auto; }`.
|
||||||
|
- Task 02 (renderChips): the `renderChips` block contains `btn.title = text` and the `scrollWidth > btn.clientWidth` → `setAttribute("aria-label"` pattern.
|
||||||
|
- Task 03: `index.html` — the `#message-input` block carries `maxlength="4000"`; `#char-count` exists, is `hidden` by default, and sits inside `.chat-bottom` before `#composer` (source order). `app.js` — `MAX_QUESTION_CHARS = 4000` + `CHAR_COUNT_SHOW_AT = 3200`; the guard `text.length > MAX_QUESTION_CHARS` + the banner copy "4,000 characters"; `updateCharCount()` defined AND called at all four mutation sites (pin each site's context). **Single-source cross-file pin:** the HTML `maxlength` value == the JS `MAX_QUESTION_CHARS` value (regex-parse both files and compare — the cap lives in one place conceptually).
|
||||||
|
|
||||||
|
### E2E (task 04) — new dedicated suite `tests/e2e/test_chip_sizing_question_cap.py`
|
||||||
|
House scaffolding (DB up, mock LLM, the fixture-KB module import, admin login — the chips are `require_user`, the phase-80 autouse `saved_chats` truncate):
|
||||||
|
1. **Truncated chip + tooltip (A1/A2 core):** save via the API a chat whose FIRST user question is LONG (300+ chars — a readable repeated phrase) with a short follow-up; reload → exactly ONE onboarding chip (phase-103 opener semantics):
|
||||||
|
- computed style `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`;
|
||||||
|
- `scrollWidth > clientWidth` (visually clipped);
|
||||||
|
- single line: `44 <= clientHeight <= 60` (a one-line pill is the 44px min-height; a wrapped two-liner is ≥ ~76px);
|
||||||
|
- `title` attribute == the full long text; `aria-label` == the full long text.
|
||||||
|
Fresh-DB contrast pin: a short SEED chip has `title` set and NO `aria-label` (not truncated).
|
||||||
|
2. **Counter threshold (A4):** `#char-count` hidden at boot; 100 chars typed → still hidden; exactly 3,500 chars in the box (a dispatched `input` event) → visible, text `3500/4000`, NO `.is-max`.
|
||||||
|
3. **Hard cap through the input path (A3):** `keyboard.insert_text("x" * 6000)` (CDP `Input.insertText` = the paste path — `maxlength` applies) → the textarea holds EXACTLY 4,000 chars; the counter reads `4000/4000 — character limit` + `.is-max`. Submit → the 4,000-char question passes the server cap (NO 422 error state) → the mock answer streams to `done` → the input is cleared and the counter hidden again.
|
||||||
|
4. **The over-cap guard (A5):** `page.evaluate` sets `#message-input.value = "x".repeat(5000)` + dispatches an `input` event (the programmatic path `maxlength` cannot stop) → counter `5000/4000 — character limit` + `.is-max` → click Send → the error banner shows the "4,000 characters" copy; NO brain bubble appended; the input STILL holds the 5,000 chars (kept for trimming — never stale).
|
||||||
|
5. **Short-flow regression:** a short question submits cleanly; the counter never becomes visible.
|
||||||
|
|
||||||
|
**Regressions (run in isolation, must stay green):** `tests/e2e/test_suggestion_chips.py` (the chip contract — phase 103's rewrite), `tests/e2e/test_pinned_composer.py` (the sticky cluster now hosts the counter), `tests/e2e/test_responsive_polish.py` (the mobile chip row + the chip AA-contrast pairs — the chip colors are unchanged), `tests/e2e/test_chat_history.py` (the send flow). Unit: `tests/unit/test_pinned_composer.py` (the `.chat-bottom` pins — untouched structure) + any `tests/unit/` source pin that conflicts with the added `app.js` lines (the executor runs `uv run pytest tests/unit/ -q` and fixes only genuine conflicts — the additions live INSIDE existing functions, so substring/context pins should survive).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `103_suggestions_session_openers` (todo) — the onboarding chips become session openers; this phase's long-chip E2E state (a 300+ char opener chip) builds on that contract and runs AFTER it. Queue order only at the code level (different files), but the E2E fixtures assume the phase-103 semantics.
|
||||||
|
- `80_history_suggestion_chips` (complete) — the chip component + the E2E fixture pattern.
|
||||||
|
- `83_chat_save_payload_limits` (complete) — the boundary-cap pin pattern (`tests/unit/test_schemas.py`).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_chip_ellipsis_css.md` — the base chip rule (single-line ellipsis) + the subsumed-override deletion + the CSS unit pins.
|
||||||
|
2. `02_chip_tooltip_aria.md` — `renderChips` full-text `title` + `aria-label`-when-truncated + the unit pins.
|
||||||
|
3. `03_composer_question_cap.md` — `maxlength` + the counter (HTML/JS/CSS) + the `handleSend` guard + the unit pins + the `test_schemas.py` boundary pin.
|
||||||
|
4. `04_e2e_suite_commit.md` — the dedicated E2E suite + regression E2Es + full gate + atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_chip_sizing_question_cap.py` (new; per-task pins above) + the `ChatRequest.message` 4,000/4,001 boundary in `tests/unit/test_schemas.py`.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_chip_sizing_question_cap.py` green in isolation (the five states above); the four named regression suites green in isolation.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the `app/` delta is nil-to-trivial (no `app/` code changes; the gate stays green).
|
||||||
|
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A 300+ char onboarding chip renders single-line (`clientHeight` ≤ 60px, ellipsized, `scrollWidth > clientWidth`) with `title` + `aria-label` == the full text (E2E).
|
||||||
|
- [ ] `#message-input` hard-caps at 4,000 through the paste path; submitting at the cap streams a mock answer (no 422); the counter is hidden again after send (E2E).
|
||||||
|
- [ ] The over-cap guard: a programmatic 5,000-char fill + Send → the cap banner, no turn, the text kept (E2E).
|
||||||
|
- [ ] The `ChatRequest.message` 4,000/4,001 boundary is pinned in `tests/unit/test_schemas.py`; the HTML `maxlength` == the JS constant (cross-file pin).
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` green in isolation; `test_suggestion_chips.py`, `test_pinned_composer.py`, `test_responsive_polish.py`, `test_chat_history.py` green in isolation.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `git diff --stat` limited to `frontend/`, the new unit file, `tests/unit/test_schemas.py`, the new E2E file, phase files (NO `app/` code diff, no migration, no `shared.js` diff).
|
||||||
|
- [ ] One atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `feat(chat): single-line suggestion chips with full-text tooltips + the visible 4,000-char question cap`); phase dir moved to `.agents/phases/complete/`.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 01 — Single-line ellipsized chips: the base `.suggestion-chip` rule + the subsumed override deletion
|
||||||
|
|
||||||
|
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A suggestion chip is ONE line at every viewport width — long text ellipsizes at the row edge instead of wrapping the pill into a multi-line "chonk" (owner A1).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — the base `.suggestion-chip` rule (~L1225) gains four declarations (keep the existing ones — font/weight/color/background/border/radius/padding/min-height/transition):
|
||||||
|
```css
|
||||||
|
white-space: nowrap;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
max-width: 100%;
|
||||||
|
min-width: 0;
|
||||||
|
```
|
||||||
|
Extend the rule's comment (house provenance style): phase 104 (owner 2026-09-12) — chips never wrap; `overflow: hidden` zeroes the flex item's automatic minimum size so `max-width: 100%` binds (desktop wrap row: 100% = the 46rem column; the ≤640px row: 100% = the visible width, the row scrolls — the phase-07 overflow contract now lives here).
|
||||||
|
2. DELETE the `.maybe-try .suggestion-chip { min-width: 0; max-width: 100%; }` override (~L702) — fully subsumed by the new base rule; fold its phase-07 provenance sentence into the base rule's comment (the step-1 comment) so the history is not lost.
|
||||||
|
3. Leave the ≤640px block's `.suggestion-chip { flex: 0 0 auto; }` (~L4331) and `.suggestions` row rule untouched.
|
||||||
|
4. `tests/unit/test_chip_sizing_question_cap.py` — CREATE the phase's unit-pins file (module docstring: pins the phase-104 chip-sizing contract in the static frontend — the house source-pin pattern) with the CSS pins:
|
||||||
|
- the `.suggestion-chip` rule block contains `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`, `max-width: 100%`, `min-width: 0` (reuse the `_rule`-style block extraction from `tests/unit/test_pinned_composer.py` / `test_wide_column_css.py` — read one and match its helpers);
|
||||||
|
- `css.count(".maybe-try .suggestion-chip") == 0` (the subsumed override is gone);
|
||||||
|
- the ≤640px mobile block still contains `.suggestion-chip { flex: 0 0 auto; }`.
|
||||||
|
5. Run `uv run pytest tests/unit/ -q` — green (the new file + no existing pin broken).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new pins ARE this task's test layer (they guard the CSS bytes — the house source-pin pattern).
|
||||||
|
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `.suggestion-chip` is single-line + ellipsized + capped at the row edge (the five declarations pinned); the `.maybe-try .suggestion-chip` override deleted with its provenance folded into the base comment
|
||||||
|
- [ ] `tests/unit/test_chip_sizing_question_cap.py` created and green; `uv run pytest tests/unit/ -q` green
|
||||||
|
- [ ] no other file changed (JS/HTML/E2E are later tasks)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 02 — Full-text hover tooltip + accessible name on every suggestion chip
|
||||||
|
|
||||||
|
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The full text of a (possibly ellipsized) chip is always recoverable: a native `title` tooltip on hover (owner A2) and the full text as the accessible name when the visible text is clipped (the house source-chip pattern).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `renderChips` (~L1024-1040), inside the per-chip loop:
|
||||||
|
- after `btn.textContent = text;` add `btn.title = text;` — the FULL text, always (the hover contract; the source-chip precedent is `chip.title = label` at ~L1331).
|
||||||
|
- after `container.appendChild(btn);` add the truncation-aware accessible name (the source-chip precedent at ~L1336-1338):
|
||||||
|
```js
|
||||||
|
if (btn.scrollWidth > btn.clientWidth) btn.setAttribute("aria-label", text);
|
||||||
|
```
|
||||||
|
(When NOT clipped the attribute stays absent — the `textContent` already carries the full text, so screen readers read it; the attribute is belt-and-suspenders for the clipped case, exactly like the source chips.)
|
||||||
|
- One short comment on the pair: phase 104 (owner 2026-09-12) — the single-line chip clips long questions; `title` is the hover reveal, `aria-label` the clipped-case accessible name (the source-chip pattern).
|
||||||
|
- Touch NOTHING else in `renderChips` (one-tap submit via `submitSuggestion`, `role="listitem"`, the container-replace contract, the `onSelect` hook).
|
||||||
|
2. `tests/unit/test_chip_sizing_question_cap.py` — extend with the `renderChips` pins (extract the `renderChips` function body from `app.js` the way the neighboring unit files do):
|
||||||
|
- the body contains `btn.title = text`;
|
||||||
|
- the body contains the `btn.scrollWidth > btn.clientWidth` guard setting `aria-label` (pin the `setAttribute("aria-label"` call inside that guard — a small slice of the function text, the house "pin the contract words" style).
|
||||||
|
3. Run `uv run pytest tests/unit/ -q` — green (including the existing `tests/unit/test_shared_page.py` pin that `renderChips` does NOT leak into `shared.js` — `shared.js` is untouched by this task).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new pins guard the `app.js` bytes; the full E2E hover/tooltip behavior is task 04.
|
||||||
|
- Coverage: **>90%** on `app/` (untouched).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] every chip rendered by `renderChips` (onboarding row AND "Maybe try" row) carries `title` = the full text, and `aria-label` = the full text when clipped
|
||||||
|
- [ ] the unit pins are green; `uv run pytest tests/unit/ -q` green
|
||||||
|
- [ ] no other file changed
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
# Task 03 — The visible 4,000-char question cap: `maxlength` + counter + the over-cap guard + the server-boundary pin
|
||||||
|
|
||||||
|
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The question-length cap the server ALREADY enforces (`ChatRequest.message max_length=4000` — a >4,000-char question 422s today with zero UI feedback) becomes VISIBLE in the composer: the textarea hard-caps input/paste, a counter appears near the cap, and a guard covers the one path that bypasses `maxlength` (the chip one-tap fill). The server cap is untouched and pinned at the boundary.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html`:
|
||||||
|
- `#message-input` (~L273-279) gains `maxlength="4000"`, with the provenance comment (house pattern — the theme inputs' "maxlength=300 mirrors the server's 300-char", ~L1054): `maxlength=4000 mirrors ChatRequest.message max_length=4000 (app/schemas.py) — the server 422s beyond; the #char-count line makes the cap visible (app.js updateCharCount)`.
|
||||||
|
- The counter element — INSIDE `.chat-bottom` (~L215), between the closing `</div>` of `.chat-actions` and the composer comment/`<form class="composer">`:
|
||||||
|
```html
|
||||||
|
<!-- Phase 104 (owner 2026-09-12): the question-length counter —
|
||||||
|
hidden until 80% of the 4,000-char cap (app.js
|
||||||
|
updateCharCount); the .is-max state uses the --err-* family
|
||||||
|
PLUS a copy change (B3: text + color, never color alone). NOT
|
||||||
|
a live region: per-keystroke feedback is decorative — the
|
||||||
|
over-cap failure path announces through the error banner
|
||||||
|
(role=alert). -->
|
||||||
|
<p class="char-count" id="char-count" hidden></p>
|
||||||
|
```
|
||||||
|
(A new child of the sticky unit — the unit stays the LAST child of `.chat-shell` and the `.composer` sticky CSS is untouched, so `tests/unit/test_pinned_composer.py` + `tests/e2e/test_pinned_composer.py` stay green; a `hidden` `<p>` adds zero height.)
|
||||||
|
2. `frontend/assets/app.js`:
|
||||||
|
- Constants directly above `autoGrow` (~L1258) with the provenance comment (mirrors the schema cap — the executor must NOT change `app/schemas.py`; the threshold is 80% of the cap, owner A4):
|
||||||
|
```js
|
||||||
|
const MAX_QUESTION_CHARS = 4000;
|
||||||
|
const CHAR_COUNT_SHOW_AT = 3200; // 80% of the cap — visible only when it matters
|
||||||
|
```
|
||||||
|
- Element grabber alongside the existing ones (next to `const suggestionsEl = …` ~L301): `const charCountEl = document.querySelector("#char-count");`
|
||||||
|
- `updateCharCount()` next to `autoGrow`:
|
||||||
|
```js
|
||||||
|
function updateCharCount() {
|
||||||
|
// RAW length (no trim): raw ≤ cap ⟹ trimmed ≤ cap, so the raw
|
||||||
|
// count is a safe superset of what the server validates.
|
||||||
|
const len = input.value.length;
|
||||||
|
if (len < CHAR_COUNT_SHOW_AT) {
|
||||||
|
charCountEl.hidden = true;
|
||||||
|
charCountEl.classList.remove("is-max");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
charCountEl.hidden = false;
|
||||||
|
const atMax = len >= MAX_QUESTION_CHARS;
|
||||||
|
charCountEl.classList.toggle("is-max", atMax);
|
||||||
|
charCountEl.textContent = atMax
|
||||||
|
? `${len}/${MAX_QUESTION_CHARS} — character limit`
|
||||||
|
: `${len}/${MAX_QUESTION_CHARS}`;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(Over-cap — reachable only via the programmatic chip-fill path — shows the honest `len`, e.g. `5123/4000 — character limit`.)
|
||||||
|
- Call `updateCharCount()` at the EXACT four `input.value` mutation sites (each already calls `autoGrow()` — add the call right after it, or fold both into the listener body):
|
||||||
|
a. the `input` listener (~L2484: `input.addEventListener("input", autoGrow)` — e.g. `input.addEventListener("input", () => { autoGrow(); updateCharCount(); })`);
|
||||||
|
b. `submitSuggestion` (~L1013-1015, after `input.value = text; autoGrow();`);
|
||||||
|
c. `handleSend` (~L2118-2119, after the post-send clear `input.value = ""; autoGrow();`);
|
||||||
|
d. `startNewChat` (~L2015-2016, after `input.value = ""; autoGrow();`).
|
||||||
|
- The `handleSend` guard — immediately AFTER `if (!text || sendBtn.disabled) return;` (~L2114) and BEFORE the clear (the input keeps the text for trimming):
|
||||||
|
```js
|
||||||
|
// maxlength caps typing + pastes, but a programmatic fill (the chip
|
||||||
|
// one-tap path) bypasses it — this guard is the never-stale backstop
|
||||||
|
// (PLAN §7.4): no turn, no clear, the user trims the kept text.
|
||||||
|
if (text.length > MAX_QUESTION_CHARS) {
|
||||||
|
showErrorBanner("Questions are limited to 4,000 characters — trim the question and try again.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(Out-of-turn banner = the `saveAsDoc` precedent, ~L731; the banner is cleared by the next user action — the existing `clearErrorBanner` call sites.)
|
||||||
|
3. `frontend/assets/styles.css` — the counter rules (near the composer styles, ~L1242+):
|
||||||
|
```css
|
||||||
|
/* Phase 104: the question-length counter — right-aligned above the
|
||||||
|
composer, hidden until 80% of the 4,000-char cap (app.js).
|
||||||
|
[executor: verify + record the ratio] --ink-soft on the app
|
||||||
|
background behind .chat-bottom is ≥4.5:1 (WCAG AA). */
|
||||||
|
.char-count { margin: 0; text-align: right; font-size: 0.75rem; line-height: 1.2; color: var(--ink-soft); }
|
||||||
|
.char-count.is-max { color: var(--err-ink); }
|
||||||
|
```
|
||||||
|
Verify BOTH pairings against the actual background the counter sits on (the app `--bg` behind `.chat-bottom` — the chat column area) and record each ratio in the comment (house style). The `.is-max` state pairs the color with the "— character limit" COPY change (B3 — never color alone).
|
||||||
|
4. `tests/unit/test_chip_sizing_question_cap.py` — extend with:
|
||||||
|
- `index.html` pins: the `#message-input` textarea block carries `maxlength="4000"`; `#char-count` exists, is `hidden` by default, and appears INSIDE `.chat-bottom` before `#composer` (source order).
|
||||||
|
- `app.js` pins: `MAX_QUESTION_CHARS = 4000` and `CHAR_COUNT_SHOW_AT = 3200`; the guard `text.length > MAX_QUESTION_CHARS` with the banner copy "4,000 characters"; `updateCharCount` defined AND its call present in each of the four mutation-site contexts (the listener, `submitSuggestion`, `handleSend`, `startNewChat` — pin each site's slice).
|
||||||
|
- **Single-source cross-file pin:** regex-parse the HTML `maxlength="(\d+)"` on the `#message-input` block and the JS `MAX_QUESTION_CHARS = (\d+)` and assert they are EQUAL (the cap lives in one place conceptually — the schema is the source, both mirror it).
|
||||||
|
5. `tests/unit/test_schemas.py` — the `ChatRequest.message` boundary pin (currently unpinned): exactly 4,000 chars validates; 4,001 → a pydantic `ValidationError` naming `message` (the backstop the UI now mirrors — the phase-83 boundary-pin pattern).
|
||||||
|
6. Run `uv run pytest tests/unit/ -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new pins (HTML/JS/CSS bytes + the cross-file constant match + the schema boundary) ARE this task's test layer; the behavioral E2E is task 04.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code changes — `app/schemas.py` is untouched, only test-pinned).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `#message-input` has `maxlength="4000"` + the provenance comment; the counter element sits in `.chat-bottom` above the composer, hidden by default
|
||||||
|
- [ ] `updateCharCount` fires at all four mutation sites; the `handleSend` guard (banner + no turn + no clear) is in place; the counter CSS is AA-verified with recorded ratios
|
||||||
|
- [ ] The HTML `maxlength` == the JS `MAX_QUESTION_CHARS` (cross-file pin); the 4,000/4,001 `ChatRequest.message` boundary is pinned in `test_schemas.py`
|
||||||
|
- [ ] `uv run pytest tests/unit/ -q` green; no `app/` file changed
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Task 04 — The dedicated E2E suite + regression E2Es + full gate + atomic commit
|
||||||
|
|
||||||
|
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The browser proves the whole contract — single-line ellipsized chips with full-text tooltips, the hard 4,000-char cap through the paste path, the counter states, and the over-cap guard — then the phase closes with every gate green and one atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_chip_sizing_question_cap.py` — NEW dedicated suite (house scaffolding: module docstring stating the phase-104 contract + the run-in-isolation command; DB up `podman compose up -d db`; the deterministic mock LLM — the fixture-KB import pattern from a sibling chat suite, e.g. `tests/e2e/test_suggestion_chips.py`; admin login via `e2e.auth_helpers.login` — the chips are `require_user`; the phase-80 autouse `saved_chats` TRUNCATE fixture so each test starts from — and leaves — an empty deployment):
|
||||||
|
- **`test_long_chip_is_single_line_ellipsized_with_full_text_tooltip`** (A1/A2 core): save via the API a chat whose FIRST user question is LONG (300+ chars — a readable repeated phrase, e.g. `"What are the correct arguments for " * 20 + "qwen on llama.cpp?"`) with one short follow-up turn; reload the chat page → exactly ONE onboarding chip (the phase-103 opener semantics — the follow-up never surfaces):
|
||||||
|
- computed style: `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`;
|
||||||
|
- `scrollWidth > clientWidth` (visually clipped — 300+ chars of ~0.5rem/char far exceeds the 46rem column);
|
||||||
|
- single line: `44 <= clientHeight <= 60` (a one-line pill sits at the 44px `min-height`; a wrapped two-liner is ≥ ~76px — the chonk);
|
||||||
|
- `get_attribute("title")` == the full long text (the hover reveal);
|
||||||
|
- `get_attribute("aria-label")` == the full long text (the clipped-case accessible name).
|
||||||
|
- **`test_short_seed_chip_has_tooltip_but_no_aria_label`**: fresh DB (seed chips) → a short chip has `title` set AND no `aria-label` (not truncated — the attribute is absent by design).
|
||||||
|
- **`test_counter_hidden_below_threshold_and_visible_above`** (A4): on the empty-state chat page, `#char-count` is hidden; type 100 chars → still hidden; put exactly 3,500 chars in `#message-input` (a dispatched `input` event — `locator.fill` does this) → `#char-count` visible, text `3500/4000`, NO `.is-max` class.
|
||||||
|
- **`test_paste_path_hard_caps_at_the_cap_and_sends`** (A3): `page.keyboard.insert_text("x" * 6000)` (CDP `Input.insertText` = the paste path — `maxlength` applies) → `#message-input` holds EXACTLY 4,000 chars; the counter reads `4000/4000 — character limit` + `.is-max`. Click Send → NO 422 error state (the 4,000-char question passes the server cap) → the mock answer streams to `done` (the brain bubble + the "Deterministic mock answer for E2E" marker) → `#message-input` cleared and `#char-count` hidden again.
|
||||||
|
- *Executor note:* if `insert_text` proves not to respect `maxlength` on the pinned Chromium build (it goes through the browser's input pipeline, like a paste — expect it to work), fall back to pinning the attribute (`maxlength == "4000"`) + a `fill`-based counter check, and record the deviation in the phase record — the guard test below still covers the bypass path.
|
||||||
|
- **`test_over_cap_programmatic_fill_hits_the_guard`** (A5): `page.evaluate` sets `#message-input.value = "x".repeat(5000)` + dispatches an `input` event (the programmatic path `maxlength` cannot stop — the chip one-tap fill) → counter `5000/4000 — character limit` + `.is-max` → click Send → the error banner shows the "4,000 characters" cap copy; NO brain bubble appended; `#message-input` STILL holds the 5,000 chars (kept for trimming — never stale, PLAN §7.4).
|
||||||
|
- **`test_short_flow_never_shows_the_counter`**: type a short question → submit → the mock answer lands; `#char-count` never becomes visible during the turn.
|
||||||
|
2. Regression E2Es — run EACH in isolation (`--no-cov`, DB up), must stay green:
|
||||||
|
- `tests/e2e/test_suggestion_chips.py` (the chip contract — phase 103's rewrite; the onboarding row this phase restyles),
|
||||||
|
- `tests/e2e/test_pinned_composer.py` (the sticky cluster now hosts the counter),
|
||||||
|
- `tests/e2e/test_responsive_polish.py` (the mobile chip row + the chip AA-contrast pairs — the chip colors are unchanged by this phase),
|
||||||
|
- `tests/e2e/test_chat_history.py` (the send/save flow).
|
||||||
|
Fix ONLY a regression whose assertion measured the OLD chip wrapping (e.g. a pin that asserted a multi-line chip height) — the asserted BEHAVIOR (a chip exists, is clickable, AA contrast) must survive; note any such fix in the commit message.
|
||||||
|
3. Full gate: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
4. `git diff --stat` — limited to `frontend/`, `tests/unit/test_chip_sizing_question_cap.py`, `tests/unit/test_schemas.py`, `tests/e2e/test_chip_sizing_question_cap.py`, and the phase files (NO `app/` code diff, no migration, no `shared.js` diff, no `pyproject.toml`/`uv.lock`).
|
||||||
|
5. Move the phase dir to `.agents/phases/complete/` and make ONE atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `feat(chat): single-line suggestion chips with full-text tooltips + the visible 4,000-char question cap`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the new suite (five tests above) IS the phase's story suite — run in isolation per AGENTS.md rule 9; the four regression suites re-prove the untouched contracts (chip component, sticky composer, contrast, send flow).
|
||||||
|
- Coverage: **>90%** on `app/` (the `app/` delta is nil — the gate must simply stay green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` green in isolation (DB up)
|
||||||
|
- [ ] `test_suggestion_chips.py`, `test_pinned_composer.py`, `test_responsive_polish.py`, `test_chat_history.py` each green in isolation
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] `git diff --stat` scoped per Work step 4 (no `app/` code diff, no `shared.js` diff)
|
||||||
|
- [ ] Phase dir moved to `.agents/phases/complete/`; one atomic `--no-gpg-sign` commit
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-14).
|
||||||
|
**Context:** The single filesystem walk choke point is `iter_importable_files` (`app/rag/importer.py` L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like `.esphome/.espressif/**`) AND hidden files — plus the well-known `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`); the extension filter (`BOR_IMPORT_EXTENSIONS`, the A9 family) then admits the rest. `import_sources` (L196+) walks every root twice when `progress` is set (the phase-64 pre-walk for the `total` denominator uses the EXACT same rules), collects `(source, rel)` into `seen`, and `_prune` (`prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root `ignore_by_root: dict[str, list[str]]` map from the `git_sources` rows in one loop — the Sync button (`app/api/sync.py::_run_sync` L233-263) and the CLI (`scripts/import_docs.py::_resolve_sources` L181-226, consumed at L269/L332-334); the map is keyed by `str(root)` — the root path string exactly as passed to `import_sources` — with extend-on-collision for shared roots. `scripts/load_test_kb.py` calls `import_sources` with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources` L514+); the per-row "Ignore paths" button (`makeRow` L357+, `ignoreBtn` ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, `id` null, get no control — phase 89 A3); non-empty lists render a `N ignored` text tag in the source cell (`.git-source-ignore-count`, `frontend/assets/styles.css` L2686, text + background, never color alone). The admin-only `PATCH /api/git-sources/{source_id}` (phase 89 A5, `app/api/git_sources.py` L374-393) today takes the REQUIRED `ignore_paths` replace list; the read shapes are `GitSourceRow` (`app/schemas.py` L508) and `GitSourceOut` (L490), the create body `GitSourceIn` (L450, optional `ignore_paths`). Alembic head is `0018`. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Each stored source carries an **index-hidden-folders flag**, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. `EXCLUDED_DIRS` stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `104_chip_sizing_question_cap` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged).
|
||||||
|
- `89_source_ignore_paths` (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the `PATCH` route this phase extends.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **Flag semantics (locked, A1).** `include_hidden=True` lifts ONLY the dot-prefixed-component skip in `iter_importable_files`: the existing check `any(part.startswith(".") or part in excluded for part in rel.parts)` becomes dot-aware only when the flag is False — e.g. `any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts)`. Consequences, all deliberate:
|
||||||
|
- Files INSIDE hidden dirs become importable (`.esphome/esp.md` indexed when ON).
|
||||||
|
- Hidden files with an importable extension also become importable (`.notes.md` — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like `.env` has no A9 extension and is never indexed).
|
||||||
|
- `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) are skipped in BOTH states — caches/VCS internals are never content.
|
||||||
|
- The `ignore` tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when `include_hidden=True`.
|
||||||
|
- **Storage (task 01).** `git_sources.include_hidden` — BOOLEAN NOT NULL, server default `false`, `Mapped[bool]` (the `documents.is_summary` Boolean precedent, `app/models.py` L136). Alembic `0019_git_source_include_hidden.py` (revises `0018`): `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`; downgrade drops the column. Existing rows read `False` (A4).
|
||||||
|
- **Importer signature (task 02).**
|
||||||
|
- `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False)` — default `False` keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause.
|
||||||
|
- `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None)` — the map is keyed by **`str(root)`** with the SAME keying convention as `ignore_by_root`; an internal `_include_hidden_for_root(root, include_hidden_by_root) -> bool` (default `False`) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so `files_total` never disagrees with the walk. `seen` is untouched in shape → `_prune` prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated.
|
||||||
|
- **API contract (task 03).**
|
||||||
|
- Schemas (`app/schemas.py`): `GitSourceIn.include_hidden: bool | None = Field(default=None)` (create-time, optional — absent → stored `False`); `GitSourceOut.include_hidden: bool`; `GitSourceRow.include_hidden: bool` (env-fallback rows report `False` — no DB row to store a flag on).
|
||||||
|
- The PATCH body model is RENAMED `GitSourceIgnoreIn` → `GitSourcePatchIn` (grep-verified: referenced only in `app/schemas.py` and `app/api/git_sources.py` — import L119 + `patch_git_source` L376) and gains:
|
||||||
|
- `ignore_paths: list[str] | None = Field(default=None)` — **absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5** (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool.
|
||||||
|
- `include_hidden: bool | None = Field(default=None)` — absent/None = unchanged; present = set.
|
||||||
|
- Both absent → 200 no-op (row untouched).
|
||||||
|
- `GET /api/git-sources` — DB rows report the stored flag; env rows `False`. `POST /api/git-sources` — both kinds accept `include_hidden`; stored `bool(payload.include_hidden)`.
|
||||||
|
- `PATCH /api/git-sources/{source_id}` (existing route, still behind `require_admin`) — applies each PRESENT field independently (404 unknown id unchanged); 200 → `GitSourceOut` (id, url, added_at, ignore_paths, include_hidden).
|
||||||
|
- **Callers (task 04).**
|
||||||
|
- `app/api/sync.py::_run_sync` — in the existing per-row loop that builds `ignore_by_root` (L233-252), build `include_hidden_by_root: dict[str, bool]` with the SAME `str(root)` keying: `include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden)` (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass `include_hidden_by_root=…` to `import_sources` (L263). Module docstring (L36-40) updated.
|
||||||
|
- `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple `(sources, ignore_by_root, include_hidden_by_root)` (manual `--source` → `(sources, {}, {})` — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; `main` unpacks (L269) and passes the map (L332-334); docstrings updated (module + `_resolve_sources` L181).
|
||||||
|
- `scripts/load_test_kb.py` — untouched (defaults).
|
||||||
|
- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: a **native labeled checkbox** in the actions cell, DOM order **Hidden · Ignore paths · Remove** (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last):
|
||||||
|
- `<label class="git-source-hidden">` wrapping `<input type="checkbox" class="git-source-hidden-box">` + visible text "Hidden"; the checkbox's `aria-label` is `Index hidden folders for ${kindLabel} source: ${value}` (setAttribute — never innerHTML; `value` is the git URL or local path, credential-safety discipline), `checked = s.include_hidden === true`; a `title` on the label explains in plain words ("When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.").
|
||||||
|
- When `s.include_hidden === true`, the source cell also shows a **"hidden on" text tag** (`.git-source-hidden-count` — the `.git-source-ignore-count` idiom: text + background, never color alone, WCAG 1.4.1), next to the `N ignored` tag.
|
||||||
|
- **§7.4 never-stale lifecycle** — `toggleHidden(s, box)`: on `change`, the box disables immediately (no double-flip); `PATCH /api/git-sources/${s.id}` with `{"include_hidden": box.checked}`; on 200 → clear the error line, `await loadSources()` (the row re-renders from the server), THEN `announce("Hidden folders enabled|disabled for <value>.")` (the phase-89 last-announce order — the confirmation lands after the reload's "N sources listed."); on non-2xx or network failure → the server detail (or the canned "Could not reach the server — the setting was not changed.") into the new page-level `role="alert"` line, and the box **reverts to the server state** (`box.checked = s.include_hidden === true`) and re-enables — the UI never claims a state the server didn't save.
|
||||||
|
- `frontend/index.html` — one new element after the `#git-sources-table-wrap` region: `<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>` (reuses the existing `.git-source-error` styling). `frontend/assets/styles.css` — near the phase-89 block (~L2657-2695): `.git-source-hidden` (inline-flex, ~44px hit height matching the action buttons, visible label), `.git-source-hidden input[type="checkbox"]` (sized, `accent-color` on the brand pair — verify + record the AA ratio in the comment, house style), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), focus ring via the GLOBAL `:focus-visible` rule (L146 — no per-control rule needed), and `.git-source-hidden-count` (copy of the `.git-source-ignore-count` rule, provenance comment citing phase 105).
|
||||||
|
- Env-fallback rows (`id` null) get **no** checkbox — the existing "from .env" tag stays (A3).
|
||||||
|
- **NOT touched:** the RAG view (`sources.js`), `app/rag/retriever.py`, the chunker, the KB tree/catalog + agent tools (DB-driven — they pick up newly indexed hidden docs for free), `app/rag/git_sources.py` (clone/pull only, no walk), the upload run (phase 90 — no scan), `AGENTS.md`, `.agents/PLAN.md`, any completed phase.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_include_hidden_column.md` — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`) + default/round-trip tests.
|
||||||
|
2. `02_importer_include_hidden.md` — `iter_importable_files`/`import_sources` flag support (walk + progress pre-walk + prune interaction + ignore composition) + unit & integration tests.
|
||||||
|
3. `03_include_hidden_api.md` — schemas (`In`/`Out`/`Row` + the `GitSourcePatchIn` rename with optional fields) + GET/POST/PATCH wiring + integration tests.
|
||||||
|
4. `04_include_hidden_pipelines.md` — wire the per-row flag into `_run_sync` and `scripts/import_docs.py` + integration tests.
|
||||||
|
5. `05_hidden_toggle_sources_ui.md` — the per-row "Hidden" checkbox on the Sources page (tag + §7.4 lifecycle + error line + a11y) + source-level unit pins.
|
||||||
|
6. `06_e2e_hidden_folders_toggle.md` — dedicated Playwright suite `tests/e2e/test_hidden_folders_toggle.py` (run in isolation), regressions, full gate, atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_importer_include_hidden.py` (new, task 02): `iter_importable_files` on a tmp fixture tree — default OFF pins today's behavior byte-identically (hidden dir + hidden file skipped), ON admits both, `EXCLUDED_DIRS` skipped in BOTH states, `ignore` tuple still bites when ON, extension filter unchanged (`.env` never indexed); the `str(root)` keying + default-False for unlisted roots; `tests/unit/test_hidden_folders_toggle.py` (new, task 05): source-level pins for the JS/HTML/CSS wiring (house pattern — read the assets as text), incl. the single-source cross-file check that the aria-label template names the source value; task 01's pins extend the existing model/migration test surfaces (a fresh row reads `include_hidden is False`; an explicit `True` round-trips).
|
||||||
|
- Integration — `tests/integration/test_importer_include_hidden.py` (new, task 02): `import_sources` against a fixture dir — hidden file produces NO `Document`/`Chunk` rows by default; WITH the map it is embedded + summarized normally; previously indexed hidden file + flag OFF → pruned on the next run; progress `total` agrees with the walk in both states; sources not in the map behave exactly as before. `tests/integration/test_git_sources_api.py` (extended, task 03): GET reports `False` default / stored `True`; POST create round-trip; PATCH bool-only, list-only, both, neither (no-op 200), the phase-89 422s unchanged for present lists, 404, anonymous 403. `tests/integration/test_sync_api.py` (extended, task 04): a local row with a hidden dir — flag False syncs zero hidden docs, True syncs them; `tests/integration/test_import_docs_git.py` (extended, task 04): the CLI DB-row path with the flag set.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_hidden_folders_toggle.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov`.
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API/pipeline additions are fully unit+integration covered).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A stored source (git, local, or uploaded) has a "Hidden" checkbox next to its "Ignore paths" button: flipping it on persists (`PATCH` 200, the source cell shows the "hidden on" tag, `GET /api/git-sources` round-trips `include_hidden: true`); the failure path reverts the box and announces the error in a `role="alert"` line.
|
||||||
|
- [ ] A sync (button or CLI) with the flag OFF indexes nothing with a dot-prefixed component (no `documents`/`chunks` rows — the byte-identical default); with the flag ON, `.hidden/note.md` is indexed, embedded, and summarized like any visible file and shows up in the KB catalog; `EXCLUDED_DIRS` content is excluded in both states.
|
||||||
|
- [ ] A2: a previously indexed hidden file is PRUNED from the KB on the next sync after the flag flips OFF (`detail.pruned` increments; the catalog no longer lists it).
|
||||||
|
- [ ] API contracts hold: PATCH bool-only / list-only / both / neither; the phase-89 fixed-detail 422s unchanged for present lists; 404 unknown id; anonymous 403 on the route.
|
||||||
|
- [ ] Env-fallback rows render no checkbox (the "from .env" tag stays); the Sources page stays WCAG-clean (visible label, focus-visible, text never color alone).
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation (DB up); regression suites `test_source_ignore_paths.py`, `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — what "on" means (owner-confirmed 2026-09-14).** The flag lifts the dot-prefixed-component skip for that source — files inside hidden folders AND hidden files with an importable extension become indexable; `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) stay excluded regardless; the extension filter always applies.
|
||||||
|
- **A2 — toggling off prunes (owner-confirmed).** Previously indexed hidden files leave the KB on the next sync (the `seen`-set prune — the phase-89 A2 / A9 precedent), exactly like newly-ignored files.
|
||||||
|
- **A3 — per stored row only (owner-confirmed).** Every stored `git_sources` row (git, local, uploaded) gets the toggle; env-fallback rows have no DB row and get no control (the phase-89 A3 precedent).
|
||||||
|
- **A4 — default off (owner-confirmed).** `include_hidden` defaults to `false` for all existing rows — byte-identical behavior until the owner flips it; the flag takes effect on the NEXT sync (no auto-sync, same as the ignore list).
|
||||||
|
- **A5 — control idiom (owner-confirmed).** A native labeled checkbox ("Hidden") in the actions cell, left of the "Ignore paths" button (Remove stays last), with a "hidden on" text tag in the source cell when enabled — not a styled switch.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
|
||||||
|
```
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 01 — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "…There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Persist the per-source hidden-folders flag: one additive, reversible BOOLEAN column on `git_sources`, server-defaulted to `false` so every pre-phase-105 row imports byte-identically (A4).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — the `GitSource` class (L220-259): add the column directly AFTER `ignore_paths` (L248-251), mirroring its docstring/provenance style (`Boolean` is already imported, L78):
|
||||||
|
```python
|
||||||
|
#: Index hidden (dot-prefixed) paths from this source (phase 105,
|
||||||
|
#: A1): True → the walk (app.rag.importer.iter_importable_files)
|
||||||
|
#: does not skip dot-prefixed components — files inside hidden
|
||||||
|
#: folders AND hidden files with an importable extension are
|
||||||
|
#: indexed; ``EXCLUDED_DIRS`` (``.venv``, ``node_modules``,
|
||||||
|
#: ``.git``, …) are excluded in BOTH states, and the extension
|
||||||
|
#: filter always applies. Takes effect on the next sync (no
|
||||||
|
#: auto-sync — the ignore-list precedent, phase 89). Server
|
||||||
|
#: default False: every pre-phase-105 row imports exactly as
|
||||||
|
#: before (A4).
|
||||||
|
include_hidden: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=False, server_default=text("false"), nullable=False
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(Extend the class docstring's one-line field inventory if it names `ignore_paths` — the module header L19 does: add `include_hidden` (phase 105) to the parenthetical.)
|
||||||
|
2. `alembic/versions/0019_git_source_include_hidden.py` (NEW — the house format of `0013_git_source_ignore_paths.py`, one additive reversible column):
|
||||||
|
- `revision = "0019"`, `down_revision = "0018"`.
|
||||||
|
- `upgrade()`: `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`.
|
||||||
|
- `downgrade()`: `op.drop_column("git_sources", "include_hidden")`.
|
||||||
|
- Module docstring: the phase-89-0013 provenance style (what the flag is, A1/A4, one additive reversible column).
|
||||||
|
3. Tests — extend the existing model/migration test surfaces (find the current `GitSource` default-pinning tests — the phase-89 column tests live in the `git_sources` unit/integration suites; add alongside them):
|
||||||
|
- A freshly inserted `GitSource` row (no `include_hidden` passed) reads `include_hidden is False` (the Python `default=False` AND the server default agree).
|
||||||
|
- An explicit `include_hidden=True` round-trips through the DB (`session.add` → `commit` → fresh session → `True`).
|
||||||
|
- The migration applies on the test DB from head (`alembic upgrade head` is part of the standard test-db fixture setup — if the suite asserts the column set of `git_sources`, add `include_hidden` to the expected set).
|
||||||
|
4. Run `uv run pytest tests/unit/ -q && uv run alembic upgrade head` (against the dev/test DB per the house quick reference) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit/integration: the default + round-trip pins above ARE this task's layer (no importer behavior yet — that is task 02).
|
||||||
|
- Coverage: **>90%** on `app/` (model-only change; the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GitSource.include_hidden` exists with `server_default=text("false")`, `nullable=False`, and the A1/A4 provenance comment
|
||||||
|
- [ ] `alembic/versions/0019_git_source_include_hidden.py` upgrades from `0018` and downgrades cleanly; the dev/test DB is at head
|
||||||
|
- [ ] Fresh-row-default-False and explicit-True round-trip tests pass; existing `git_sources` suites stay green
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Task 02 — Importer flag support: `iter_importable_files` / `import_sources` honor `include_hidden` per root
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the walk choke point flag-aware: `iter_importable_files` gains `include_hidden` (default `False` — every existing caller byte-identical) and `import_sources` gains `include_hidden_by_root` (same `str(root)` keying as phase 89's `ignore_by_root`), used by BOTH the phase-64 progress pre-walk and the processing loop. Pruning falls out for free through the untouched `seen` set (A2).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — `iter_importable_files` (L161-194):
|
||||||
|
- Signature: `def iter_importable_files(root: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS, ignore: tuple[str, ...] = (), include_hidden: bool = False) -> list[Path]:`
|
||||||
|
- The skip check (L184) becomes flag-aware — ONE expression, byte-identical when the flag is False:
|
||||||
|
```python
|
||||||
|
if any(
|
||||||
|
(not include_hidden and part.startswith(".")) or part in excluded
|
||||||
|
for part in rel.parts
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
```
|
||||||
|
- Docstring: the "Skips:" sentence gains — "…when ``include_hidden`` is False (the default): any path with a dot-prefixed component; when True, dot-prefixed components are ADMITTED (files inside hidden folders, and hidden files) and only *excluded* is consulted (A1 — caches/VCS internals are never content). The *ignore* tuple composes additively in both states."
|
||||||
|
2. `app/rag/importer.py` — `import_sources` (L196+):
|
||||||
|
- Keyword-only param after `ignore_by_root`: `include_hidden_by_root: dict[str, bool] | None = None`.
|
||||||
|
- New private helper next to `_ignore_for_root` (L145-160), same style:
|
||||||
|
```python
|
||||||
|
def _include_hidden_for_root(
|
||||||
|
root: Path, include_hidden_by_root: dict[str, bool] | None
|
||||||
|
) -> bool:
|
||||||
|
"""The per-root hidden-folders flag (phase 105, A1).
|
||||||
|
|
||||||
|
Keyed by ``str(root)`` — the root string exactly as the caller
|
||||||
|
passed it in ``sources`` (the ``_ignore_for_root`` convention,
|
||||||
|
phase 89): ``True`` only for roots the caller lists as True;
|
||||||
|
unlisted/``None`` roots are ``False`` — every existing caller
|
||||||
|
behaves byte-identically (A4).
|
||||||
|
"""
|
||||||
|
return bool((include_hidden_by_root or {}).get(str(root), False))
|
||||||
|
```
|
||||||
|
- The progress pre-walk (the `if progress is not None:` block, ~L244-252) passes `include_hidden=_include_hidden_for_root(root, include_hidden_by_root)` to its `iter_importable_files` call — `files_total` must agree with the walk in both states.
|
||||||
|
- The processing loop: alongside `ignore = _ignore_for_root(root, ignore_by_root)` (~L272), add `include_hidden = _include_hidden_for_root(root, include_hidden_by_root)` and pass both into the `iter_importable_files` call (~L273-275).
|
||||||
|
- `import_sources` docstring: after the `ignore_by_root` paragraph, the mirror paragraph: "``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the stored flag: ``True`` admits dot-prefixed components for that root (``EXCLUDED_DIRS`` and the extension filter still apply; the ignore tuple composes additively). Unlisted/``None`` roots are ``False`` — byte-identical to pre-phase-105. A file that was indexed with the flag ON and is walked again with it OFF simply never enters ``seen``, so the next ``prune=True`` run deletes its row automatically (A2 — the A9/phase-89 precedent)."
|
||||||
|
- Module docstring "Scope" paragraph (L20-22): append the flag clause ("…skipped, plus the well-known exclusion list — UNLESS the source's phase-105 hidden-folders flag admits dot-prefixed paths; the exclusion list always applies").
|
||||||
|
3. `tests/unit/test_importer_include_hidden.py` (NEW) — `iter_importable_files` on a `tmp_path` tree (extensions `frozenset({".md"})`), fixture layout:
|
||||||
|
```
|
||||||
|
visible.md .hidden/note.md .notes.md
|
||||||
|
.venv/junk.md node_modules/x.md .hidden/.deep.md
|
||||||
|
keep/ok.md
|
||||||
|
```
|
||||||
|
- **Default (flag False) — today's behavior pinned byte-identically:** result == `[keep/ok.md, visible.md]` (sorted); hidden dir, hidden file, `.venv`, `node_modules` all absent.
|
||||||
|
- **Flag True (A1):** result contains `.hidden/note.md`, `.notes.md`, `.hidden/.deep.md`, `visible.md`, `keep/ok.md`; STILL excludes `.venv/junk.md` and `node_modules/x.md` (EXCLUDED_DIRS in both states).
|
||||||
|
- **Composition with `ignore`:** flag True + `ignore=(".hidden",)` → `.hidden/*` gone, `.notes.md` present (additive).
|
||||||
|
- **Extension filter unchanged:** flag True with `.env`-like file `.env` (no A9 extension) → never listed.
|
||||||
|
- `_include_hidden_for_root` unit pins: `None` map → False; unlisted root → False; listed `True` → True; listed `False` → False; keying is `str(root)` (two `Path` objects, equal strings, same answer).
|
||||||
|
4. `tests/integration/test_importer_include_hidden.py` (NEW) — `import_sources` against a fixture dir (the `tests/integration/test_importer_ignore.py` harness — fake `Embedder` from `tests/fakes.py` or the module's own fake, real test DB):
|
||||||
|
- Layout: `visible.md`, `.hidden/note.md` (non-markdown `.hidden/data.yaml` too, to prove the summary path runs for admitted hidden files), `.venv/junk.md`.
|
||||||
|
- **Run 1 (default, no map):** `Document` rows exist for `visible.md` ONLY — no row for `.hidden/note.md` (A4 byte-identical); `summary` stats unchanged.
|
||||||
|
- **Run 2 (`include_hidden_by_root={str(root): True}`):** `.hidden/note.md` + `.hidden/data.yaml` get `Document` + `Chunk` rows (embedded via the fake), the yaml gets a `summary` row (the phase-30 path), `.venv/junk.md` still absent (A1).
|
||||||
|
- **A2 prune:** with the rows from run 2 committed, run 3 with `prune=True` and NO map (flag off) → `.hidden/*` rows deleted (`summary.pruned >= 2`), `visible.md` untouched.
|
||||||
|
- **Progress agreement:** with `progress=…` set, `total` equals the visible-only count in run 1's rules and the extended count in run 2's rules (the pre-walk uses the same flag).
|
||||||
|
- **Unlisted roots untouched:** two roots, map lists only one as True — the other root's hidden file stays out (per-root, not global).
|
||||||
|
5. Run `uv run pytest tests/unit/test_importer_include_hidden.py tests/integration/test_importer_include_hidden.py tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -q` — green (the phase-89 suites are the byte-identical regression proof).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration as above are this task's layer; API/pipeline/UI come later.
|
||||||
|
- Coverage: **>90%** on `app/` (the new helper + both call sites are fully covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `iter_importable_files(..., include_hidden=False)` is the default and its default-state result is byte-identical to pre-task (phase-89 importer suites green)
|
||||||
|
- [ ] `include_hidden=True` admits dot-prefixed components, keeps `EXCLUDED_DIRS` + extension filter + `ignore` tuple in force (A1)
|
||||||
|
- [ ] `import_sources(include_hidden_by_root=…)` drives BOTH the pre-walk and the loop through `_include_hidden_for_root`; unlisted/`None` → False
|
||||||
|
- [ ] Flag-off re-run prunes previously indexed hidden docs (A2); progress `total` agrees with the walk in both states
|
||||||
|
- [ ] `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Task 03 — API: `include_hidden` on GET/POST/PATCH (the `GitSourcePatchIn` rename)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button)…"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Expose the flag through the admin API: `GET`/`POST` carry it, and the existing `PATCH /api/git-sources/{id}` becomes the single per-row edit endpoint for BOTH the ignore list and the toggle — each field optional, present-field-wins, so the toggle's PATCH sends only `{"include_hidden": …}` and the phase-89 dialog's PATCH (which always sends the list) keeps byte-identical semantics.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py`:
|
||||||
|
- `GitSourceIn` (L450-477): add after `ignore_paths`:
|
||||||
|
```python
|
||||||
|
include_hidden: bool | None = Field(default=None)
|
||||||
|
```
|
||||||
|
+ docstring clause: "``include_hidden`` (phase 105) is optional at create time (absent → stored ``False`` — A4)."
|
||||||
|
- `GitSourceOut` (L490-506): add `include_hidden: bool` (after `ignore_paths`) + docstring clause (the stored flag; `False` for a row created without it).
|
||||||
|
- `GitSourceRow` (L508-528): add `include_hidden: bool` + docstring clause (**env-fallback rows report `False`** — no DB row to store a flag on, the `ignore_paths: []` precedent).
|
||||||
|
- **RENAME** `GitSourceIgnoreIn` (L530-543) → `GitSourcePatchIn` and rework its body (grep-verified references: only `app/schemas.py` + `app/api/git_sources.py` L119/L376 — no tests import it):
|
||||||
|
```python
|
||||||
|
class GitSourcePatchIn(BaseModel):
|
||||||
|
"""``PATCH /api/git-sources/{source_id}`` body (phase 89 A5;
|
||||||
|
extended phase 105).
|
||||||
|
|
||||||
|
Each field is independent and OPTIONAL: absent/None leaves the
|
||||||
|
row's value unchanged; PRESENT applies. ``ignore_paths`` when
|
||||||
|
present keeps the phase-89 A5 REPLACE semantics (the body list,
|
||||||
|
normalized + A4-validated, becomes the row's whole list — empty
|
||||||
|
list clears all; every pre-phase-105 client always sends the
|
||||||
|
list, so their behavior is byte-identical). ``include_hidden``
|
||||||
|
(phase 105) when present sets the stored flag. Both absent →
|
||||||
|
200 no-op (the row is untouched).
|
||||||
|
"""
|
||||||
|
|
||||||
|
ignore_paths: list[str] | None = Field(default=None)
|
||||||
|
include_hidden: bool | None = Field(default=None)
|
||||||
|
```
|
||||||
|
2. `app/api/git_sources.py`:
|
||||||
|
- Import L119: `GitSourceIgnoreIn` → `GitSourcePatchIn`.
|
||||||
|
- `list_git_sources` (L202-245): DB row construction (~L226-233) gains `include_hidden=row.include_hidden`; the env-fallback row (~L240) gains `include_hidden=False`; the endpoint docstring's field list mentions it.
|
||||||
|
- `create_git_source` (L249-280): response construction (L278) gains `include_hidden=row.include_hidden`.
|
||||||
|
- `_create_git_row` (L315, row construction ~L326-335) and `_create_local_row` (L340, ~L358-368): both gain `include_hidden=bool(payload.include_hidden)` (absent → `False`, A4).
|
||||||
|
- `patch_git_source` (L374-393) — new body:
|
||||||
|
```python
|
||||||
|
def patch_git_source(
|
||||||
|
source_id: uuid.UUID,
|
||||||
|
payload: GitSourcePatchIn,
|
||||||
|
db: Session = Depends(get_db), # noqa: B008
|
||||||
|
) -> GitSourceOut:
|
||||||
|
"""Edit one source's ignore list and/or hidden-folders flag.
|
||||||
|
|
||||||
|
Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
|
||||||
|
id; each PRESENT body field applies independently —
|
||||||
|
``ignore_paths`` REPLACES the list (normalized + A4-validated,
|
||||||
|
fixed 422 details); ``include_hidden`` sets the flag. Both
|
||||||
|
absent → 200 no-op. Returns the updated row's public shape
|
||||||
|
(id, url, added_at, ignore_paths, include_hidden).
|
||||||
|
"""
|
||||||
|
row = db.get(GitSource, source_id)
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(status_code=404, detail="git source not found")
|
||||||
|
if payload.ignore_paths is not None:
|
||||||
|
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
|
||||||
|
if payload.include_hidden is not None:
|
||||||
|
row.include_hidden = payload.include_hidden
|
||||||
|
db.commit()
|
||||||
|
db.refresh(row)
|
||||||
|
return GitSourceOut(
|
||||||
|
id=row.id,
|
||||||
|
url=row.url,
|
||||||
|
added_at=row.added_at,
|
||||||
|
ignore_paths=row.ignore_paths,
|
||||||
|
include_hidden=row.include_hidden,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- Router module docstring (L1-30ish — the contract list): the PATCH line extends to "the ignore list (replace) and/or the hidden-folders flag (phase 105) — each optional, present-wins".
|
||||||
|
3. `tests/integration/test_git_sources_api.py` — extend the phase-89 PATCH section (the suite already pins 404 + anonymous 403 + the A4 fixed-detail 422s):
|
||||||
|
- `GET` — a fresh stored row reports `include_hidden: false`; an env-fallback row (table empty + `BOR_GIT_SOURCES` monkeypatched, the suite's existing pattern) reports `include_hidden: false`.
|
||||||
|
- `POST` — `kind="local"` create with `include_hidden: true` → 201 body `include_hidden: true`; without the field → `false` (A4).
|
||||||
|
- `PATCH` matrix (one stored row, list `["a/b"]`, flag `false`):
|
||||||
|
- `{"include_hidden": true}` → 200, flag `true`, list UNCHANGED `["a/b"]` (the toggle's exact payload).
|
||||||
|
- `{"ignore_paths": ["c/d"]}` → 200, list REPLACED, flag UNCHANGED `false` (the dialog's exact payload — byte-identical to phase 89).
|
||||||
|
- `{"ignore_paths": [], "include_hidden": true}` → both applied.
|
||||||
|
- `{}` → 200 no-op (list + flag unchanged).
|
||||||
|
- the phase-89 422s still fire for a PRESENT bad list (>200 entries / empty-after-normalization / >500-char entry — the fixed details), and a bad list does NOT half-apply the flag (assert the flag is untouched after a 422 with both fields present).
|
||||||
|
- 404 unknown id; anonymous `PATCH` 403 (the existing pin already covers the route — extend it to include a bool-only body so the pin proves the toggle path is gated too).
|
||||||
|
4. Run `uv run pytest tests/integration/test_git_sources_api.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration as above is this task's layer (the unit schema pins ride on the integration 422/shape checks, house pattern for this router).
|
||||||
|
- Coverage: **>90%** on `app/` (every new/changed branch — both field-present/absent combos — is exercised — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET` reports the flag for DB rows and `false` for env rows; `POST` create round-trips it (absent → `false`)
|
||||||
|
- [ ] `GitSourcePatchIn` exists with two optional fields; `GitSourceIgnoreIn` is gone repo-wide (grep-verified)
|
||||||
|
- [ ] PATCH: bool-only, list-only, both, neither (no-op 200); the phase-89 A4 fixed-detail 422s unchanged for present lists; a 422 never half-applies the other field; 404 + anonymous 403 hold
|
||||||
|
- [ ] `uv run pytest tests/integration/test_git_sources_api.py -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Task 04 — Pipelines: the Sync button + the CLI honor the per-row flag
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Build the `include_hidden_by_root` map at the two live import entry points — the in-app Sync (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py::_resolve_sources`) — with the SAME `str(root)` keying and collision rule as the phase-89 `ignore_by_root`, and pass it to `import_sources`. The upload run needs nothing (phase 90 — no scan) and `scripts/load_test_kb.py` keeps its defaults.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/sync.py` — `_run_sync` (the per-row loop L233-252 + the `import_sources` call L262-264):
|
||||||
|
- After `ignore_by_root: dict[str, list[str]] = {}` (L233): `include_hidden_by_root: dict[str, bool] = {}`
|
||||||
|
- Inside the loop, next to the phase-89 `ignore_by_root.setdefault(...)` (L251-252) — the comment cites phase 105 + the same sibling/repo-name collision note:
|
||||||
|
```python
|
||||||
|
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
|
||||||
|
# the SAME root string the importer sees; a shared-root
|
||||||
|
# collision ORs — if EITHER row says "index hidden", the
|
||||||
|
# root does (the ignore-map union's boolean mirror).
|
||||||
|
include_hidden_by_root[str(root)] = (
|
||||||
|
include_hidden_by_root.get(str(root), False)
|
||||||
|
or bool(row.include_hidden)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
- The call (L262-264): add the kwarg — `summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root, include_hidden_by_root=include_hidden_by_root)`
|
||||||
|
- Module docstring (L30-45, the pipeline list that names "honoring each row's ``ignore_paths`` (phase 89…)"): extend the clause with "and its ``include_hidden`` flag (phase 105 — the per-root hidden-folders map, same per-row construction)".
|
||||||
|
2. `scripts/import_docs.py`:
|
||||||
|
- `_resolve_sources` docstring (L181-185): "Returns ``(sources, ignore_by_root)``" → "Returns ``(sources, ignore_by_root, include_hidden_by_root)`` (phase 89; phase 105 adds the per-root flag map — the flag is stored per row, manual ``--source`` dirs and the legacy fallback have no rows and import with the empty map: hidden paths skipped, A4)."
|
||||||
|
- All THREE return paths become 3-tuples:
|
||||||
|
- the `cli_sources` early return (~L190): `return [path.expanduser() for path in cli_sources], {}, {}`
|
||||||
|
- the rows branch: after the `ignore_by_root` dict init (L206), add `include_hidden_by_root: dict[str, bool] = {}`; inside the loop next to the phase-89 extend (L224-225), the same OR assignment as sync.py (comment: phase 105); the return (L226): `return sources, ignore_by_root, include_hidden_by_root`
|
||||||
|
- the legacy fallback return (~L227): `return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}`
|
||||||
|
- `main` (L269): `sources, ignore_by_root, include_hidden_by_root = _resolve_sources(args.source, settings)` (the comment above it, L258, extends to name the flag map).
|
||||||
|
- The `import_sources` call (L332-335): add `include_hidden_by_root=include_hidden_by_root,`.
|
||||||
|
- Module docstring: the phase-89 line about the per-root ignore map gains the flag clause.
|
||||||
|
3. Tests:
|
||||||
|
- `tests/integration/test_sync_api.py` — extend with the flag (the suite's local-row + fixture-dir pattern; the fixture dir gains `.hidden/note.md` + a visible file):
|
||||||
|
- Row with `include_hidden=False` (default) + `POST /api/sync` → terminal status `success`, the hidden file has NO `documents` row (A4).
|
||||||
|
- Same row flipped to `True` (task-03 PATCH or direct model set, the suite's idiom) + sync → the hidden file IS indexed (`documents` row present; `detail.files` counts it).
|
||||||
|
- `tests/integration/test_import_docs_git.py` (the CLI DB-row suite) — one test: a local row with a hidden file + `include_hidden=True` → running the CLI main path (the suite's existing invocation harness) indexes the hidden file; `False` → not (the CLI's map is built, not lost — the regression this phase most plausibly breaks is here).
|
||||||
|
4. Run `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration as above is this task's layer; the importer itself is task 02's, the UI task 05's.
|
||||||
|
- Coverage: **>90%** on `app/` (the sync map-building branch is exercised; `scripts/` is outside the `--cov=app` gate — the CLI test is the behavioral pin, house pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `_run_sync` builds `include_hidden_by_root` in the existing per-row loop (OR on collision) and passes it to `import_sources`
|
||||||
|
- [ ] `_resolve_sources` returns the 3-tuple on ALL three return paths (manual, rows, legacy fallback) and `main` passes the map through
|
||||||
|
- [ ] Sync with the flag off indexes no hidden docs; with it on, does — E2E-provable through `detail` + `documents` rows; the CLI behaves the same
|
||||||
|
- [ ] `uv run pytest tests/integration/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
# Task 05 — The per-row "Hidden" toggle on the Sources page (checkbox + tag + §7.4 lifecycle + a11y)
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The owner-visible half of the feature: every stored row on the Sources page gets a labeled **Hidden** checkbox in the actions cell, left of its "Ignore paths" button (DOM order Hidden · Ignore paths · Remove — A5); when on, the source cell shows a "hidden on" text tag; the flip PATCHes the flag with the §7.4 never-stale lifecycle and reverts to the server state on failure.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — inside `#view-git-sources`, directly AFTER the `#git-sources-table-wrap` region closes (~L632), the page-level error line for the toggle (the ignore dialog carries its own error INSIDE the modal; the checkbox lives in the table, so its error lives at page level — reuses the existing `.git-source-error` styling):
|
||||||
|
```html
|
||||||
|
<!-- Phase 105: the per-row "Hidden" toggle's error line — the
|
||||||
|
checkbox is a table-cell control (no dialog of its own), so
|
||||||
|
its failure announces here (role=alert; git-sources.js
|
||||||
|
showHiddenError). Hidden until a PATCH fails. -->
|
||||||
|
<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>
|
||||||
|
```
|
||||||
|
2. `frontend/assets/git-sources.js` — `makeRow` (L357+):
|
||||||
|
- **State tag** — in the `urlTd` block, right after the phase-89 `N ignored` count-tag block (~L374-382), the same idiom:
|
||||||
|
```js
|
||||||
|
/* Phase 105 (A5): the "hidden on" state tag — the
|
||||||
|
.git-source-ignore-count idiom (TEXT + background, never
|
||||||
|
color alone — WCAG 1.4.1), so the flag is readable at a
|
||||||
|
glance without hovering the checkbox. */
|
||||||
|
if (s.id && s.include_hidden === true) {
|
||||||
|
const hiddenTag = document.createElement("span");
|
||||||
|
hiddenTag.className = "git-source-hidden-count";
|
||||||
|
hiddenTag.textContent = "hidden on";
|
||||||
|
urlTd.append(hiddenTag);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **The checkbox** — in the `if (s.id) {` actions-cell branch (~L390), BEFORE the `ignoreBtn` construction (~L399), so DOM order is Hidden · Ignore paths · Remove:
|
||||||
|
```js
|
||||||
|
/* Phase 105 (A5): the per-row hidden-folders toggle — a native
|
||||||
|
labeled checkbox (the WCAG focus/label idiom) LEFT of the
|
||||||
|
"Ignore paths" button; Remove stays last. Stored rows only
|
||||||
|
(A3 — env-fallback rows fall through to the "from .env"
|
||||||
|
tag). The aria-label is the ONLY place `value` appears
|
||||||
|
(setAttribute — never innerHTML). Checked state comes from
|
||||||
|
the SERVER row (s.include_hidden), never from a prior local
|
||||||
|
flip (§7.4 — makeRow only ever renders server state). */
|
||||||
|
const hiddenLabel = document.createElement("label");
|
||||||
|
hiddenLabel.className = "git-source-hidden";
|
||||||
|
hiddenLabel.title =
|
||||||
|
"When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.";
|
||||||
|
const hiddenBox = document.createElement("input");
|
||||||
|
hiddenBox.type = "checkbox";
|
||||||
|
hiddenBox.className = "git-source-hidden-box";
|
||||||
|
hiddenBox.checked = s.include_hidden === true;
|
||||||
|
hiddenBox.setAttribute(
|
||||||
|
"aria-label",
|
||||||
|
`Index hidden folders for ${kindLabel} source: ${value}`,
|
||||||
|
);
|
||||||
|
hiddenLabel.append(hiddenBox, document.createTextNode("Hidden"));
|
||||||
|
hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox));
|
||||||
|
actTd.appendChild(hiddenLabel);
|
||||||
|
```
|
||||||
|
- **`toggleHidden(s, box)`** — new function next to `saveIgnorePaths` (the phase-89 §7.4 section, ~L593+), same announce/reload discipline:
|
||||||
|
```js
|
||||||
|
/* Phase 105: the hidden-folders toggle — PATCH { include_hidden }
|
||||||
|
only (the row's list is untouched — the PATCH body's optional
|
||||||
|
fields, task 03). §7.4 never-stale: the box disables at once
|
||||||
|
(no double-flip while the PATCH is out); on 200 the row
|
||||||
|
re-renders from the server (loadSources) and the
|
||||||
|
CONFIRMATION is the LAST announcement (the reload's "N
|
||||||
|
sources listed." lands first — the phase-89 order); on
|
||||||
|
failure the box REVERTS to the server state and the detail
|
||||||
|
lands in #git-sources-hidden-error (role=alert). */
|
||||||
|
function toggleHidden(s, box) {
|
||||||
|
const value = s.kind === "local" ? (s.path ?? s.url) : s.url;
|
||||||
|
const wanted = box.checked;
|
||||||
|
box.disabled = true; // a PATCH is out — the box must not flip twice
|
||||||
|
fetch(`/api/git-sources/${s.id}`, {
|
||||||
|
method: "PATCH",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
credentials: "same-origin",
|
||||||
|
body: JSON.stringify({ include_hidden: wanted }),
|
||||||
|
})
|
||||||
|
.then(async (r) => {
|
||||||
|
if (r.ok) {
|
||||||
|
hideHiddenError();
|
||||||
|
await loadSources();
|
||||||
|
announce(`Hidden folders ${wanted ? "enabled" : "disabled"} for ${value}.`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const detail = await apiDetail(
|
||||||
|
r, `Could not update the hidden-folders setting (${r.status}).`,
|
||||||
|
);
|
||||||
|
showHiddenError(detail);
|
||||||
|
box.checked = s.include_hidden === true; // revert to server state
|
||||||
|
box.disabled = false;
|
||||||
|
})
|
||||||
|
.catch(() => {
|
||||||
|
showHiddenError("Could not reach the server — the setting was not changed.");
|
||||||
|
box.checked = s.include_hidden === true;
|
||||||
|
box.disabled = false;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function showHiddenError(message) {
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.textContent = message;
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.hidden = false;
|
||||||
|
}
|
||||||
|
function hideHiddenError() {
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.textContent = "";
|
||||||
|
if (hiddenErrorEl) hiddenErrorEl.hidden = true;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(Also call `hideHiddenError()` at the top of `loadSources`'s success path (~L315, after `hideLoadError()`) so a healed list clears the stale line — the phase-89 "happy path heals the error state" precedent.)
|
||||||
|
- Element grabber next to the other page-local grabs (~L245-255): `const hiddenErrorEl = root.querySelector("#git-sources-hidden-error");`
|
||||||
|
- Module header comment block (L1-177ish, the contract list): add the phase-105 entry — "the per-row Hidden checkbox (makeRow) → PATCH {include_hidden} (task 03's optional field) → loadSources + announce; failure reverts the box + #git-sources-hidden-error (role=alert); env-fallback rows get no checkbox (A3)".
|
||||||
|
3. `frontend/assets/styles.css` — after the phase-89 block (the `.git-source-ignore-count` rule ends ~L2695):
|
||||||
|
```css
|
||||||
|
/* Phase 105 (A5): the per-row "Hidden" checkbox — the actions
|
||||||
|
cell, LEFT of the "Ignore paths" button (JS builds it,
|
||||||
|
git-sources.js makeRow). A native labeled checkbox: the
|
||||||
|
visible "Hidden" text + the checkbox's own aria-label
|
||||||
|
(full source value). ~44px hit height matches the action
|
||||||
|
buttons; :focus-visible via the GLOBAL rule (L146) — no
|
||||||
|
per-control ring needed. */
|
||||||
|
.git-source-hidden {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 0.4rem;
|
||||||
|
height: 44px;
|
||||||
|
padding: 0 0.5rem;
|
||||||
|
font-size: 0.8rem;
|
||||||
|
color: var(--ink);
|
||||||
|
cursor: pointer;
|
||||||
|
user-select: none;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
/* --brand checkbox fill: the native check renders in --bg on
|
||||||
|
--brand (the house 5.2:1 brand pairing); [executor: verify the
|
||||||
|
checked-state contrast of the native widget in both themes +
|
||||||
|
record the ratio here — house style]. */
|
||||||
|
.git-source-hidden input[type="checkbox"] {
|
||||||
|
width: 1.05rem;
|
||||||
|
height: 1.05rem;
|
||||||
|
margin: 0;
|
||||||
|
accent-color: var(--brand);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.git-source-hidden:disabled { opacity: 0.5; cursor: wait; }
|
||||||
|
/* Phase 105 (A5): the "hidden on" state tag — a copy of the
|
||||||
|
.git-source-ignore-count idiom (TEXT + background, never
|
||||||
|
color alone — WCAG 1.4.1: --ink on --bg 16.7:1). */
|
||||||
|
.git-source-hidden-count {
|
||||||
|
display: inline-block;
|
||||||
|
margin-left: 0.55rem;
|
||||||
|
padding: 0.08rem 0.5rem;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 999px;
|
||||||
|
font-size: 0.72rem;
|
||||||
|
font-weight: 600;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--bg);
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
4. `tests/unit/test_hidden_folders_toggle.py` (NEW) — source-level pins (the house pattern: read the asset files as text; mirror `tests/unit/test_source_ignore_paths.py`'s structure if it exists, else the closest JS-pinning suite):
|
||||||
|
- `git-sources.js`: `makeRow` contains the checkbox construction — class `git-source-hidden-box`, `type` checkbox, the aria-label template `` `Index hidden folders for ${kindLabel} source: ${value}` ``; `checked = s.include_hidden === true`; the label is appended to `actTd` BEFORE the ignore button (pin the slice order: the hidden-label block precedes the `ignoreBtn` construction); `toggleHidden` defined with `PATCH` + `body: JSON.stringify({ include_hidden: wanted })`; the failure branch reverts (`box.checked = s.include_hidden === true`) AND re-enables; the success branch announces AFTER `await loadSources()` (pin the order in the source slice).
|
||||||
|
- **Cross-file pin (single source of truth for the field name):** the JS body `include_hidden` key and the Python `GitSourcePatchIn.include_hidden` field (regex-parsed from `app/schemas.py`) are the SAME string — a rename on either side breaks the wire contract and this test.
|
||||||
|
- `index.html`: `#git-sources-hidden-error` exists, carries `role="alert"`, `hidden`, class `git-source-error`, and sits INSIDE `#view-git-sources` after `#git-sources-table-wrap` (source order).
|
||||||
|
- `styles.css`: the `.git-source-hidden`, `.git-source-hidden input[type="checkbox"]`, `.git-source-hidden:disabled`, `.git-source-hidden-count` rules exist; the checkbox rule sets `accent-color`.
|
||||||
|
- The `N ignored`-style tag text pin: `hidden on` literal present in the JS (the tag copy is the state — text, never color alone).
|
||||||
|
5. Run `uv run pytest tests/unit/test_hidden_folders_toggle.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit (source-level pins) is this task's layer; the behavioral E2E is task 06.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` changes in this task — the gate is unaffected; keep it green).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every stored row renders Hidden · Ignore paths · Remove in the actions cell; env-fallback rows render no checkbox
|
||||||
|
- [ ] `checked` comes only from server state; a "hidden on" tag appears in the source cell iff the flag is on
|
||||||
|
- [ ] Flip → box disables → `PATCH {"include_hidden": …}` → 200: error cleared, list reloaded, confirmation announced LAST; failure: box reverts + re-enables, detail in the `role="alert"` line; the line heals on a successful load
|
||||||
|
- [ ] WCAG: visible label + full-value `aria-label` on the checkbox, global `:focus-visible` ring, tag is text (never color alone), checkbox checked-state contrast verified + recorded in the CSS comment
|
||||||
|
- [ ] The JS field-name pin matches `app/schemas.py` (cross-file test); `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Task 06 — E2E: `tests/e2e/test_hidden_folders_toggle.py` (isolation) + regressions + full gate + commit
|
||||||
|
|
||||||
|
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — the whole item, proven end to end.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One dedicated Playwright suite proving the TODO item through the REAL page + REAL API + REAL sync pipeline (mock LLM, no git, no network — a `kind="local"` row over a fixture dir, the `test_source_ignore_paths.py` module pattern verbatim): default-off byte-identity, toggle-on indexes hidden folders, toggle-off prunes them (A2), env rows have no control, a11y + the error line. Then the phase's full gate and the single atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_hidden_folders_toggle.py` (NEW) — copy the module scaffolding of `tests/e2e/test_source_ignore_paths.py` (the module-env `BOR_GIT_SOURCES` URL that is NEVER synced/cloned; `e2e/auth_helpers.login`; `e2e/conftest.py` `ADMIN_PASSWORD`/`SESSION_SECRET`/`USE_REAL_LLM`/`_wait_http`; the sync helper: `POST /api/sync` → poll `GET /api/sync/status` to terminal; the `source_dir` fixture built under `tmp_path_factory`) with the fixture tree:
|
||||||
|
```
|
||||||
|
visible.md
|
||||||
|
.hidden/note.md <- the phase-105 subject
|
||||||
|
.venv/junk.md <- EXCLUDED_DIRS: never indexed, both states (A1)
|
||||||
|
```
|
||||||
|
Contract under test (docstring):
|
||||||
|
- anonymous: the `#git-sources-gate` sign-in gate, the manager hidden, NO `/api/git-sources` call on load, 403 on `GET`/`POST /api/git-sources` AND `PATCH /api/git-sources/{id}` with a bool-only body (the phase-89 anonymous pin extended to the toggle payload);
|
||||||
|
- A4: with the default row, a sync indexes `visible.md` ONLY — `detail.files` counts one, `.hidden/note.md` has no `documents` row, the checkbox renders UNCHECKED and no "hidden on" tag;
|
||||||
|
- A1: flipping the checkbox on (the real click) → the PATCH 200 lands (the "hidden on" tag appears, the announcer `role=status` fires the confirmation AFTER the reload line) → sync → `.hidden/note.md` IS indexed (`documents` row present; the KB catalog lists it — the tree/catalog is DB-driven, no extra surface); `.venv/junk.md` STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders CHECKED (server state);
|
||||||
|
- A2: flipping it OFF (real click) → sync → `detail.pruned` includes the hidden doc, the catalog no longer lists it, the tag is gone;
|
||||||
|
- A3: the env-fallback view (the table-empty state) renders the "from .env" tag with NO Hidden checkbox and NO "Ignore paths" button;
|
||||||
|
- a11y + error surface: the checkbox has a full accessible name containing the source location (`Index hidden folders for local source: …`), is keyboard-focusable (Tab reaches it, the global `:focus-visible` ring applies), the tag text is "hidden on" (never color alone); `#git-sources-hidden-error` exists with `role="alert"` and stays `hidden` through the happy path.
|
||||||
|
Test → contract mapping (one test per bullet, the phase-89 suite's shape):
|
||||||
|
1. `test_anonymous_gate_and_403s`
|
||||||
|
2. `test_hidden_off_by_default`
|
||||||
|
3. `test_toggle_on_indexes_hidden_folders`
|
||||||
|
4. `test_toggle_off_prunes_hidden`
|
||||||
|
5. `test_env_fallback_rows_have_no_toggle`
|
||||||
|
6. `test_toggle_a11y_and_error_surface`
|
||||||
|
2. **Regressions** — each in isolation (DB up), all green:
|
||||||
|
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` (the phase-89 suite — the PATCH body rename + the actions-cell order must not break it; its dialog still sends the list and gets byte-identical replace semantics)
|
||||||
|
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
|
||||||
|
3. **Full gate** (AGENTS.md rule 9 — non-negotiable):
|
||||||
|
- `uv run pytest` (unit + integration) green
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%**
|
||||||
|
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation
|
||||||
|
- `uv run ruff check . && uv run pyright` clean
|
||||||
|
4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block:
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
|
||||||
|
```
|
||||||
|
(`TODO.md` is cleared to the bare `# TODO` by the conversion step BEFORE this commit lands — the items now live in this phase; if the pipeline commits per task instead, fold `TODO.md` into this phase's final commit and move the phase dir to `.agents/phases/complete/105_hidden_folders_toggle/` as the last action, per the pipeline gate.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written).
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/e2e/test_hidden_folders_toggle.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up)
|
||||||
|
- [ ] The five regression E2E suites pass in isolation (the phase-89 suite first — it is the rename's canary)
|
||||||
|
- [ ] The full gate is green: unit+integration, TOTAL coverage >90%, ruff + pyright clean
|
||||||
|
- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0019 + scripts + frontend + tests + the cleared `TODO.md` + the phase files)
|
||||||
|
- [ ] The TODO item is done: the owner can flip "Hidden" per input next to its Ignore paths button, and the next sync indexes (or prunes, when off) the dot-prefixed paths of that source — with `.venv`/`node_modules`/`.git`/… always excluded
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,50 @@
|
|||||||
|
# Task 01 — `documents.created_at` + `documents.created_at_manual` (model + alembic `0020`)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database"; "the timestamp can't be null so just set it to today's date during the migration"; "This timestamp should be editable" (the manual flag is D1's phase-97-precedent half).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Persist the document creation date: one additive, reversible migration adding `created_at` (NOT NULL, server-defaulted to the migration moment — every existing deployment row reads "today") and `created_at_manual` (default false — the owner-correction lock, D1) to `documents`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — the `Document` class (L101-130): add the two columns directly AFTER `indexed_at` (L112), mirroring its docstring/provenance style (`DateTime`/`Boolean`/`func`/`text` are already imported):
|
||||||
|
```python
|
||||||
|
#: The document's CREATION date (phase 106, D1/D2/D3) — sourced at
|
||||||
|
#: sync time (git last-commit date for git sources, file mtime for
|
||||||
|
#: local dirs / unpacked uploads), normalized by
|
||||||
|
#: :func:`app.rag.doc_dates.normalize_doc_date` (undetermined or
|
||||||
|
#: future → today; UTC). NOT NULL: pre-phase-106 rows backfill to
|
||||||
|
#: the migration moment (≈ today — the owner's instruction) and the
|
||||||
|
#: next sync refreshes them (the importer's unchanged path,
|
||||||
|
#: task 04 — a sync may move a date OLDER, D4). Distinct from
|
||||||
|
#: ``indexed_at`` (the INDEX time, untouched).
|
||||||
|
created_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), server_default=func.now(), nullable=False
|
||||||
|
)
|
||||||
|
#: True only while ``created_at`` is the OWNER'S correction (phase
|
||||||
|
#: 106, D1 — the ``folder_summaries.manually_edited`` phase-97
|
||||||
|
#: precedent): set ONLY by ``PATCH /api/documents/date``
|
||||||
|
#: (task 05); the sync-time importer SKIPS the refresh on a manual
|
||||||
|
#: row (the correction survives syncs, D4) and a content change
|
||||||
|
#: RESETS both the date and the flag (a new version = a new date).
|
||||||
|
created_at_manual: Mapped[bool] = mapped_column(
|
||||||
|
Boolean, default=False, server_default=text("false"), nullable=False
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(If the module header's one-line `documents` field inventory names `indexed_at`, add `created_at`/`created_at_manual` (phase 106) to the parenthetical.)
|
||||||
|
2. `alembic/versions/0020_documents_created_at.py` (NEW — the house format of `0019_git_source_include_hidden.py`):
|
||||||
|
- `revision = "0020"`, `down_revision = "0019"`.
|
||||||
|
- `upgrade()`: `op.add_column("documents", sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False))` then `op.add_column("documents", sa.Column("created_at_manual", sa.Boolean(), server_default=sa.text("false"), nullable=False))`.
|
||||||
|
- `downgrade()`: `op.drop_column("documents", "created_at_manual")` then `op.drop_column("documents", "created_at")`.
|
||||||
|
- Module docstring: the phase-106 provenance (what the date is, D1/D2/D3/D4, the NOT-NULL backfill-to-today behavior, one additive reversible migration, A13).
|
||||||
|
3. Tests — `tests/integration/test_migration_0020.py` (NEW), mirroring `tests/integration/test_migration_0019.py` VERBATIM in shape (the real-Alembic `alembic` fixture that starts/ends at head; `information_schema` column-contract assertions; the explicit 0019 → 0020 step so later migrations cannot break the pins): the 0019 `documents` schema (incl. `indexed_at`, `summary`) survives the upgrade; both new columns exist with the full contract — `timestamp with time zone` NOT NULL default `now()` / `boolean` NOT NULL default `false`; a `documents` row inserted while the DB is at `0019` backfills `created_at ≈ now()` (assert within a few seconds of the upgrade moment) and `created_at_manual is False`; downgrade to `0019` → both columns GONE (A13) while the row + its content survive; upgrade back to `0020` → both columns back (round-trip); the ORM contract agrees — a freshly inserted `Document` (nothing passed) reads `created_at_manual is False` + non-null `created_at`, and an explicit `created_at` + `created_at_manual=True` round-trips through a fresh session.
|
||||||
|
4. Run `uv run pytest tests/integration/test_migration_0020.py -q` (DB up) + `uv run alembic upgrade head` on the dev/test DB — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the migration upgrade/downgrade + server-default pins above ARE this task's layer (no importer behavior yet — task 04 writes these columns).
|
||||||
|
- Coverage: **>90%** on `app/` (model/migration-only change — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `Document.created_at` (NOT NULL, `server_default=func.now()`) and `Document.created_at_manual` (NOT NULL, `server_default=text("false")`) exist with the D1/D2/D3/D4 provenance comments
|
||||||
|
- [ ] `alembic/versions/0020_documents_created_at.py` upgrades from `0019` and downgrades cleanly; the dev/test DB is at head; existing rows read `created_at ≈ now()` (the backfill) and `created_at_manual is False`
|
||||||
|
- [ ] Fresh-row-defaults + explicit-values round-trip tests pass; existing suites stay green
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# Task 02 — `app/rag/doc_dates.py`: the date normalization choke point (D3)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "If the date of a document can't be determined or is in the future then assume that document was created today."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One pure, stdlib-only module that every document date passes through — the importer (task 04) and the date-edit API (task 05) both call it, so the today/future/naive rules live in exactly one place and can be pinned by unit tests without a database.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/doc_dates.py` (NEW):
|
||||||
|
```python
|
||||||
|
"""Document-creation-date sourcing + normalization (phase 106, D2/D3).
|
||||||
|
|
||||||
|
Every document date the importer writes and every date the owner
|
||||||
|
edits passes through :func:`normalize_doc_date` — the single choke
|
||||||
|
point for the owner's rules: an UNDETERMINED date (no source signal)
|
||||||
|
and a FUTURE date (beyond a small clock-skew tolerance) both assume
|
||||||
|
the document was created TODAY (UTC). Naive source timestamps (zip
|
||||||
|
DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch-
|
||||||
|
based values rendered as UTC; aware ones are converted to UTC.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
#: Clock-skew tolerance (D3): a source date up to this far in the
|
||||||
|
#: FUTURE is a drifting clock, not a future document — it keeps its
|
||||||
|
#: date. Beyond it, the owner's rule applies (→ today).
|
||||||
|
FUTURE_SKEW_TOLERANCE = timedelta(days=1)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime:
|
||||||
|
"""*raw* → the stored UTC creation date (the D3 rule, pinned).
|
||||||
|
|
||||||
|
``now`` is injectable (tests); it defaults to
|
||||||
|
``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*;
|
||||||
|
naive *raw* → treated as UTC; aware *raw* → converted to UTC;
|
||||||
|
*raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*.
|
||||||
|
The result always carries full precision (no date-truncation —
|
||||||
|
the display formats, the storage doesn't).
|
||||||
|
"""
|
||||||
|
```
|
||||||
|
Plus:
|
||||||
|
```python
|
||||||
|
def file_mtime_datetime(path: Path) -> datetime:
|
||||||
|
"""The file's mtime as an aware UTC datetime (the D2 fallback).
|
||||||
|
|
||||||
|
Epoch mtimes are tz-agnostic — UTC is the correct rendering
|
||||||
|
(zip DOS timestamps and tar mtimes pass through the same
|
||||||
|
:func:`normalize_doc_date` after unpacking, task 03).
|
||||||
|
"""
|
||||||
|
return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)
|
||||||
|
```
|
||||||
|
Implementation notes: for the naive case, attach UTC (`raw.replace(tzinfo=UTC)`) rather than assuming local time (the homelab host TZ is irrelevant — source mtimes are epoch values); for the aware case, `raw.astimezone(UTC)`; compare the future check in aware space.
|
||||||
|
2. `tests/unit/test_doc_dates.py` (NEW) — the boundary matrix (pure function, no DB):
|
||||||
|
- `None` → exactly `now` (inject a fixed `now`);
|
||||||
|
- naive `2020-05-01T12:00` → `2020-05-01T12:00+00:00` (UTC-attached, not local-converted);
|
||||||
|
- aware `2020-05-01T08:00-04:00` → `2020-05-01T12:00+00:00` (converted);
|
||||||
|
- future by 23 h (just INSIDE the tolerance) → keeps its date;
|
||||||
|
- future by 25 h (beyond) → `now`;
|
||||||
|
- exactly `now + FUTURE_SKEW_TOLERANCE` → keeps its date (the boundary is strict-greater);
|
||||||
|
- `file_mtime_datetime` on a tmp file with a `os.utime`'d mtime → the expected UTC datetime (±1 s tolerance for mtime granularity);
|
||||||
|
- the module imports nothing but stdlib (a source-level pin, the house pattern — grep the file for `import` lines).
|
||||||
|
3. Run `uv run pytest tests/unit/test_doc_dates.py -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the matrix above IS this task's layer (the callers land in tasks 04/05).
|
||||||
|
- Coverage: **>90%** on `app/` (new module fully covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `app/rag/doc_dates.py` exists with `FUTURE_SKEW_TOLERANCE` (1 day), `normalize_doc_date` (None→today, naive→UTC, aware→convert, future-beyond-tolerance→today, full precision kept) and `file_mtime_datetime`
|
||||||
|
- [ ] `tests/unit/test_doc_dates.py` pins the full boundary matrix (incl. the strict-greater 1-day boundary and the naive-is-UTC rule) and passes
|
||||||
|
- [ ] No non-stdlib imports in the module; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 03 — Source date extraction: mtime-preserving unpack + `file_commit_dates` (D2/D10)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — dates come "via their git timestamp or via file metadata (hopefully) preserved in the tar or zip archive process."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the source-of-truth date actually EXIST at the filesystem/checkout level: the zip/tar unpacker restores member mtimes (uploads stop losing their dates), and `scripts/git_sync.py` gains a single-call per-file last-commit-date walk (the A11 git site) with the verified shallow-vs-local behavior pinned.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/archive_upload.py` — mtime preservation (D2), regular files only, every safety check + the zip-bomb cap UNCHANGED:
|
||||||
|
- `_unpack_zip` (L214-238): after the `with zf.open(member) as src: _write_capped(...)` block for a regular file, restore the member's DOS mtime:
|
||||||
|
```python
|
||||||
|
mtime = datetime(*member.date_time, tzinfo=timezone.utc).timestamp()
|
||||||
|
os.utime(dest, ns=(mtime, mtime))
|
||||||
|
```
|
||||||
|
(`datetime`/`timezone` from the stdlib — add the import; DOS `date_time` is a tz-agnostic epoch value, UTC-rendered exactly like an mtime — the task-02 convention.)
|
||||||
|
- `_unpack_tar` (L241-270): in the `member.isreg()` branch, after `_write_capped(...)`: `os.utime(dest, ns=(member.mtime, member.mtime))` (tar `mtime` is epoch seconds — `ns=` accepts a float seconds value).
|
||||||
|
- Directories, symlinks, and hardlinks are untouched (only regular files are ever indexed). A failed unpack still removes the partial tree (the `utime` calls sit inside the existing try/except flow — an `OSError` there is caught by `unpack_archive`'s handler exactly like any other write failure).
|
||||||
|
- Update the module docstring's guarantees list with the mtime-preservation line (phase 106, D2).
|
||||||
|
2. `scripts/git_sync.py` — `file_commit_dates(dest: Path) -> dict[str, datetime]` (NEW public function, exported in `__all__`):
|
||||||
|
- Runs ONE `run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (the A11 single-invocation site — the module docstring's git-inventory sentence gains this command).
|
||||||
|
- Parse: lines matching `@@` start a commit (ISO-strict `%cI` → `datetime.fromisoformat`, aware); subsequent non-empty lines until the next `@@`/blank-then-`@@` are repo-relative paths (split on whitespace like git's name-only output, normalize `\` → `/`, lstrip a leading `/`). Per path, the FIRST sighting wins (the walk is newest-first) — that is the file's last-commit date.
|
||||||
|
- **Fail-soft (pinned):** `GitSyncError` (git missing/failed) or ANY parse anomaly → `logger.warning` + return `{}` — the importer (task 04) falls back to file mtimes; a date walk must never break a sync.
|
||||||
|
- Module docstring: what it is, the one-git-call contract, and the VERIFIED checkout behavior (owner-permission source: this phase's ask, 2026-09-13): a local-path checkout cloned by `clone_or_pull` keeps FULL history (`--depth` is ignored in local clones — git's own warning) → TRUE per-file dates; a URL-transport checkout is shallow and git reports the TIP commit as every existing file's last commit (the shallow boundary is each file's history root) → a uniform per-repo tip date (D10 — no intra-repo distortion, real cross-repo signal).
|
||||||
|
3. Tests:
|
||||||
|
- `tests/unit/test_archive_upload_dates.py` (NEW): build in `tmp_path` — a zip with one member whose `ZipInfo.date_time` is an old fixed tuple (e.g. `(2020, 1, 2, 3, 4, 6)` → 2020-01-02 03:04:06 UTC) and a tar with one member `mtime=1577934246` (2020-01-02 03:04:06) — `unpack_archive` → the extracted file's `st_mtime` equals the member's (±1 s, mtime granularity). The existing archive-upload suite (`tests/unit/test_archive_upload*.py` — glob to find it) stays green (no safety behavior moved).
|
||||||
|
- `tests/integration/test_git_file_dates.py` (NEW — real `git` in the test environment, the `test_import_docs_git.py` precedent for git availability; skip cleanly if `git` is absent, that suite's pattern): in `tmp_path_factory` build a scratch repo with two files committed at controlled `GIT_COMMITTER_DATE`s (file A 2020-01-02, file B touched again 2024-06-15 — the 2026-09-13 verification recipe): (a) `clone_or_pull` from the LOCAL path → `file_commit_dates` returns A's 2020 date and B's 2024 date (true per-file); (b) a `file://` shallow clone (run `git clone --depth 1 file://…` directly in the test — the test harness, not `clone_or_pull`, makes this one) → EVERY file's date is the TIP commit's (2024-06-15) (D10 pinned); (c) a directory without `.git` / a `git` failure → `{}` (fail-soft, no raise).
|
||||||
|
4. Run `uv run pytest tests/unit/test_archive_upload_dates.py tests/integration/test_git_file_dates.py -q` (DB up for the integration file's `db` fixture only if used — keep it DB-free: `file_commit_dates` takes a path, no session) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the zip/tar mtime pins + the safety-suite regression.
|
||||||
|
- Integration: the git walk against real scratch repos (both checkout kinds + the fail-soft path) — DB-free.
|
||||||
|
- Coverage: **>90%** on `app/` (the unpacker branches + the new parser fully covered — the validate.sh gate; `scripts/` is outside the `app/` coverage denominator but the integration suite pins its behavior).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A zip and a tar with old member timestamps unpack to files carrying those mtimes (regular files only; safety/cap behavior byte-identical — the existing suite green)
|
||||||
|
- [ ] `scripts/git_sync.py::file_commit_dates` exists, is the ONLY new git invocation (through `run_git`), returns `{path: last_commit_datetime}` with first-sighting-wins parsing, and fails soft to `{}`
|
||||||
|
- [ ] The verified behavior is pinned: local clone → true per-file dates; `file://` shallow clone → tip date for every file
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 04 — Importer: source the date on every upsert, refresh on unchanged, protect manual (D4)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database and updated when sources are synced"; "It's totally fine for a sync to cause a document or folder's date to get older."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The importer writes `documents.created_at` from the source (git map → mtime fallback → `normalize_doc_date`) on every add/update, REFRESHES it on the unchanged path (the backfill-correction case — an existing row stamped "today" by the migration gets its real date on the next sync even when the content didn't change), skips the refresh on manually corrected rows, and counts date-only refreshes in a new additive `dates_updated` counter. Both live entry points (Sync button, CLI) feed the git-date map.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py`:
|
||||||
|
- `ImportSummary` (L64-111): add `dates_updated: int = 0` (docstring: files whose `created_at` was refreshed on the UNCHANGED path — content untouched, D4) and the `dates_updated=%d` term in `log()`'s `import: summary …` line (PLAN §9 — append it AFTER `summary_errors`, before `formats`, so existing prefix assertions survive).
|
||||||
|
- `import_sources` (L213-340): new keyword `doc_dates_by_root: dict[str, dict[str, datetime]] | None = None` (AFTER `include_hidden_by_root`), docstring paragraph in the `include_hidden_by_root` style: keyed by `str(root)` — the root string exactly as passed in *sources*; maps a source-relative POSIX path to its RAW source date (git last-commit, task 03); ONLY git roots are listed — unlisted roots (local dirs, uploads) take the mtime fallback; `None` (default) changes nothing for existing callers (the mtime fallback applies — which IS the behavior change: unchanged files now refresh their date, D4). In the processing loop (L281+): `dates_map = (doc_dates_by_root or {}).get(str(root), {})` and pass `raw_date=dates_map.get(rel)` into `_index_file`. The progress pre-walk is UNTOUCHED (dates change no file count).
|
||||||
|
- `_index_file` (L376-471): new keyword `raw_date: datetime | None = None`:
|
||||||
|
- Resolve once, up top: `if raw_date is None: raw_date = file_mtime_datetime(full_path)` (import from `app.rag.doc_dates`).
|
||||||
|
- **added branch** (L409-417): `created_at=normalize_doc_date(raw_date)` on the new `Document(…)`; `created_at_manual` stays the column default (`False`).
|
||||||
|
- **updated branch** (L418-421): `doc.created_at = normalize_doc_date(raw_date)` and `doc.created_at_manual = False` (a content change resets a previous correction — the correction referred to the old content; D4).
|
||||||
|
- **unchanged branch** (L401-404, currently the early return): BEFORE returning — if `doc.created_at_manual` → return unchanged (log the existing line, the correction survives — D1/D4); else `target = normalize_doc_date(raw_date)`; if `target != doc.created_at` → `doc.created_at = target`, `session.commit()`, `summary.dates_updated += 1`, `logger.info("import: date-refreshed source=%s path=%s date=%s", source, rel, doc.created_at.isoformat())`; return. (A date-only refresh is still counted `unchanged` — `added/updated/pruned` are untouched → no `sources_meta` bump, no overview/folder-summary regeneration: the gate keys on content, D4.)
|
||||||
|
- Module docstring: the Scope/workflow paragraph gains the date rule (two sentences — sourced on add/update, refreshed on unchanged unless manual, D2/D4).
|
||||||
|
2. `app/api/sync.py` — `_run_sync` (the per-row loop L233-263): build `doc_dates_by_root: dict[str, dict[str, datetime]] = {}` alongside the other two maps; for `kind=git` rows, AFTER `clone_or_pull` returns: `doc_dates_by_root[str(root)] = file_commit_dates(root)` (import `file_commit_dates` next to the existing `clone_or_pull` import, L100); local rows add nothing (mtime fallback). Pass `doc_dates_by_root=doc_dates_by_root` to `import_sources` (L300-303). The success `detail` dict (L370-382) gains `"dates_updated": summary.dates_updated` (additive key, after `"summary_errors"`). The module docstring's pipeline step 4 gains the third-map clause.
|
||||||
|
3. `scripts/import_docs.py` — `_resolve_sources` (L168-226): build the same map for the git rows it clones (after the `clone_or_pull` call, L220) and return it as a 4th tuple element `(sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root)` — manual `--source` dirs and env-fallback rows contribute nothing (no row, no clone → no map entry → mtime fallback); update the return docstring. `main` unpacks the 4-tuple (the L269-ish unpack) and passes the map to `import_sources` (L331-334). Module docstring updated.
|
||||||
|
4. `scripts/load_test_kb.py` — untouched (the `None` default).
|
||||||
|
5. Tests:
|
||||||
|
- `tests/unit/test_importer_dates.py` (NEW — the `tests/unit/test_importer_include_hidden.py` scaffolding: fake LLM from `tests/fakes.py` + a tmp fixture tree; run against the `db` session the house unit pattern uses for importer tests — read `test_importer_include_hidden.py` first and mirror its session handling):
|
||||||
|
- a file `os.utime`'d to 2020-01-02 imports with `created_at` ≈ that instant (added);
|
||||||
|
- unchanged re-import with the mtime moved to 2021 → `created_at` refreshed, `summary.unchanged == 1` AND `summary.dates_updated == 1` (content counts preserved);
|
||||||
|
- unchanged re-import with the same mtime → `dates_updated == 0`;
|
||||||
|
- a row with `created_at_manual=True` + moved mtime → date UNTOUCHED (the D1 lock) and `dates_updated == 0`;
|
||||||
|
- a content change on a manual row → date reset from source AND `created_at_manual is False`;
|
||||||
|
- `doc_dates_by_root` map entry beats the mtime (the git case: map says 2020, mtime says now → 2020 stored);
|
||||||
|
- a future mtime (2030) → `created_at` folds to today (D3 through the importer).
|
||||||
|
- `tests/integration/test_importer_dates.py` (NEW — real Postgres, the `tests/integration/test_importer_e2e.py` fake-LLM pattern): the backfill-correction case — a row first imported with a "today" mtime, its file then `os.utime`'d back to 2019 (content identical) → the second `import_sources` run stores the 2019 date (`added/updated/pruned` all 0, `dates_updated == 1`) AND `sources_meta`'s version is UNBUMPED (the date-only-refresh gate, D4 — seed the version row first, read it after); a pruned/manual matrix as needed for coverage.
|
||||||
|
- Regression sweep (run, and update ONLY exact-string pins that break — the `import: summary` line gained a term and the sync `detail` gained a key): `uv run pytest tests/unit/test_importer*.py tests/integration/test_importer*.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q`.
|
||||||
|
6. Run the full unit + integration importer slice — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the semantic matrix above (added/updated/unchanged × manual × map vs mtime × future) against the fake LLM.
|
||||||
|
- Integration: real Postgres for the backfill-correction + no-version-bump pins.
|
||||||
|
- Coverage: **>90%** on `app/` (the new branches in `importer.py` + the sync detail all covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `import_sources` accepts `doc_dates_by_root` (str(root)-keyed, git-only, None = byte-identical for existing callers); `_index_file` sources added/updated dates from the map → mtime fallback → `normalize_doc_date` and resets `created_at_manual` on content change
|
||||||
|
- [ ] The unchanged path refreshes `created_at` (date may go OLDER — no monotonic guard), counts it in `dates_updated` (new field + log term), skips manual rows, and NEVER counts toward `added/updated/pruned` (no `sources_meta` bump, no overview/summary regeneration)
|
||||||
|
- [ ] The Sync button and the CLI both feed the map (git rows only, after clone); `scripts/load_test_kb.py` untouched; the success sync `detail` carries `dates_updated`
|
||||||
|
- [ ] The regression slice above is green (exact-string log/detail pins updated in place where they break); `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Task 05 — Date API surface: reads, the admin date edit, and the tree's dates (D7/D8/D9)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors"; the catalog needs the file dates + folder last-updated (the UI in task 08 renders exactly what this task serves).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Serve the date everywhere the UI (task 08) and the viewer need it — `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` (files: `created_at`; folders/sources: the derived subtree-max `updated_at`, D9) — and add the admin-only `PATCH /api/documents/date` (set + revert, the phase-57 gate/idiom, D7).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py`:
|
||||||
|
- `DocSummary` (L228-235): add `created_at: str` (ISO-8601 — the `indexed_at` docstring style: "verbatim from the row").
|
||||||
|
- `DocContent` (L343-358): add `created_at: str` (after `summary`, before `content` — group the metadata).
|
||||||
|
- NEW `DateUpdate` (the `SummaryUpdate` shape, L361-373): `source: str`, `path: str`, `date: str | None` (docstring: an ISO date `YYYY-MM-DD` or full ISO datetime; **null/absent = the CLEAR** — drop the manual flag, the stored date stands until the next sync refresh; a malformed non-null value 422s through Pydantic… correction: `str` passes any string — the handler parses (step 3); the 422 comes from the handler, not the model, so the error detail can name the field).
|
||||||
|
- NEW `DateResult`: `source: str`, `path: str`, `created_at: str`, `created_at_manual: bool` (echoes the stored state — the viewer re-renders from it).
|
||||||
|
- `KbTreeFile` (L241-256): add `created_at: str` (verbatim from the catalogue row — the `indexed_at` field's docstring pattern).
|
||||||
|
- `KbTreeFolder` (L259-290) and `KbTreeSource` (L293-320): add `updated_at: str | None` (docstring: the subtree's MAX document `created_at` — D9, derived, never stored; `null` for a 0-document source, the `summary: str | None` shape).
|
||||||
|
2. `app/api/docs.py`:
|
||||||
|
- `list_indexed_documents` (L80-119): add `Document.created_at` to the select AND the `group_by` (the `indexed_at` twin, L104/L107); `DocSummary(..., created_at=row.created_at.isoformat())`.
|
||||||
|
- `get_document_content` (L121-162): `created_at=doc.created_at.isoformat()` in the `DocContent` (L161 site).
|
||||||
|
- NEW `PATCH /api/documents/date` (route order: next to `update_document_summary`, L164-232 — `require_admin` dependency, the phase-57 gate):
|
||||||
|
```python
|
||||||
|
@router.patch("/documents/date", response_model=DateResult)
|
||||||
|
def update_document_date(payload: DateUpdate, db: Session = Depends(get_db),
|
||||||
|
_admin: None = Depends(require_admin)) -> DateResult:
|
||||||
|
```
|
||||||
|
Logic (DB-only — the `/documents/content` row-lookup rule, no filesystem, no LLM/embedding call — a date is never embedded, the phase-57 no-LLM contrast): look up the row by `(source, path)` → none → 404 `{"detail": "document not found"}` (row-lookup semantics, the traversal-string-is-not-a-row note). `payload.date` truthy → `parsed = datetime.fromisoformat(payload.date)` (a bare `YYYY-MM-DD` and full ISO datetimes both parse; `ValueError` → 422 `{"detail": "date must be an ISO date or datetime (e.g. 2024-06-15)"}`) → `doc.created_at = normalize_doc_date(parsed)` (import from `app.rag.doc_dates` — D3: a manually set FUTURE date also folds to today, consistency with the sourced path) → `doc.created_at_manual = True`. `payload.date` falsy (null/absent — the CLEAR) → `doc.created_at_manual = False` only (the stored date stands; the next sync refreshes it — the API cannot re-read the source, D7). `db.commit()`; return `DateResult` with the stored `created_at.isoformat()` + flag.
|
||||||
|
- The tree (task-05 half of D8/D9): `TreeDocRow` (L296-302) becomes the 6-tuple `(source, path, title, chunks, indexed_at, created_at)` (both ISO strings — the builder stays pure over plain types); `_folder_counts` / `_level_children` / `_source_node` thread a 6th element through their tuple unpacks (the `_`-named slots gain the date) and `_level_children`/`_source_node` compute each folder/source's `updated_at`: the MAX of the direct files' `created_at` and the children's `updated_at` values (ISO-8601 strings compare correctly lexicographically — they're all the same `isoformat()` shape; document that in the builder docstring) — `None` when the node has no documents at all (the 0-document registered source). `KbTreeFolder(…, updated_at=…)` / `KbTreeFile(…, created_at=…)` / `KbTreeSource(…, updated_at=…)` at their construction sites (L342-375, L467-490). `list_kb_tree` (L492-551): add `Document.created_at` to the query's select + group_by (the `indexed_at` twin, L543-546) and the `doc_rows` comprehension. The `build_kb_tree` docstring gains the D9 clause (updated_at = subtree max, derived, None for empty).
|
||||||
|
3. Tests:
|
||||||
|
- `tests/unit/test_kb_tree_builder.py` (extended — the pure builder): file nodes carry `created_at` verbatim; a nested fixture asserts each folder's + the source's `updated_at` = the subtree max (a deeper file's date wins over a shallow sibling's); a 0-document registered source → `updated_at is None` and no children; the ls↔`group_folder_listing` cross-check tests (L210-280) still pass with the extended tuples (task 06 changes the agent side — until then the rows stay 6-tuples on BOTH sides only after task 06; for THIS task the cross-check compares file `(path, title[, chunks, indexed_at])` projections — read the current assertions and keep them green: the tree builder's file tuples are internal to the builder, the cross-check uses the builder's OUTPUT nodes, so it should pass unchanged — verify and pin).
|
||||||
|
- `tests/integration/test_docs_api_dates.py` (NEW — the `tests/integration/test_docs_api.py` scaffolding: real app + `db` fixture, an admin cookie where that suite gets one): seed two documents in a nested folder (distinct `created_at`s via direct row writes):
|
||||||
|
- `GET /api/docs` (admin) reports `created_at` per row (and `indexed_at` unchanged);
|
||||||
|
- `GET /api/documents/content` carries `created_at`;
|
||||||
|
- `GET /api/docs/tree` — the file node's `created_at` verbatim, the parent folder's and the source's `updated_at` = the max, a registered-but-empty source → `updated_at: null`;
|
||||||
|
- the PATCH matrix — set `2020-01-02` → 200 + response echoes the stored ISO + `created_at_manual: true` + a re-GET confirms; set a full ISO datetime → accepted; malformed `"not-a-date"` → 422 (the detail names the field); `date: null` → 200 + `created_at_manual: false` + the stored date UNCHANGED; a future date `"2999-01-01"` → stored `created_at` folds to today (D3); unknown `(source, path)` → 404 `document not found`; anonymous → 403 (the gate).
|
||||||
|
4. Run `uv run pytest tests/unit/test_kb_tree_builder.py tests/integration/test_docs_api_dates.py -q` (DB up) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pure builder's date threading (max computation, None-for-empty, verbatim file dates).
|
||||||
|
- Integration: the full API matrix (reads + PATCH set/malformed/clear/future/404/403) against real Postgres.
|
||||||
|
- Coverage: **>90%** on `app/` (the new route + the builder branches covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` serve `created_at` (files) and `updated_at` (folders/sources — subtree max, `null` when empty, derived in the pure builder, D9)
|
||||||
|
- [ ] `PATCH /api/documents/date` is admin-only, DB-only, no-LLM: set (ISO date or datetime, future folds to today, `created_at_manual=true`), clear (null → flag drops, date stands), 422 malformed, 404 unknown pair, 403 anonymous — the phase-57 split intact (viewer stays user-gated)
|
||||||
|
- [ ] `tests/unit/test_kb_tree_builder.py` + `tests/integration/test_docs_api_dates.py` pass; existing docs-API suites stay green
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Task 06 — LLM surfaces: the date rides every document the model sees (D5)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "After this phase, all documents must include the date when fed to the LLM."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The date appears on all three document surfaces the model reads — the HIGH prompt's `<document>` block (the retrieved top-N), the `read` tool result (agent-fetched), and the `ls` file lines (catalog drill) — with the retriever's raw-SQL detached rows gaining the column, the E2E mock's prompt regex updated in lockstep (house rule), and the existing format pins updated mechanically. No persona/teaching copy changes (phase 03 convention).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/retriever.py` — the column plumbing (the vector path returns ORM rows — the column comes for free; only the two raw-SQL detached-row paths need it):
|
||||||
|
- `_LEXICAL_SQL` (L140-157): add `d.created_at AS created_at,` (after `d.indexed_at`).
|
||||||
|
- `_NAME_HIT_SQL` (L277-300): same column addition.
|
||||||
|
- The two detached `Document(…)` reconstructions (L360-372 in `_name_hit_chunks`, L414-424 in `_lexical_candidates`): pass `created_at=row.created_at`.
|
||||||
|
2. `app/rag/prompts.py` — `build_high_prompt` (L371-376): the block becomes
|
||||||
|
```python
|
||||||
|
blocks = [
|
||||||
|
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
|
||||||
|
f'date="{doc.created_at:%Y-%m-%d}">\n'
|
||||||
|
f"{doc.content}\n"
|
||||||
|
"</document>"
|
||||||
|
for doc in documents
|
||||||
|
]
|
||||||
|
```
|
||||||
|
(the UTC date part; the attribute APPENDED after `title` — the only position, always present since `created_at` is NOT NULL). Update the function's docstring line describing the block's identity attributes. The deflection path (`build_deflect_prompt` — titles only) is untouched, and its byte-identity pins hold (no documents involved).
|
||||||
|
3. `app/rag/agent.py` — two surfaces:
|
||||||
|
- **`read` result** (L1173-1185): the FIRST line stays `Document {doc.source}/{doc.path}:` BYTE-IDENTICAL (the E2E mock's `_READ_RESULT_PREFIX` header contract — `_read_results` strips exactly that header to recover the path); the date is the SECOND line, both the truncated (L1176-1181) and plain (L1183) results:
|
||||||
|
```python
|
||||||
|
f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content[:cap]}\n{TRUNCATION_MARKER}\n…"
|
||||||
|
```
|
||||||
|
/ `f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content}"`.
|
||||||
|
- **`ls` file line** (appended — NEVER inserted before `title`, where the mock's non-greedy `path` capture would swallow it): `_source_document_rows` (L658-669) returns `(path, title, created_iso_date)` triples (add `Document.created_at` to the select, format `%Y-%m-%d` in the comprehension); `group_folder_listing` (L707-780) — `rows: Sequence[tuple[str, str, str]]`, the file output becomes `(source, path, title, date)` 4-tuples (the subfolder tuples + count are untouched); `render_folder_listing` (L856-905) renders `f"source: {source} | path: {path} | title: {title} | date: {date}"`; the `ls_folder` (L782-796) + `NOT_A_FOLDER` branch (L1121-1130) + `ls_top` source lines are UNCHANGED in shape (source/folder lines carry no date — only FILE lines are documents). Docstrings updated (the `LS_MAX_FILE_LINES` comment's line-format phrase, the module header's L73 format line).
|
||||||
|
4. `app/api/docs.py` — the ls↔tree cross-check (D9/phase-97 invariant): the docstrings at L326/L376/L461 name the compared shapes — update them to the extended file tuples; `build_kb_tree`'s OUTPUT nodes already carry `created_at` (task 05), so the cross-check test's node-side comparisons gain the date field (step 6).
|
||||||
|
5. `tests/e2e/mock_llm.py` — `_DOCUMENT_BLOCK_RE` (L785-789): make the date attribute an OPTIONAL group so the mock tolerates pre- and post-phase shapes:
|
||||||
|
```python
|
||||||
|
_DOCUMENT_BLOCK_RE = re.compile(
|
||||||
|
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
|
||||||
|
r'title="[^"]*"(\sdate="[^"]*")?>\n(?P<content>.*?)\n</document>',
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
`title=` docstring comment (L780-784) updated. `_CATALOG_LINE_RE` (L873-875) and `_READ_RESULT_PREFIX` (L867) are UNCHANGED by design (verified: the appended ` | date: …` lands in the greedy `title: .+$` tail; the read first line is byte-identical).
|
||||||
|
6. Tests + pin updates:
|
||||||
|
- `tests/unit/test_prompts_dates.py` (NEW): the HIGH block renders `<document source="S" path="P" title="T" date="YYYY-MM-DD">` with the date = the row's UTC date part (inject a fixed `created_at`); the deflection prompt is byte-identical to the pre-phase text for the same inputs (the A8 byte-identity contract holds); the `read` result — both shapes — has the identical first line and the `date:` second line (truncated variant: marker + notice still follow); the `ls` line ends with ` | date: YYYY-MM-DD` and the 50-line cap note is unchanged.
|
||||||
|
- `tests/integration/test_agent_tools_dates.py` (NEW — the `tests/integration/test_agent_tools.py` scaffolding): real rows with distinct `created_at`s — execute a `read` tool call → result second line = the stored date, first line unchanged; an `ls` drill → every file line carries its date in the appended field.
|
||||||
|
- **Existing-pin sweep (mechanical, test files only)** — run and update exact-string pins that break: `uv run pytest tests/unit/test_retriever.py tests/unit/test_agent.py tests/unit/test_kb_tree_builder.py tests/integration/test_agent_tools.py tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the detached-`Document` constructors in test fixtures that set fields explicitly may need `created_at` where the SQL now returns it — the model default covers ORM inserts; raw-SQL projections are app-side, so fixture rows created via the ORM already have the column).
|
||||||
|
7. Run the sweep + new suites — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: prompt block / read line / ls line format pins (the byte-identity contracts).
|
||||||
|
- Integration: the tool surfaces against real rows.
|
||||||
|
- Coverage: **>90%** on `app/` (the new SQL columns + render branches covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The HIGH prompt's `<document>` block carries `date="YYYY-MM-DD"` (after `title`, always present); the deflection prompt stays byte-identical (A8)
|
||||||
|
- [ ] The `read` result carries `date: YYYY-MM-DD` as its second line (first line byte-identical — the mock header contract); the `ls` FILE line ends with ` | date: YYYY-MM-DD` (source/folder lines unchanged); the ls↔tree cross-check still holds
|
||||||
|
- [ ] `_DOCUMENT_BLOCK_RE` is date-tolerant (optional group); `_CATALOG_LINE_RE`/`_READ_RESULT_PREFIX` untouched; the mock serves post-phase prompts correctly (a quick smoke: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` green in isolation, DB up)
|
||||||
|
- [ ] The existing-pin sweep is green (test-file-only updates); `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# Task 07 — Recency boost: newer documents rank higher, without breaking retrieval (D6)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process (so make sure to test with documents that have the correct answer but are older against documents that are similar and newer but don't quit correctly answer the question). This will be a fine line to walk, so testing is crucial here."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A small, env-tunable, kill-switchable additive recency term on the RRF-fused score — applied once in `retrieve()` after `fuse()` — so a fresh document gets a bounded head start on near-ties while an older document that ACTUALLY answers the question keeps its rank. The owner's scenario is pinned by a permanent real-Postgres battery with deterministic axis vectors. The A7 math, the A8 cosine gate, `query_log.top_score`, and the never-truncated contract are untouched.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — the hybrid block (L163-174, after `rrf_k`):
|
||||||
|
```python
|
||||||
|
#: Recency boost on the RRF-fused retrieval score (phase 106, D6): the
|
||||||
|
#: MAXIMUM additive score a zero-age document gets —
|
||||||
|
#: ``fused + recency_boost * exp(-age_days / recency_half_life_days)``
|
||||||
|
#: (``app.rag.retriever.apply_recency_boost``, applied in
|
||||||
|
#: ``retrieve()`` after ``fuse()``). ``0`` = off — the pre-phase
|
||||||
|
#: ranking is byte-identical (the kill switch); negative values fail
|
||||||
|
#: startup loudly (the ``agent_max_rounds`` validator pattern).
|
||||||
|
#: 0.001 ≈ a 1-2 rank head start on a 60+ RRF scale — enough to break
|
||||||
|
#: near-ties toward the newer document, far below the gap between a
|
||||||
|
#: document that answers and one that merely resembles (the
|
||||||
|
#: phase-106 fine-line battery pins it).
|
||||||
|
recency_boost: float = 0.001
|
||||||
|
#: Age (days) at which the recency boost halves (phase 106, D6).
|
||||||
|
#: ``<= 0`` fails startup loudly (same validator family).
|
||||||
|
recency_half_life_days: int = 365
|
||||||
|
```
|
||||||
|
Add the startup validator (find `agent_max_rounds`'s field-validator and follow it — fail loudly naming the field): `recency_boost < 0` → error; `recency_half_life_days <= 0` → error. `.env.example` — document `BOR_RECENCY_BOOST` + `BOR_RECENCY_HALF_LIFE_DAYS` (the hybrid section, the existing comment style).
|
||||||
|
2. `app/rag/retriever.py`:
|
||||||
|
- NEW pure function (module-level, next to `fuse`):
|
||||||
|
```python
|
||||||
|
def apply_recency_boost(
|
||||||
|
chunks: Sequence[RetrievedChunk],
|
||||||
|
*,
|
||||||
|
now: datetime | None = None,
|
||||||
|
weight: float | None = None,
|
||||||
|
half_life_days: int | None = None,
|
||||||
|
) -> list[RetrievedChunk]:
|
||||||
|
```
|
||||||
|
Defaults from `get_settings()` when omitted; `now` defaults to `datetime.now(UTC)`. For each chunk: `age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0)` (a future `created_at` clamps to 0 — consistent with D3's today-folding), `score = score + weight * math.exp(−age_days / half_life_days)` (import `math`; `replace(rc, score=new_score)` — never mutate inputs, the `fuse` convention). Return the list re-sorted with the EXISTING deterministic key `(−score, −cosine, document.path, position)` — with `weight=0` every score is untouched and the order is byte-identical (pinned). Docstring: the D6 contract, the magnitude rationale (0.001 ≈ 1-2 ranks on the k=60 scale — rank 1 vs 2 in one list differs by ~0.00026, rank 1 vs 10 by ~0.0021), the untouched surfaces (A8 gate = cosine, `query_log.top_score` = cosine, `weak_hit_titles` = titles only, the never-truncated top-N), and the single-apply-site rule (`retrieve()` only — chat API + `eval_retrieval` inherit it).
|
||||||
|
- `retrieve()` (L398-425): after `return fuse(vector, lexical, settings.rrf_k)` → apply: `fused = fuse(...)`; `if settings.recency_boost > 0: return apply_recency_boost(fused)`; `return fused` (weight-0 callers pay nothing).
|
||||||
|
3. `scripts/eval_retrieval.py` — the printed top-N table gains two columns: the document's `created_at` (UTC date) and the post-boost effective score (the script calls `retrieve()`, which now applies the boost — print both the raw fused and effective where they differ, or just effective + date; keep the verdict column). Docstring line updated.
|
||||||
|
4. Tests:
|
||||||
|
- `tests/unit/test_retriever_recency.py` (NEW — fake rows, no DB): age 0 → `+weight` exact; age = half-life → `+weight*exp(-1)` (±1e-9); age 10× half-life → ~`+weight*exp(-10)` (assert `< weight * 1e-3`); future date → full weight (the clamp); `weight=0` → the returned list's `(score, order)` is byte-identical to the input (the kill-switch pin); a tie on raw score breaks toward the newer document; the sort key's `(path, position)` tie-break still applies when scores AND cosines are equal (two docs, same age).
|
||||||
|
- `tests/integration/test_recency_boost.py` (NEW — real Postgres, `tests/integration/test_name_hit_lexical.py`'s axis-vector idiom VERBATIM: `D=768` unit vectors, exact cosines, `TRUNCATE chunks, documents` fixture, `retrieve()` + `select_documents()` with settings overrides via the house settings-override pattern — check how that suite's siblings inject settings, e.g. `monkeypatch` on `get_settings` or `Settings(_env_file=None, …)`):
|
||||||
|
1. **THE OWNER SCENARIO (old-correct beats new-similar).** Question `"How did I configure the backup retention policy?"`. Doc A `backups/retention.md`, `created_at=2020-01-01`: the exact answer — chunk vector = the question vector's axis (cosine 1.0) + its exact tokens in the chunk text (top FTS rank). Doc B `backups/retention-draft.md`, `created_at=yesterday` (the test computes `now − 1d`): topically similar (shares `backup retention policy` tokens — a solid FTS hit at rank 2-3) but a weaker vector (half-parallel axis → cosine ~0.707) and its text says the policy is "under review, no decision yet" (no answer). Assert with DEFAULTS: `select_documents(...)[0].path == "backups/retention.md"` AND the fused (pre-boost, computed via `fuse` directly in the test for the margin) gap A−B ≥ 3× the zero-age boost (record the measured margin in the test docstring — the "comfortable margin" requirement). Assert AGAIN with `recency_boost=0` (settings override): A still first (no-regression pin — relevance alone ordered them).
|
||||||
|
2. **The boost is real (near-tie flips toward newer).** Docs C (2019) and D (yesterday) with IDENTICAL chunk text + IDENTICAL vectors (a true tie — same fused score, cosine, FTS rank; the deterministic sort key would otherwise order by path, and path is set so the OLDER sorts first lexicographically, e.g. `c-older.md` < `d-newer.md`). With defaults: D (newer) is first. With `weight=0`: C (older) is first (proving the boost — not drift — is the differentiator).
|
||||||
|
3. **Decay end-to-end:** the same C/D pair with D aged to `half_life + 365` days (≈ `weight*e^{-3}` ≈ 0.00005, below the tie gap 0) → C first again (the boost faded — recency is an age signal, not a binary).
|
||||||
|
4. **The gate is untouched:** the owner-scenario question's `max cosine` (the A8 input) equals the pre-boost run's (assert on the retrieved chunks' `cosine` values — the boost never touches them).
|
||||||
|
- If test 1's measured margin under the DEFAULTS is thin (< 3× the boost) or the scenario flips, tune the DEFAULTS (0.001/365 are the starting point — the owner re-tunes live via the env) until old-correct wins comfortably, and record the final margin in the docstring. The test asserts the SEMANTICS (A first, margin ≥ 3× boost), never the exact floats.
|
||||||
|
5. Run `uv run pytest tests/unit/test_retriever_recency.py tests/integration/test_recency_boost.py -q` (DB up) — green; then `uv run pytest tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the retriever's existing contract suites stay green — the boost is ON by default in them, so any drift surfaces here).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the decay/weight/clamp/tie/kill-switch pins (pure function).
|
||||||
|
- Integration: the fine-line battery on real Postgres with exact axis cosines — the owner's scenario + the near-tie flip + the decay + the cosine-gate-untouched pin.
|
||||||
|
- Coverage: **>90%** on `app/` (config validator + retriever branches covered — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `Settings.recency_boost` (default 0.001, 0 = byte-identical off, negative fails startup) + `recency_half_life_days` (default 365, `<= 0` fails startup); `.env.example` documents both
|
||||||
|
- [ ] `apply_recency_boost` is pure (defaults from settings, `now` injectable, inputs unmutated, the existing 4-key sort) and is applied in `retrieve()` after `fuse()` and ONLY there — chat API + `eval_retrieval` inherit it; `eval_retrieval` prints the date + effective score
|
||||||
|
- [ ] The owner's scenario is pinned: older-correct beats newer-similar under defaults (margin ≥ 3× the zero-age boost, recorded) AND with the boost off; the near-tie flips toward the newer with the boost on and back without; the decay pin holds; the A8 cosine input is untouched
|
||||||
|
- [ ] `tests/unit/test_retriever_recency.py` + `tests/integration/test_recency_boost.py` + the two existing retriever-contract suites green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Task 08 — UI: `Created` file column, `Updated` folder column, viewer `Created` badge (D8)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "The UI must also show a date for every document at the top of that document when the user clicks it"; "I would also like to see a last updated dates/timestamps on folders before the description column but after the documents column in the UI"; "For files, include a date/timestamp before the 'indexed' column in the UI."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Render the dates the task-05 APIs serve: the file table's `Created` column (before `Indexed`), the folder/source table's `Updated` column (after `Documents`, before `Description`), and the clicked document's `Created` badge in the shared viewer core's top meta row (modal + full page). The date EDITOR is task 09 — this task ships display only.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — the two header rows (RAG view):
|
||||||
|
- File table (L485-492): insert `<th scope="col">Created</th>` between `<th scope="col">Chunks</th>` and `<th scope="col">Indexed</th>`.
|
||||||
|
- Folder table (L470-478): insert `<th scope="col">Updated</th>` between `<th scope="col">Documents</th>` and `<th scope="col">Description</th>`.
|
||||||
|
(No other shell markup — the rows are built by JS; a brief phase-106 comment above each inserted `<th>` in the house style.)
|
||||||
|
2. `frontend/assets/sources.js`:
|
||||||
|
- `makeRow` (L1375-1410): the cell loop (L1401) becomes `for (const value of [d.title, String(d.chunks), fmtDate(d.created_at), fmtDate(d.indexed_at)])` — the `Created` cell lands BEFORE `Indexed` (D8 verbatim). The loop's plain-`td` shape can't carry per-cell titles, so the date cells get one refinement: build the `Created` cell explicitly (a `td` with `textContent = fmtDate(d.created_at)` AND `title = d.created_at` — the ISO hover/precision value, the path-cell `title` idiom) between the `chunks` and `Indexed` cells (the E2E asserts on the locale-stable `title`, not on `toLocaleString` output). The row object fed from tree file nodes (L1341-1350) gains `created_at: f.created_at` (task 05's tree shape — the flat `GET /api/docs` path, if `makeRow` is still fed from it anywhere, carries `created_at` too — grep `makeRow(` call sites and extend every one).
|
||||||
|
- `makeSourceRow` (L1217-1235) + `makeFolderRow` (L1237-1261): between the count `td` and the description cell, one new `td` — `const updatedTd = document.createElement("td"); updatedTd.textContent = s.updated_at ? fmtDate(s.updated_at) : "–";` (the `statLast` null idiom, L1294 — `None` for a 0-document source, D9). `title` attribute = the ISO value (hover precision on the ellipsized cell, the `makeRow` path-cell idiom).
|
||||||
|
- `renderLevel`/`treeStats` — UNCHANGED (the stat cards keep their `indexed_at` "last indexed" semantics — the owner asked for the column, not the cards).
|
||||||
|
3. `frontend/assets/document.js` — `renderDocument` (L118-176, the ONE shared core — the modal AND `/document.html` render through it): the `.doc-meta` badge row (L123-129) gains the badge BEFORE the `Indexed` one:
|
||||||
|
```js
|
||||||
|
metaBadge("doc-created", `Created ${fmtDate(doc.created_at)}`),
|
||||||
|
metaBadge("doc-indexed", `Indexed ${fmtDate(doc.indexed_at)}`),
|
||||||
|
```
|
||||||
|
(the date at the top of a clicked document, D8). `doc-created` is the NEW class — the badge's `title` attribute carries the full ISO timestamp (the `titleEl` ellipsis-precision idiom, L122-124). `document-modal.js` needs no change (it calls the shared core with its own `metaEl` — the module docstring's contract is unchanged; verify the modal's meta element exists — it does: `metaEl` L48).
|
||||||
|
4. `frontend/assets/styles.css` — next to the existing `.doc-indexed` rule (grep for it): `.doc-created` — same badge family (the `doc-indexed` rule copied, provenance comment citing phase 106 D8); the new table cells need no new CSS beyond what `.docs-table` already styles (verify the column count change doesn't break the table's responsive rules — the `#docs-table`/`.kb-folders-table` grid/width rules: if a rule hard-codes the column count, extend it). WCAG: the date text reuses the table ink (≥4.5:1 by construction — record the verified pair in the comment, house style); the badge contrast mirrors `doc-indexed`'s recorded ratio.
|
||||||
|
5. `tests/unit/test_sources_dates.py` (NEW — the house read-the-assets-as-text pattern, `tests/unit/test_source_ignore_paths.py`'s sibling style):
|
||||||
|
- `frontend/index.html` — both header rows' cell ORDER pinned (the `<th>` sequence strings: `Source | Path | Title | Chunks | Created | Indexed` and `Folder | Documents | Updated | Description`);
|
||||||
|
- `frontend/assets/sources.js` — the `makeRow` value-list order (`created_at` before `indexed_at`), the `updatedTd` null→`"–"` branch present in BOTH row builders, the file-row object carries `created_at`;
|
||||||
|
- `frontend/assets/document.js` — the badge order in the meta row (`doc-created` before `doc-indexed`), the `Created ` label + `fmtDate(doc.created_at)` template, the single-source cross-file check that the `doc-created` class exists in `styles.css`;
|
||||||
|
- `frontend/assets/styles.css` — the `.doc-created` rule present with a provenance comment.
|
||||||
|
6. Run `uv run pytest tests/unit/test_sources_dates.py -q` + the existing sources/JS unit suites — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level wiring pins above (order, null handling, cross-file class check).
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code this task — the gate is the full-suite one, held by the other tasks; the JS pins are the house frontend-test pattern).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The file table shows `Created` between `Chunks` and `Indexed` (formatted like the `Indexed` cell — `fmtDate`); the folder/source table shows `Updated` between `Documents` and `Description` (subtree max from the tree API, `–` when null)
|
||||||
|
- [ ] The clicked document's top meta row carries `Created <date>` BEFORE `Indexed` in BOTH the modal and `/document.html` (one shared core — no per-surface copy)
|
||||||
|
- [ ] `tests/unit/test_sources_dates.py` pins the orders + null branch + cross-file class and passes; existing JS unit suites stay green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 09 — The admin date editor in the viewer (D7, the phase-57 idiom)
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An admin-only inline date editor in the shared viewer core (modal + page, where task 08 put the badge): set a corrected date (→ `PATCH /api/documents/date`, `created_at_manual` locks it against syncs, D1) or revert to sync-managed (the CLEAR — flag drops, the date stands until the next sync refreshes). Non-admins see the byte-identical task-08 badge row — no button, no wiring, no network call (the phase-57 split, owner-locked).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document.js` (extend the task-08 core — the phase-57 `wireSummaryEdit` idiom verbatim in structure; read it first, L180+):
|
||||||
|
- After the `Created` badge (task 08's insertion point), the admin gate: `void docAdminReady().then((admin) => { if (admin) wireDateEdit(metaEl, doc); })` — the module-cached `docAdminReady()` promise (the phase-57/79 single-request-per-page convention — no extra fetch). Anonymous / token holders / a failed whoami: the badge row stays exactly what task 08 built (byte-for-byte).
|
||||||
|
- `wireDateEdit(metaEl, doc)`:
|
||||||
|
- **The button** — a text button `Edit date` (`.doc-date-edit`, the `.kb-summary-edit`/`.doc-summary-edit` button family — reuse the existing edit-button class if its styling fits, else a sibling class in `styles.css` with the provenance comment), inserted after the Created badge, `aria-label` = `Edit creation date: ${doc.source}/${doc.path}` (setAttribute — never innerHTML).
|
||||||
|
- **The editor** (opened on click — the badge row swaps in-place, the summary editor's swap pattern): the `Edit date` button is replaced by a container holding a native `<input type="date">` (value = `doc.created_at`'s UTC date part — `new Date(doc.created_at).toISOString().slice(0, 10)`; `aria-label="Document creation date"`) + `Save` / `Cancel` text buttons + a `role="status"` live line (the phase-57 status-line shape). `Save` with an empty input → the clear path (see below) is NOT implicit — an empty `type=date` input is disabled-look only: disable Save when empty (an explicit `Revert` link below handles the clear — no accidental wipes).
|
||||||
|
- **Revert affordance** (the D7 CLEAR, the phase-57 "clear = explicit" contrast): a `Revert to sync` text link/button in the editor container (the muted marker style) → sends `{source, path, date: null}`.
|
||||||
|
- **§7.4 never-stale lifecycle** (the phase-57/89 last-announce order): on Save/Revert — the editor controls disable IMMEDIATELY (no double-submit); `PATCH /api/documents/date` with `{source: doc.source, path: doc.path, date: <input.value>}` (or `date: null` for the revert); on 200 → the badge's text re-renders from the RESPONSE's `created_at` (`Created ${fmtDate(res.created_at)}` — the UI shows exactly what the server stored, never the input's optimistic value), the status line announces `Date saved for <source>/<path>.` / `Reverted to sync-managed date.` (the `role=status` live line + the shared announcer where the page has one — follow whatever `wireSummaryEdit` uses), the editor collapses back to the badge + `Edit date` button; on non-2xx or network failure → the server `detail` (or the canned `Couldn't save the date — try again.` on a plain network error) into a `role="alert"` line (the phase-89 error-line idiom — the nearest existing error surface in this file), the input reverts to the stored date, the controls re-enable — the UI never claims a state the server didn't save.
|
||||||
|
- **No other surface:** the editor lives in `renderDocument`'s shared core only — the modal and the page both get it (both already call the core with `docAdminReady` available — verify `docAdminReady` is reachable in the modal's bundle context; `document-modal.js` imports `renderDocument` from this module, so the wiring rides along with the module — no second copy).
|
||||||
|
2. `frontend/assets/styles.css` — the editor's controls (the `.kb-summary-edit` / summary-editor rule family as the model, near it): `.doc-date-edit` (the button), the date input (sized, the global `:focus-visible` ring applies — no per-control rule, the phase-105 checkbox idiom), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), `role="alert"` line (the `.git-source-error` styling reuse or a local sibling), provenance comments citing phase 106 D7; contrast ≥4.5:1 verified + recorded in comments (house style).
|
||||||
|
3. `tests/unit/test_date_editor.py` (NEW — the read-the-assets-as-text pattern, task 08's suite extended or a sibling):
|
||||||
|
- `frontend/assets/document.js`: `wireDateEdit` exists and is called ONLY behind `docAdminReady()`'s `if (admin)` (a source-level pin — the string sequence `docAdminReady().then` … `wireDateEdit`); the PATCH URL is `/api/documents/date` (the single-source cross-file check — the endpoint string appears exactly once in the JS, matching `app/api/docs.py`'s route); the response-driven badge re-render (the `res.created_at` reference, NOT `input.value`); the revert link sends `date: null`; the disable-on-submit + revert-on-failure branches exist (the error-line `role="alert"` + the re-enable); the aria labels (`Edit creation date: `, `Document creation date`);
|
||||||
|
- `frontend/assets/styles.css`: the editor classes present with provenance comments.
|
||||||
|
4. Run `uv run pytest tests/unit/test_date_editor.py -q` + task 08's suite + the phase-57 suite's unit pins — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the source-level wiring pins above (gate, endpoint, response-driven render, §7.4 branches, a11y strings).
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code this task — the endpoint's coverage landed in task 05; the gate is the full-suite one).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An admin sees an `Edit date` affordance next to the Created badge in BOTH the modal and the page (shared core — one implementation); the editor sets the date (input → `PATCH /api/documents/date` → the badge re-renders from the RESPONSE) and offers `Revert to sync` (→ `date: null`, the manual flag drops)
|
||||||
|
- [ ] The §7.4 lifecycle holds: controls disable on submit, a failure reverts the input to the stored value + announces in a `role="alert"` line + re-enables; the happy path announces through the live line after the badge update
|
||||||
|
- [ ] A non-admin / token holder / failed-whoami viewer is byte-for-byte the task-08 badge row (no button, no wiring, no extra request — the phase-57 split)
|
||||||
|
- [ ] `tests/unit/test_date_editor.py` passes; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# Task 10 — E2E: `tests/e2e/test_document_dates.py` (isolation) + regressions + full gate + commit
|
||||||
|
|
||||||
|
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — the whole item, proven end to end (dates sourced → stored → shown → editable → retrieval-weighted).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One dedicated Playwright suite proving the owner's item through the REAL page + REAL API + REAL importer (mock LLM — deterministic token-overlap embeddings, so the cosine/retrieval behavior is production-shaped; no git, no network — a local fixture dir with `os.utime`'d mtimes, built under `tmp_path_factory`, NEVER the shared `tests/fixtures/docs` whose 13-file counts are pinned by other suites). Then the phase's full gate and the single atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_document_dates.py` (NEW) — module scaffolding from `tests/e2e/test_retrieval_quality.py` (`_import_fixtures`'s Settings-with-mock-port pattern, `_run_in_thread`, `_reset_db`, `e2e.auth_helpers.login`, the `app_url`/`mock_llm`/`db_ready` fixtures, the `source-chip` assertions) with the dedicated fixture tree (the module builds it ONCE per module under `tmp_path_factory`, `os.utime`'d — a `mkdocs + utime` helper at the top):
|
||||||
|
```
|
||||||
|
backups/retention.md utime 2020-01-01 03:04:06Z — THE CORRECT answer:
|
||||||
|
"The backup retention policy is 30 days; snapshots
|
||||||
|
are pruned nightly…" (rich in the question's tokens)
|
||||||
|
backups/retention-draft.md utime = now (default mtime) — the SIMILAR-but-wrong
|
||||||
|
doc: shares "backup retention policy" wording,
|
||||||
|
concludes "under review, no decision yet"
|
||||||
|
legacy/old-doc.md utime 2019-06-15 — single-doc folder (a clean
|
||||||
|
folder-`Updated` max: the 2019 date alone)
|
||||||
|
future/forward.md utime 2999-01-01 — the future-date case (→ today, D3)
|
||||||
|
```
|
||||||
|
Contract under test (docstring) — six tests, one per bullet:
|
||||||
|
1. **`test_dates_landed_on_import`** — import the tree (real importer, mock LLM, in a thread): admin-cookie `GET /api/docs` — `retention.md`'s `created_at` ISO date-part = `2020-01-01`, `old-doc.md`'s = `2019-06-15`, `forward.md`'s = TODAY (the D3 future-fold, the test computes today in UTC); `GET /api/docs/tree` — file nodes carry the same dates; the `legacy` folder node's `updated_at` = the 2019 date (single-doc max), the source node's `updated_at` = the max of all (the `now`/today side); `indexed_at` on every row is UNCHANGED in meaning (still ≈ import time, after the created dates).
|
||||||
|
2. **`test_file_and_folder_columns`** — real form login → the RAG view → the file table header order `… Chunks · Created · Indexed` (the `<th>` sequence) and the drilled-in rows: `retention.md`'s Created cell `title` attribute = the ISO string (locale-stable — task 08's idiom) and its text contains `2020`; the folder table header order `Folder · Documents · Updated · Description`; at the top level the source row's `Updated` cell is non-empty; drilled into `legacy`'s parent, the `legacy` folder row's `Updated` cell `title` carries `2019-06-15`.
|
||||||
|
3. **`test_viewer_shows_date_at_top`** — click `retention.md`'s row link (the real click — the same-page modal, phase 26): the modal's top meta row contains a badge with text starting `Created` whose `title` attribute = the 2020 ISO, and it DOM-precedes the `Indexed` badge (the date at the top of the clicked document, D8); the badge row also still shows `Indexed` + the source/format badges (no regression).
|
||||||
|
4. **`test_old_correct_beats_new_similar`** — THE OWNER SCENARIO end to end: ask `How did I configure the backup retention policy?` → the grounded answer arrives (mock marker, no deflection), the FIRST `.source-chip` = `backups/retention.md` (the OLDER correct doc beats the newer similar one — the real retriever + the default recency boost over the mock's token-overlap embeddings); `query_log` — one row, `deflected is False`, `sources` contains `backups/retention.md`. (If the fixture wording doesn't produce the order under the DEFAULTS — the token-overlap geometry differs from task 07's axis vectors — adjust the FIXTURE TEXT until the old-correct doc is the clear top-1 (more exact question-phrase overlap in `retention.md`, the draft sharing only loose keywords), and record the working wording + the reason in the test docstring. Do NOT change the boost defaults here — task 07 owns them.)
|
||||||
|
5. **`test_date_edit_and_sync_preserves`** — the admin-only edit through the REAL UI: open `legacy/old-doc.md` in the modal → the `Edit date` button is present (admin session) → click → set the date input to `2021-05-05` → Save → the badge re-renders from the response (title = a 2021 ISO) → admin-cookie `GET /api/docs` confirms `2021-05-05`. Re-run the import (in a thread, same tree — the mtimes are untouched): `old-doc.md` keeps `2021-05-05` (the manual flag, D1) while `retention.md` still reads 2020 (refreshed, not stale) and `forward.md` still reads today. Then the REVERT: open the editor again → `Revert to sync` → re-run the import → `old-doc.md`'s date is refreshed back to `2019-06-15` (the flag dropped — sync manages it again).
|
||||||
|
6. **`test_anonymous_gate_and_editor_a11y`** — anonymous (no login): the RAG view shows the sign-in gate (no tables), a raw `PATCH /api/documents/date` with a date payload → 403; signed in (admin): the `Edit date` button's accessible name contains `legacy/old-doc.md` (the aria-label), the editor's date input has the `Document creation date` accessible name, is keyboard-reachable (Tab from the button), the status line is `role="status"` (and the error path's line `role="alert"` exists in the DOM — the phase-57/89 surfaces); the badge text pairs (text + formatting, never color alone — the monochrome-theme contract, B5).
|
||||||
|
2. **Regressions** — each in isolation (DB up), all green (task 06 changed pinned formats — the `ls` line, the `read` result, the `<document>` block; task 04's unchanged-path date refresh must not move any content count):
|
||||||
|
- `uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov` (the fixture-import + ranking E2E — the mock-regex canary)
|
||||||
|
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` (the `<document>` block)
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` (the `ls` line format)
|
||||||
|
- `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` (the `read` result shape)
|
||||||
|
- `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` + `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (the tree shape + the tables)
|
||||||
|
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` + `uv run pytest tests/e2e/test_edit_summaries.py -v --no-cov` (the viewer core + the sibling admin-edit idiom)
|
||||||
|
- `uv run pytest tests/e2e/test_import_documents.py -v --no-cov` + `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` (importer counts + the sync detail)
|
||||||
|
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` (phase 105 — the importer map idiom)
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
|
||||||
|
(Where a suite pins a pre-phase format EXACTLY — an `ls` line without the date field, a `read` result without the `date:` line — update the pin in that test file to the phase-106 shape (mechanical, the new field is deterministic). A suite that breaks for any OTHER reason is a regression — fix the product code in its owning task's files, keep this phase's contract as written.)
|
||||||
|
3. **Full gate** (AGENTS.md rule 9 — non-negotiable):
|
||||||
|
- `uv run pytest` (unit + integration) green
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%**
|
||||||
|
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` green in isolation
|
||||||
|
- `uv run ruff check . && uv run pyright` clean
|
||||||
|
4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block:
|
||||||
|
```bash
|
||||||
|
git add app/ alembic/versions/0020_documents_created_at.py scripts/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(dates): document dates end to end — sourced at sync, shown in UI, editable, recency-weighted in retrieval"
|
||||||
|
```
|
||||||
|
(If the pipeline commits per task instead, fold everything into this phase's final commit and move the phase dir to `.agents/phases/complete/106_document_dates/` as the last action, per the pipeline gate.)
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written).
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/e2e/test_document_dates.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up)
|
||||||
|
- [ ] The owner's scenario holds end to end: the older document that answers the question is the FIRST cited source over the newer similar one (defaults, real retriever, mock embeddings)
|
||||||
|
- [ ] The date edit round-trips through the real UI + API and SURVIVES a re-import (manual flag), and `Revert to sync` hands management back to the next import
|
||||||
|
- [ ] All ten regression suites pass in isolation (the format-pin updates are mechanical and live in the test files)
|
||||||
|
- [ ] The full gate is green: unit + integration, TOTAL coverage >90%, ruff + pyright clean
|
||||||
|
- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0020 + scripts + frontend + tests + `.env.example` + the phase files)
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Phase 107 — True per-file document dates for URL git sources: full-history checkouts (revisits phase 106 D10)
|
||||||
|
|
||||||
|
**Source:** Owner bug report 2026-09-16 (chat): after the phase-106 re-sync of brain.reeseapps.com, the git source `https://gitea.reeseapps.com/services/homelab.git` shows `active/container_bifrost` created **8/16/2026** — "completely wrong, container_bifrost is > 6 months old at this point"; "This is a git source, so it should be easy to tell when a document was last edit[ed]".
|
||||||
|
|
||||||
|
**Story:** n/a (owner bug report, phase-106 follow-up — the phase's E2E suite proves the fix end to end).
|
||||||
|
|
||||||
|
**Context / verified root cause (2026-09-16, scratch + dev checkouts of the live homelab repo):** `clone_or_pull` (`scripts/git_sync.py` L63) clones URL-transport sources with `--depth 1` — phase 28's strategy, which phase 106's **D10 explicitly locked** ("No clone-strategy change… `--depth 1` stays"), predicting the consequence: "URL git sources are shallow → every file carries the repo's TIP-commit date (uniform within the repo…); revisit only if the owner later wants intra-repo recency on URL sources." In a shallow checkout git cannot see history past the shallow boundary (= the tip commit), so `file_commit_dates` (phase 106, `git log --name-only --format=@@%cI` first-sighting-wins) returns the **tip commit's date for EVERY file in the repo** — that uniform tip date is what the live site now displays for `container_bifrost` (the live clone's tip; the dev clone of the same repo shows 2026-09-07 for every file, `git rev-parse --is-shallow-repository` → `true`, exactly ONE `@@` line in the date walk). After `git fetch --unshallow` (438 commits visible) the TRUE last-commit date of `active/container_bifrost/bifrost.md` is **2026-05-05T06:26:40-04:00** — months older than the displayed date. Local-PATH git sources were never affected (git ignores `--depth` for local clones → full history → true dates — which is why the bug only surfaced on the URL source). Everything DOWNSTREAM of the checkout (the importer's `doc_dates_by_root` map + first-sighting-wins walk, D3 normalization, D4 sync semantics, the API/LLM/UI date surfaces, the recency boost) is CORRECT as built — the wrong value is produced at the checkout, so the fix is confined to `clone_or_pull` plus the docstrings/tests that enshrine the shallow assumption.
|
||||||
|
|
||||||
|
**⚠ LOCKED-DECISION REVISIT (AGENTS.md rule 3 — flagged, not silent):** this phase REVISITS phase 106's locked decision **D10** (shallow clone strategy). The revisit is owner-authorized: D10 itself names this exact trigger ("revisit only if the owner later wants intra-repo recency on URL sources") and the owner's 2026-09-16 report is precisely that request. D10's tip-date expectation is hereby SUPERSEDED; every other phase-106 decision (D1–D9 — storage/flag, provenance walk, normalization, sync semantics, LLM surfaces, recency boost, UI columns, derived folder dates) stands unchanged and simply receives true dates. No `PLAN.md` change is involved (D10 is a phase-level decision, not a PLAN §2 anchor).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Git-source checkouts keep FULL history — a fresh `clone_or_pull` clones without `--depth 1`, and any EXISTING shallow checkout (including the live + dev homelab checkouts) self-heals via `git fetch --unshallow` on its next sync — so `file_commit_dates` yields the TRUE per-file last-commit date for every git source (local and URL). A `file://` E2E proves it end to end: an old file shows its old commit date and a tip-touched file shows the tip date, in the API, the Sources tables, and the document viewer — instead of the uniform tip date the bug produced.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `106_document_dates` (complete) — the entire date pipeline this phase corrects AT THE SOURCE: `file_commit_dates` (task 03), the importer's `doc_dates_by_root` map + D4 refresh semantics (task 04), D3 normalization, and the API/LLM/UI surfaces. Only D10's shallow-clone assumption is revised; its suites are the regression gate.
|
||||||
|
- `28_git_based_sources` (complete) — `scripts/git_sync.py` (the ONLY git-invocation site, A11 — `run_git` contract, `GitSyncError` semantics) and `clone_or_pull` itself. `app/core/docs_push.py`'s own `--depth 1/100` fetches operate on the DOCS repo (phase 18/59) and are NOT touched.
|
||||||
|
- `32_admin_sync_button` (complete) — `tests/e2e/test_sync_button.py`'s app-server idiom: per-module env with `BOR_GIT_SOURCES=file://<fixture repo>` + its own `BOR_SOURCES_DIR`, a real fixture repo built via `git` subprocesses with controlled commit dates, the real in-app sync. The new E2E copies it.
|
||||||
|
- `99_kb_tree_table_and_back_nav` (complete) — the Sources RAG-view tables + `GET /api/docs/tree` the E2E asserts on (the `Created`/`Updated` columns phase 106 D8/D9 added).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **The fix — `scripts/git_sync.py::clone_or_pull` (task 01; the single clone/pull entry point BOTH sync entry points call — `app/api/sync.py` L296 and `scripts/import_docs.py` L237 — so one change fixes the UI Sync button and the CLI at once):**
|
||||||
|
1. **Fresh checkout** (dest absent or without `.git`): `run_git(["git", "clone", url, str(dest)], cwd=dest.parent)` — the `--depth 1` flags are REMOVED (full history on the first clone for every transport: https/ssh/`file://`/local-path). One-time cost only — subsequent syncs are incremental; the KB repos are small homelab-docs repos. Deliberately NO env knob to restore shallow: shallow == the bug (D12).
|
||||||
|
2. **Existing checkout** (dest has `.git`): probe first — `run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest)`; stdout.strip() == `"true"` → `run_git(["git", "fetch", "--unshallow"], cwd=dest)` (the ONE-TIME self-heal for checkouts that are already shallow — live + dev homelab included — so the fix reaches deployed sites WITHOUT a re-clone: the next sync after deploy unshallows, pulls, and re-dates), then `run_git(["git", "pull", "--ff-only"], cwd=dest)` as today. Probe `"false"` → straight to `git pull --ff-only` (the common steady-state path — one extra cheap probe per sync, no network fetch).
|
||||||
|
3. **Fail loud (D12):** ANY of the probe/unshallow/pull steps raising `GitSyncError` propagates exactly like today's clone/pull failures — the sync aborts with the named repo + reason (the `app/api/sync.py` 502 surface, the CLI traceback). NEVER a silent fallback: continuing a failed unshallow would silently re-serve tip dates (the bug), and falling back to mtimes would be worse. A broken checkout failing loudly is the phase-28 contract.
|
||||||
|
- Every git invocation still goes through `run_git` (A11 — the module docstring's git-inventory sentence lists the three new/changed commands).
|
||||||
|
- `file_commit_dates` is UNCHANGED in code — on a full-history checkout its existing newest-first, first-sighting-wins walk already returns the true per-file last commit (verified 2026-09-16: 438 commits, `bifrost.md` → 2026-05-05). ONLY its docstring + the module docstring lose the "shallow URL → uniform tip date" narrative and state the new guarantee: every `clone_or_pull` checkout is full-history → TRUE per-file dates for ALL git sources.
|
||||||
|
- **Downstream — deliberately untouched:** the importer's `doc_dates_by_root` plumbing, D1/D3/D4 (storage, normalization, sync-refresh semantics — including "a date may go OLDER", which is exactly how the wrong tip dates self-correct on the first post-fix sync: every git document's stored date is refreshed to its true commit date, `dates_updated` counts them, `sources_meta` does NOT bump — a date-only refresh is `unchanged` per D4), the admin date API, the LLM surfaces, the UI columns, the recency boost. No migration, no data fix — the sync IS the fix. `created_at_manual = true` rows keep the owner's corrections (D1) — correct as designed.
|
||||||
|
- **Stale narratives to correct (same task as the code — a comment that lies is a bug):** `scripts/git_sync.py` module docstring (the L18-28 "Per-file last-commit dates" block: the URL-shallow bullet is false after this phase), `clone_or_pull`'s docstring (L48-51 "shallow, depth 1" + the behavior list), `file_commit_dates`'s docstring (L104-111), `scripts/import_docs.py` docstring L19 ("first run, shallow ``--depth 1``" → "first run, full history"), `app/api/sync.py` per-row comment (L298-303, "shallow URL checkouts → the tip date, D10" → true per-file dates for all git sources).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_full_history_checkouts.md` — `clone_or_pull`: no-`--depth` clone + the shallow-probe/`fetch --unshallow` self-heal + fail-loud; the unit argv pins, the real-git integration pins (true per-file dates over `file://` + the existing-shallow self-heal), and the five stale docstring/comment sites.
|
||||||
|
2. `02_e2e_git_source_dates.md` — dedicated Playwright suite `tests/e2e/test_git_source_dates.py` (isolation): a two-commit `file://` fixture repo, a real in-app sync, true per-file dates asserted in the API + the Sources tables + the viewer badge.
|
||||||
|
3. `03_gates_and_commit.md` — full gate (unit + integration, coverage >90%, the new E2E + the three regression suites in isolation, ruff + pyright), one atomic `--no-gpg-sign` commit, phase dir → `complete/`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_git_sync.py` (task 01): fresh-clone argv re-pinned to `["git", "clone", url, str(dest)]` (NO `--depth`); the existing-checkout path asserts probe-then-pull; NEW: probe `"true"` → `fetch --unshallow` THEN `pull --ff-only` (argv + order); NEW: `fetch --unshallow` failure → `GitSyncError` propagates (D12 fail-loud); the `git clone --depth 1 … failed (exit 128)` match-string updated to the new argv.
|
||||||
|
- Integration — `tests/integration/test_git_file_dates.py` (task 01, real `git`, DB-free, the git-availability skip pattern): the module docstring's D10 tip-date expectation is REPLACED with the full-history guarantee; `test_shallow_file_clone_yields_tip_date_for_every_file` (which pinned THE BUG) is REPLACED by `test_url_clone_yields_true_per_file_dates` — the same two-commit recipe (`a.md`/`docs/deep.md` committed 2020-01-02, `b.md` touched again at the 2024-06-15 tip), but `clone_or_pull(f"file://{scratch_repo}", dest)` (a URL transport, through the real function) must yield `{"a.md": DATE_A, "b.md": DATE_B, "docs/deep.md": DATE_A}` — the regression pin: pre-fix this returned `DATE_B` for all three; NEW `test_existing_shallow_checkout_self_heals` — the harness builds a `--depth 1` `file://` clone directly (simulating the deployed checkouts: dates uniform tip, `is-shallow` true), then `clone_or_pull(url, dest)` → no longer shallow + true per-file dates; the local-clone true-date test and every fail-soft test stay green unchanged.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_git_source_dates.py` (task 02): `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` with the DB up.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate; `scripts/` is outside the `app/` denominator but fully pinned by the suites above).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A fresh `clone_or_pull` over a URL transport yields a NON-shallow checkout (unit argv pin: no `--depth`; integration: `git rev-parse --is-shallow-repository` on the `file://` clone → `false`) with TRUE per-file last-commit dates (2020 file stays 2020, tip-touched file gets the tip date — NOT uniform).
|
||||||
|
- [ ] An EXISTING shallow checkout (made with `--depth 1`, like every deployed one) unshallows on its next `clone_or_pull` (integration pin: probe → `fetch --unshallow` → pull; dates true afterwards) and a non-shallow checkout takes the plain pull path (unit argv pins); a failed unshallow aborts the sync with `GitSyncError` (fail loud, D12).
|
||||||
|
- [ ] `tests/integration/test_git_file_dates.py` green with the D10 tip-date test replaced by the true-date regression pin; `tests/unit/test_git_sync.py` green with the updated argv pins.
|
||||||
|
- [ ] E2E green in isolation: after a real in-app sync of a two-commit `file://` fixture, `GET /api/docs` carries `created_at[:10]` = the OLD commit date for the old file and the TIP date for the new file (the two DIFFER — the bug made them identical); the Sources file table's `Created` column renders the two different years; the old document's viewer `Created` badge carries the old date (ISO `title`); the folder `Updated` columns are the subtree maxes (old folder 2020, new folder 2024).
|
||||||
|
- [ ] The phase-106 regression suites green in isolation: `uv run pytest tests/e2e/test_document_dates.py -v --no-cov`, `tests/e2e/test_sync_button.py -v --no-cov`, `tests/e2e/test_git_sources_admin.py -v --no-cov`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] The five stale shallow/tip-date narratives (git_sync module + two function docstrings, import_docs docstring, sync.py comment) now describe full-history checkouts — no remaining claim that URL sources carry tip dates.
|
||||||
|
- [ ] One `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **D11 — Full-history git checkouts (SUPersedes phase 106 D10; owner-authorized 2026-09-16 — D10's own revisit clause).** Fresh `clone_or_pull` checkouts clone WITHOUT `--depth 1`; existing checkouts are probed with `git rev-parse --is-shallow-repository` and, while shallow, `git fetch --unshallow` before the usual `git pull --ff-only` (one-time self-heal of deployed checkouts, no re-clone). Consequence: `file_commit_dates` returns the TRUE per-file last-commit date for EVERY git source — local and URL — and the next sync after deploy refreshes every git document's stored `created_at` to its true commit date (D4's "may go older" makes this a plain date-only refresh: no `sources_meta` bump, manual corrections survive, `dates_updated` counts them).
|
||||||
|
- **D12 — Fail loud, never silently re-shallow (the house fail-loud rule applied to the self-heal).** A probe/unshallow/pull failure raises `GitSyncError` and aborts the sync exactly like any clone/pull failure; there is no env knob to restore shallow clones or to skip the self-heal — a shallow checkout would silently re-serve the uniform tip date, i.e. the bug this phase fixes.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add scripts/git_sync.py scripts/import_docs.py app/api/sync.py tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(git): full-history checkouts so URL sources get true per-file document dates"
|
||||||
|
```
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Task 01 — `clone_or_pull` full-history checkouts: no `--depth 1` + the shallow self-heal (D11/D12)
|
||||||
|
|
||||||
|
**Phase:** `107_git_full_history_dates` · **Source:** owner bug report 2026-09-16 — URL git sources show the repo TIP date for every document (`container_bifrost` "created 8/16/2026", months off) because phase 106 D10 locked `--depth 1` shallow URL clones.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make every `clone_or_pull` checkout carry FULL git history — fresh clones without `--depth 1`, existing shallow checkouts self-healing via `git fetch --unshallow` before the usual fast-forward — so `file_commit_dates` (unchanged) yields true per-file last-commit dates for all git sources; and correct every docstring/comment that enshrines the old shallow assumption.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `scripts/git_sync.py::clone_or_pull` — the behavior change (keep the signature, the `dest`/`.git` dispatch, and the `GitSyncError` semantics; EVERY invocation through `run_git`, A11):
|
||||||
|
- Fresh checkout (dest absent or without `.git`): `dest.parent.mkdir(parents=True, exist_ok=True)` then `run_git(["git", "clone", url, str(dest)], cwd=dest.parent)` — the `"--depth", "1"` arguments are REMOVED (full history for every transport).
|
||||||
|
- Existing checkout (`.git` present): probe `shallow = run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest).strip() == "true"`; if `shallow` → `run_git(["git", "fetch", "--unshallow"], cwd=dest)` (the one-time self-heal — deployed checkouts, live + dev included, become full-history on their NEXT sync with no re-clone); then `run_git(["git", "pull", "--ff-only"], cwd=dest)` as today. Probe `"false"` → straight to the pull (steady state: one cheap local probe, no network).
|
||||||
|
- **D12 fail-loud:** do NOT catch `GitSyncError` — probe, unshallow, and pull failures all propagate exactly like today's clone/pull failures (the sync aborts with the named repo + git's stderr; never a silent fallback to tip dates or mtimes).
|
||||||
|
- Docstrings (the lies are the rest of the bug):
|
||||||
|
- module docstring: L3 "(shallow, depth 1)" → full-history clone; the L18-28 "Per-file last-commit dates (phase 106, D2/D10)" block → rewrite for phase 107: a `clone_or_pull` checkout is FULL-history for every transport (fresh: no `--depth`; existing: `--is-shallow-repository` probe + one-time `git fetch --unshallow` self-heal before the `--ff-only` pull) → `file_commit_dates` yields TRUE per-file last-commit dates for ALL git sources (local AND URL); cite the 2026-09-16 verification (shallow walk = one tip commit; after `--unshallow` the true per-file dates, e.g. `bifrost.md` 2026-05-05 not the 2026-09-07 tip).
|
||||||
|
- `clone_or_pull` docstring (L48-51 + the behavior bullets): "shallow, first run" → "full history, first run"; the bullet list gains the probe/unshallow step; drop "(shallow: the KB is re-imported incrementally anyway)".
|
||||||
|
- `file_commit_dates` docstring (L104-111): the "local FULL history / URL shallow → uniform TIP date (D10)" paragraph → "every `clone_or_pull` checkout is full history → true per-file last-commit dates for all git sources (phase 107 D11 — supersedes phase 106 D10)"; the fail-soft paragraph stays.
|
||||||
|
2. `scripts/import_docs.py` — docstring L19: "(first run, shallow ``--depth 1``) or fast-forwarded" → "(first run, full history — no ``--depth``; an existing shallow checkout is unshallowed first, phase 107) or fast-forwarded". No code change here (it already calls `clone_or_pull` and then `file_commit_dates` — the fix flows through).
|
||||||
|
3. `app/api/sync.py` — per-row comment (L298-303): replace "(local-path checkouts → true per-file dates, shallow URL checkouts → the tip date, D10)" with "(full-history checkouts → true per-file last-commit dates for every git source — phase 107 D11 supersedes phase 106 D10's shallow tip-date behavior)". No code change.
|
||||||
|
4. Tests (run `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` — DB-free):
|
||||||
|
- `tests/unit/test_git_sync.py` (subprocess fully faked, argv pins — update the module docstring's first lines too: "``git clone`` (full history, fresh dest) or a shallow-probe + optional ``git fetch --unshallow`` + ``git pull --ff-only`` (existing checkout)"):
|
||||||
|
- `test_clone_or_pull_clones_when_dest_has_no_git_dir` → `call["argv"] == ["git", "clone", url, str(dest)]` (NO `--depth`) — same cwd-parent assertion.
|
||||||
|
- `test_clone_or_pull_creates_missing_parent_before_clone` → same argv re-pin.
|
||||||
|
- `test_clone_or_pull_pulls_when_git_dir_exists` → the fake `run_git` must now answer the probe (stdout `"false\n"`) before the pull: `calls[0]["argv"] == ["git", "rev-parse", "--is-shallow-repository"]`, `calls[1]["argv"] == ["git", "pull", "--ff-only"]`, both cwd=dest.
|
||||||
|
- NEW `test_clone_or_pull_unshallows_existing_shallow_checkout` → probe stdout `"true\n"` → `calls[0]` probe, `calls[1]["argv"] == ["git", "fetch", "--unshallow"]`, `calls[2]["argv"] == ["git", "pull", "--ff-only"]` (order pinned: unshallow BEFORE pull).
|
||||||
|
- NEW `test_clone_or_pull_unshallow_failure_propagates` → probe `"true"`, unshallow raises `GitSyncError("git fetch --unshallow failed (exit 128): fatal: …")` → `pytest.raises(GitSyncError, match="--unshallow")` (D12 fail-loud) and the pull is NEVER called.
|
||||||
|
- the error-path test's match string `r"git clone --depth 1 .* failed \(exit 128\): fatal: repository not found"` → `r"git clone .* failed \(exit 128\): fatal: repository not found"`.
|
||||||
|
- The fake-run helper must dispatch by argv (clone/probe/unshallow/pull) — extend it, keep the existing `_FakeProc` shape.
|
||||||
|
- `tests/integration/test_git_file_dates.py` (real `git`, DB-free, keep the `GIT` skip mark + the `scratch_repo` recipe + every fail-soft/parser test):
|
||||||
|
- module docstring: replace the "shallow URL-transport clone → TIP date for EVERY file (D10)" bullet with the phase-107 guarantee (a `clone_or_pull` `file://` checkout is full history → true per-file dates; an existing shallow checkout self-heals) and note the D10 supersession.
|
||||||
|
- REPLACE `test_shallow_file_clone_yields_tip_date_for_every_file` with `test_url_clone_yields_true_per_file_dates(scratch_repo, tmp_path)`: `clone_or_pull(f"file://{scratch_repo}", tmp_path / "url-clone")` → `file_commit_dates(dest) == {"a.md": DATE_A, "b.md": DATE_B, "docs/deep.md": DATE_A}` (the REGRESSION PIN — pre-fix this returned `DATE_B` for all three files) AND `run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest).strip() == "false"` (the checkout is not shallow).
|
||||||
|
- NEW `test_existing_shallow_checkout_self_heals(scratch_repo, tmp_path)`: the harness builds a shallow checkout directly (`_git(tmp_path, "clone", "-q", "--depth", "1", f"file://{scratch_repo}", str(dest))` — simulating the pre-phase deployed checkouts) → pre-heal: `is-shallow` true and `file_commit_dates` uniform `DATE_B` for all three files; then `clone_or_pull(f"file://{scratch_repo}", dest)` → `is-shallow` false and true per-file dates (same dict as the previous test).
|
||||||
|
- `test_local_clone_yields_true_per_file_dates` stays green unchanged (local path was always full history).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: argv + order pins for the fresh-clone (no `--depth`), probe-then-pull, probe→unshallow→pull, and fail-loud paths (the fake subprocess never sees the real git — deterministic).
|
||||||
|
- Integration: real `git` scratch repos — URL-transport true dates, the self-heal lifecycle, the pre-existing fail-soft battery.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` code changes this task — the gate still passes; `scripts/` is pinned by the suites above).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `clone_or_pull` fresh-clone argv is `["git", "clone", url, str(dest)]` (unit-pinned); existing checkouts take probe → (unshallow iff shallow) → `pull --ff-only` (unit-pinned, order asserted); a failed unshallow raises `GitSyncError` before the pull runs (D12)
|
||||||
|
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` green, including `test_url_clone_yields_true_per_file_dates` (per-file true dates over `file://` — the bug's regression pin) and `test_existing_shallow_checkout_self_heals`
|
||||||
|
- [ ] No remaining "shallow"/"tip date" narrative in `scripts/git_sync.py`, `scripts/import_docs.py`, or `app/api/sync.py` describing CURRENT behavior (phase-106 phase records in `complete/` are history — untouched)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean; full `uv run pytest` green
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 02 — E2E: true per-file git dates end to end (`tests/e2e/test_git_source_dates.py`)
|
||||||
|
|
||||||
|
**Phase:** `107_git_full_history_dates` · **Source:** owner bug report 2026-09-16 — the UI must show the document's true last-commit date (a `file://` URL git source is the transport-true stand-in for the live https source; pre-fix it rendered the uniform tip date).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Add the phase's dedicated Playwright suite (A16, run in isolation) that proves the fix end to end: a real `file://` git fixture with two commits of controlled dates, a real in-app admin sync, and assertions that the OLD file shows its OLD commit date and the NEW file shows the TIP date — in `GET /api/docs` / `GET /api/docs/tree` (deterministic ISO) and in the Sources tables + document viewer (locale-tolerant).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_git_source_dates.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_sync_button.py` verbatim in shape: module-scoped `app_server` with per-module env, `tmp_path_factory`-scoped real git fixture, per-test fresh-KB fixture, `e2e.auth_helpers.login`, the conftest `_wait_http`/`USE_REAL_LLM` imports; module docstring: story n/a — owner bug report 2026-09-16; the isolation command; what each test pins):
|
||||||
|
- **Fixture repo** (real `git` subprocesses, controlled `GIT_COMMITTER_DATE`/`GIT_AUTHOR_DATE` + fixed identity, the `sync_git_repo` helper shape — two commits, MID-YEAR dates so the browser's locale/TZ rendering of the year is stable in any timezone, the `test_document_dates.py` L455 lesson):
|
||||||
|
- commit one @ `2020-06-15T12:00:00Z` adds `old/old-note.md` (body: a sentence about an old, stable note).
|
||||||
|
- commit two (the tip) @ `2024-06-15T12:00:00Z` adds `recent/recent-note.md` (body: a sentence about a recent note).
|
||||||
|
- App env: `BOR_GIT_SOURCES=file://<repo>` + its own `BOR_SOURCES_DIR` (fresh dir, the `test_sync_button.py` L141-143 pattern); mock LLM; the session app (no git sources) is never started in this isolated run (no port clash).
|
||||||
|
- **Per-test sync helper:** admin login → the admin Sources page → click "Sync sources" → wait for the success detail (the `test_sync_button.py` lifecycle wait, generous timeout — real clone + mock-LLM embed).
|
||||||
|
- **Tests:**
|
||||||
|
1. `test_api_created_dates_are_true_per_file` — after a sync: `GET /api/docs` (admin, the page context's request client) — the row with `path == "old/old-note.md"` has `created_at[:10] == "2020-06-15"`, the row with `path == "recent/recent-note.md"` has `created_at[:10] == "2024-06-15"`, and the two values DIFFER (the regression assertion — the phase-106 bug made every URL-source file carry the tip date, i.e. both `2024-06-15`). `GET /api/docs/tree`: the file nodes carry those `created_at` verbatim; the `old` folder's `updated_at[:10] == "2020-06-15"`, the `recent` folder's `updated_at[:10] == "2024-06-15"`, the source node's `updated_at[:10] == "2024-06-15"` (subtree max, phase 106 D9 — now over TRUE dates).
|
||||||
|
2. `test_sources_tables_render_distinct_created_dates` — the RAG view's FILE table: the row whose path cell contains `old/old-note.md` has its `Created` cell (the column phase 106 D8 placed between `Chunks` and `Indexed`) matching a regex for the year `2020`; the `recent/recent-note.md` row's `Created` cell matches `2024`; the two cells' text differs. The FOLDER table: the `old` row's `Updated` cell (between `Documents` and `Description`) matches `2020`, the `recent` row's matches `2024`. (Year-regex assertions — `toLocaleString` rendering is locale/TZ-dependent; the year is stable for the mid-year fixture dates. `textContent` reads only — never set innerHTML.)
|
||||||
|
3. `test_viewer_created_badge_is_the_true_git_date` — open the old document (click its row/title, the phase-26 same-page modal): the viewer's `.doc-created` badge (phase 106, before the `Indexed` badge) has `title == "2020-06-15T12:00:00+00:00"` (the raw ISO — `metaBadge`'s title, deterministic under any locale/TZ, the house solution to the L455 rendering trap) and its visible text starts with `Created `; close the modal (Escape) — the viewer reverts cleanly.
|
||||||
|
4. `test_page_a11y_and_no_cdn_basics` — the standard light pass (AGENTS.md rules 5/6, the `test_git_sources_admin.py` a11y test shape): landmarks on the Sources view, the `Created`/`Updated` `<th>` cells present in both tables, ≥4.5:1-free text (no new color), 3px `:focus-visible` on a table row link, same-origin assets only (no external `src`/`href`).
|
||||||
|
- Fresh-KB fixture: truncate `documents`, `chunks`, `git_sources`, `sources_meta`, `kb_overview` (the `test_sync_button.py` `clean_kb` shape) so each test's sync counts are its own.
|
||||||
|
2. Run `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` (DB up: `podman compose up -d db`) — all four tests green in isolation.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (the task IS the test): real `file://` URL-transport clone through `clone_or_pull` (task 01's full-history path) + the real importer + the real API + the real UI — the owner's scenario, transport-true.
|
||||||
|
- Coverage: the suite is `--no-cov`; it exercises `app/` (sync, docs API, tree builder) and `scripts/git_sync.py` for real — the `app/` >90% gate is unaffected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` green in isolation (DB up) — all four tests
|
||||||
|
- [ ] The regression assertion holds: the two fixture documents' `created_at` values DIFFER (2020 vs 2024) in the API, the tables, and the viewer — the uniform-tip-date bug is provably gone for URL transports
|
||||||
|
- [ ] The suite leaves no other suite's fixtures touched (isolation: the session app never starts, own `BOR_SOURCES_DIR`, per-test DB reset)
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# Task 03 — Full gate: suites, coverage, lint/types, regression E2Es, atomic commit
|
||||||
|
|
||||||
|
**Phase:** `107_git_full_history_dates` · **Source:** AGENTS.md rules 8/9 — the test gates are non-negotiable; one atomic, professional commit per completed phase.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Run the complete quality gate for the phase — unit + integration green, `app/` coverage >90%, the new E2E + the three phase-106/28/35 regression E2E suites green in isolation, ruff + pyright clean — then land the single `--no-gpg-sign` Conventional Commit and move the phase directory to `complete/`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. DB up: `podman compose up -d db` (and `uv run alembic upgrade head` if the dev DB is behind — no NEW migration this phase, so head is unchanged).
|
||||||
|
2. Unit + integration: `uv run pytest` — green.
|
||||||
|
3. Coverage: `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90% (no `app/` code changed this phase; this is the regression check on the phase-106 suites + the new E2E's `app/` exercise).
|
||||||
|
4. E2E, each in isolation (`--no-cov`):
|
||||||
|
- the NEW suite: `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov`
|
||||||
|
- regressions (the three suites whose behavior this phase touches or that pin phase-106 dates end to end):
|
||||||
|
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov`
|
||||||
|
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`
|
||||||
|
5. Lint + types: `uv run ruff check . && uv run pyright` — clean.
|
||||||
|
6. UI Structure Check (AGENTS.md rule 5): the phase adds NO UI of its own — the date columns/badge asserted by task 02 are phase-106 UI, re-verified only; confirm nothing in `frontend/` changed this phase (`git status` shows none) so the byte-identical contracts are untouched.
|
||||||
|
7. Commit (exactly one atomic commit, `--no-gpg-sign`, the 00_phase.md message):
|
||||||
|
```bash
|
||||||
|
git add scripts/git_sync.py scripts/import_docs.py app/api/sync.py tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(git): full-history checkouts so URL sources get true per-file document dates"
|
||||||
|
```
|
||||||
|
8. Move the phase directory: `mv .agents/phases/todo/107_git_full_history_dates .agents/phases/complete/` (the pipeline gate does this on success — do it only after step 7 succeeds and include the move in the SAME commit's tree if the gate script does not, per the house protocol; check `.agents/pipeline.log` / the phased-execution gate for how prior phases recorded the move and match it).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task runs, not writes, the gate: every command above must pass before the commit exists.
|
||||||
|
- Coverage: **>90%** on `app/` (TOTAL line of the `term-missing` report).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` green in isolation (DB up)
|
||||||
|
- [ ] `tests/e2e/test_document_dates.py`, `tests/e2e/test_sync_button.py`, `tests/e2e/test_git_sources_admin.py` each green in isolation
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] Exactly one new commit, message `fix(git): full-history checkouts so URL sources get true per-file document dates`, signed with `--no-gpg-sign`; `git status` clean afterwards (only gitignored runtime artifacts aside)
|
||||||
|
- [ ] Phase dir at `.agents/phases/complete/107_git_full_history_dates/`
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Phase 108 — History wire check: verify (or fix) the "missing first turn" follow-up bug
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L4 (owner 2026-09-16): "I've noticed at least one instance where a follow-up chat is missing the first message and response as context. So if I ask 'What is my name' and then 'What did I just ask you?' the model responds 'This is the first question you've asked'. But if I send a third message 'What was the previous question' the model responds correctly 'What did I just ask you?' Just check if there's a bug, there may not be and this was user error"
|
||||||
|
|
||||||
|
**Story:** n/a (owner bug report, phase-74 follow-up — the phase's E2E proves the wire end to end).
|
||||||
|
|
||||||
|
**Context (traced 2026-09-16):** the chat-history wire landed in phase 74 and is three layers deep: (1) the CLIENT maps the `bor.chat.v1` conversation record minus the current question into the request body — `conversation.slice(0, -1)` → `{who, text, thinking?}` per turn (`frontend/assets/app.js` L2247, invariant comment L2229-2246 — the phase-49 retry and phase-53 stale-regen paths pop the old answer before re-sending, so `slice(0,-1)` is exactly the prior turns); (2) the SERVER trims + maps — `history_to_messages` (`app/rag/prompts.py` L199-252): walks NEWEST-FIRST, keeps turns while BOTH budgets hold (`history_max_turns` default **40**, `history_max_chars` default **24_000** — `app/config.py` L94/L101), drops a whole turn on overflow, returns the kept window chronological; `user`→user, `brain`→assistant with `reasoning_content` ONLY when thinking is non-empty (A4); (3) the ENDPOINT splices the block between the system prompt and the current user message on BOTH turn branches (deflected + grounded — `app/api/chat.py` L353-358, "BOTH branches below … reuse the same block"). A short 2-turn conversation is orders of magnitude under both budgets, and both the client mapping and the trimmer READ correctly — so this phase is a deterministic three-layer VERIFICATION with a built-in fix branch, not a rewrite. The wire oracle already exists: the mock LLM's `HISTORY_TRIGGER = "echo my history"` (`tests/e2e/mock_llm.py` L597; checked at L1671 BEFORE the DEFLECT_MODE branch — the echo fires on both branches) answers with `_history_echo(body)` (L1492): a byte-stable `history: N prior messages; last answer tail: <last 24 chars of the most recent prior assistant message, or "none">; thinking: yes|no` — exactly what the owner's scenario needs. The phase-74 E2E (`tests/e2e/test_llm_history.py`) already asserts on this echo, deriving the expected tail from the localStorage record; the existing suites to extend live at `tests/unit/test_history.py` (the pure trimmer) and `tests/integration/test_chat_api.py` (the `HISTORY`/`HISTORY_MESSAGES` idiom L1620-1656 + the `_stream_chat_with_history` helper L1659).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove — at the trimmer, the endpoint, and the full browser wire — that a follow-up question carries the COMPLETE prior conversation (the owner's exact 2- and 3-turn scenarios, byte-exact via the history echo), and either ship the minimal fix at the layer that reproduces the missing-first-turn symptom or record the verdict "no bug — model behavior/user error" with the pins as the permanent guard.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `74_llm_chat_history` (complete) — the feature under verification: the `history` request field, the trimmer, both-branch splicing, the echo marker, and the suites this phase extends. All its pins are regression gates.
|
||||||
|
- `17_thinking_display` (complete) — the record's `thinking` key and the A4 `reasoning_content` wire convention the echo's `thinking: yes|no` term covers.
|
||||||
|
- `49_retry_answer` / `53_stale_saved_chats` (complete) — the client paths (retry, stale-regen) that POP the old answer before re-sending; the client-invariant comment names them — if the client layer ever reproduces, their pop logic is the first suspect.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **The three layers, each isolating a suspect (tasks 01-02):**
|
||||||
|
1. **Unit — the trimmer** (task 01): the owner's exact shape — a 2-turn history (user Q1, brain R1) under the default budgets → ALL turns kept, chronological, roles mapped, thinking mapped (non-empty → `reasoning_content`, empty/absent → key absent). If this fails, the bug is in `history_to_messages` and nothing else needs running.
|
||||||
|
2. **Integration — the endpoint** (task 01): the SAME 2-turn history through the real `POST /api/chat` (the `test_chat_api.py::_stream_chat_with_history` idiom): the SSE turn completes AND the LLM request the turn made carries exactly `[system, user Q1, assistant R1, user Q2]` (captured per the house fake-LLM pattern). If layer 1 passes and this fails, the bug is in the endpoint plumbing (the `request.history` → `hist` → prompt splice, one of the two branches).
|
||||||
|
3. **E2E — the full client wire** (task 02): the owner's exact 3-message scenario in the browser, echo markers on turns 2 and 3 (see task 02 for the messages + expected echoes). If layers 1-2 pass and this fails, the bug is in the CLIENT record→history mapping (push/pop timing, the phase-49/53 paths, localStorage restore).
|
||||||
|
- **The verdict (task 03, D13):** all three green → NO BUG: the wire is proven complete at every layer; the reported instance is model behavior/user error (the owner's own hypothesis). The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3). A failure at layer N → the bug reproduces at layer N; the executor makes the MINIMAL fix in that layer, re-runs the failing layer green, and the verdict records the fix + evidence. `VERDICT.md` (NEW file inside this phase dir) is written BEFORE the commit and states: the layer outcomes, the verdict, and (if fixed) the one-line root cause.
|
||||||
|
- **NOT touched (D13/D14):** the A10 stateless contract, the budget defaults (40 turns / 24k chars), the `bor.chat.v1` record schema, the echo's format (the existing marker IS the oracle — D14: NO new mock marker this phase), any phase-74 pin (regression), `PLAN.md`, completed phases.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_server_wire_verification.md` — layer 1 (unit pins on the trimmer for the owner's 2-turn shape) + layer 2 (integration: real endpoint, captured LLM request = full prior history).
|
||||||
|
2. `02_client_e2e_owner_scenario.md` — layer 3: new E2E `tests/e2e/test_history_wire_check.py` (isolation) — the owner's exact 3-message scenario, byte-exact echo assertions on turns 2 and 3.
|
||||||
|
3. `03_verdict_fix_or_pin.md` — read the layer outcomes; fix the reproducing layer (or record "no bug"); `VERDICT.md`; full gate; atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_history.py` (extended): the 2-turn-under-budget keep-all pin + the role/thinking mapping for that shape (the existing budget/trim pins stay green — regression).
|
||||||
|
- Integration — `tests/integration/test_chat_api.py` (extended): the 2-turn request through the real endpoint with the captured-LLM-request assertion (the house fake-LLM capture pattern; the existing phase-74 history tests stay green).
|
||||||
|
- E2E (mandatory, A16) — `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` with the DB up.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Layer 1 green: a 2-turn history under the default budgets survives `history_to_messages` whole, chronological, correctly mapped (unit pin).
|
||||||
|
- [ ] Layer 2 green: a real `POST /api/chat` with a 2-turn history makes the LLM request `[system, user Q1, assistant R1, user Q2]` — the server wire is proven complete (or the bug is fixed here).
|
||||||
|
- [ ] Layer 3 green: the owner's scenario in the browser — turn 2's echo shows `history: 2 prior messages` + R1's exact 24-char tail; turn 3's echo shows `history: 4 prior messages` + R2's tail (or the bug is fixed at the client).
|
||||||
|
- [ ] `VERDICT.md` exists in the phase dir: layer outcomes + the verdict (fixed-at-layer-N with root cause, or "no bug — model behavior") — written before the commit.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation; `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase 74) green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit (message per the verdict — see Commit); phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **D13 — Verify-or-fix protocol (owner-instructed: "Just check if there's a bug, there may not be").** The phase's deliverable is the three-layer pins + a recorded verdict. Code changes happen ONLY when a layer reproduces the missing-turn symptom, are MINIMAL, and are confined to the reproducing layer — no A10 contract change, no budget-default change, no record-schema change, no new endpoint. If no layer reproduces, the phase ships tests-only.
|
||||||
|
- **D14 — The existing echo IS the oracle.** The phase-74 `echo my history` marker (`_history_echo`) is reused unmodified — its `N prior messages` count + `last answer tail` are exactly the owner-scenario assertions; NO new mock marker is added this phase (new markers land only in phases that change prompt/tool shapes).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
# verdict = no bug (tests-only):
|
||||||
|
git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"
|
||||||
|
|
||||||
|
# verdict = bug found (adjust <layer> to the fix site):
|
||||||
|
git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"
|
||||||
|
```
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 01 — Server wire verification: the trimmer (unit) + the endpoint (integration)
|
||||||
|
|
||||||
|
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "a follow-up chat is missing the first message and response as context … Just check if there's a bug."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove (or disprove) the two SERVER layers of the history wire for the owner's exact 2-turn shape: a short history must survive `history_to_messages` whole and reach the LLM as the complete prior conversation on the real `POST /api/chat`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_history.py` (EXISTING — extend, keep every pin green) — add the owner-shape pins for the DEFAULT budgets (no env overrides; construct `Settings` the file's existing `_settings()` way):
|
||||||
|
- `test_short_two_turn_history_kept_whole_and_chronological` — `history = [user "What is my name?", brain "Your name is Reese."]` (the owner's own Q1/R1) → `history_to_messages` returns exactly `[{"role": "user", "content": "What is my name?"}, {"role": "assistant", "content": "Your name is Reese."}]` — both turns, chronological, no trim, no reordering.
|
||||||
|
- `test_two_turn_history_thinking_mapping` — the same 2-turn history with the brain turn carrying a non-empty `thinking` → the assistant message gains `reasoning_content` (A4); with `thinking` empty/absent → the key is ABSENT (not an empty string).
|
||||||
|
- (If either pin fails: STOP — layer 1 reproduces the bug. Fix `app/rag/prompts.py::history_to_messages` minimally (D13), keep this task's pins + the existing suite green, and note the root cause for task 03's `VERDICT.md`. Do not touch the budget defaults.)
|
||||||
|
2. `tests/integration/test_chat_api.py` (EXISTING — extend next to the phase-74 history block, L1620-1670) — add the endpoint-layer pin:
|
||||||
|
- Reuse the file's `_stream_chat`/`_stream_chat_with_history` helpers + fake-LLM capture pattern (read the file's existing setup first — match its house idiom for capturing what the LLM was called with).
|
||||||
|
- `test_endpoint_two_turn_history_reaches_the_llm` — `POST /api/chat {message: "What did I just ask you?", history: [{who: user, text: "What is my name?"}, {who: brain, text: "Your name is Reese."}]}` → the SSE stream completes (`done`), and the chat request the turn made to the LLM carries, IN ORDER, the system prompt, `user "What is my name?"`, `assistant "Your name is Reese."`, then the current `user` question — i.e. the 2 prior turns are NOT dropped (the owner's symptom would be their absence). Assert on the captured `messages` list (roles + contents, exact).
|
||||||
|
- The turn may be LOW/deflected with an empty KB (the history block is branch-independent — pinned in phase 74) or HIGH with one seeded fixture doc (the file's existing seeding idiom) — either is fine; pick what the file's helpers make easiest and say so in a comment.
|
||||||
|
- (If this fails while layer 1 passed: the bug is in the endpoint plumbing — `app/api/chat.py`'s `request.history` → `hist` → prompt splice. Fix minimally (D13), keep this pin + the phase-74 pins green, note the root cause for task 03.)
|
||||||
|
3. Run `uv run pytest tests/unit/test_history.py tests/integration/test_chat_api.py -v` (DB up: `podman compose up -d db`) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the owner-shape trimmer pins (keep-all + mapping) alongside the existing budget pins.
|
||||||
|
- Integration: the real endpoint with a captured LLM request — the server wire proven (or fixed) at the exact layer.
|
||||||
|
- Coverage: **>90%** on `app/` (no `app/` change unless a fix is needed; the gate still passes).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/unit/test_history.py` green with the two new owner-shape pins (or the trimmer fixed + pinned)
|
||||||
|
- [ ] `tests/integration/test_chat_api.py` green with `test_endpoint_two_turn_history_reaches_the_llm` (or the endpoint fixed + pinned)
|
||||||
|
- [ ] Every pre-existing pin in both files still green (no regression)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean; the layer-1/layer-2 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Task 02 — Client E2E: the owner's exact 3-message scenario, byte-exact via the history echo
|
||||||
|
|
||||||
|
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — the owner's repro: Q1 "What is my name?" → Q2 "What did I just ask you?" (model claims it's the first question) → Q3 "What was the previous question?" (model answers correctly).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove (or disprove) the THIRD layer — the full browser wire: the localStorage conversation record → `conversation.slice(0,-1)` mapping → request body → the LLM — using the owner's exact scenario and the phase-74 `echo my history` oracle (D14: no new marker).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_history_wire_check.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_llm_history.py`: module-scoped mock-LLM app, the fixture-docs import for a non-empty KB, `e2e.auth_helpers.login`, the localStorage `bor.chat.v1` record reads, per-test conversation reset; module docstring: story n/a — owner bug report 2026-09-16, the isolation command, and what each test pins). Isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (DB up).
|
||||||
|
- **The echo oracle, recalled** (mock `_history_echo`, byte-stable): `history: N prior messages; last answer tail: <LAST 24 CHARS of the most recent prior assistant message's content, or "none">; thinking: yes|no` — N = non-system messages before the LAST user message (the current question excluded); checked BEFORE the DEFLECT_MODE branch, so the echo fires whatever gate branch the turn takes (the owner's questions may deflect — that's fine, the marker is in the USER message).
|
||||||
|
- **Tests:**
|
||||||
|
1. `test_cold_start_echo_shows_no_phantom_history` — fresh conversation; ask `echo my history` as the FIRST message → the answer bubble contains `history: 0 prior messages; last answer tail: none; thinking: no` (the cold-start pin: no phantom prior turns).
|
||||||
|
2. `test_owner_scenario_three_turns_carry_the_full_prior_history` — the owner's exact scenario, echo marker APPENDED to turns 2 and 3 (their words preserved verbatim as the prefix):
|
||||||
|
- T1: `What is my name?` → R1 (the mock's deterministic answer — read R1's raw text from the `bor.chat.v1` record's brain entry, NOT from the rendered DOM).
|
||||||
|
- T2: `What did I just ask you? echo my history` → R2's bubble text must contain `history: 2 prior messages; last answer tail: {R1[-24:]}; thinking: no` (R1 = the record's brain text; `thinking: no` — T1 never triggered the thinking marker). **THE regression pin: the owner's bug renders this as `0 prior messages` / `last answer tail: none`.**
|
||||||
|
- T3: `What was the previous question? echo my history` → R3's bubble text must contain `history: 4 prior messages; last answer tail: {R2[-24:]}` (R2 = the echo answer itself — also from the record).
|
||||||
|
- Read the expected tails from the localStorage record AFTER each turn persists (the `test_llm_history.py` pattern — the record the client saved IS what the client sends next, so what the record shows is what the model received).
|
||||||
|
- If this test fails while tasks 01's layers passed: the bug is in the CLIENT mapping (suspects, in order: the `conversation.slice(0,-1)` sites, the phase-49 retry / phase-53 stale-regen pop paths, the record persistence timing — `frontend/assets/app.js` L2187/L2247). Fix minimally (D13), keep this test + `tests/e2e/test_llm_history.py` green, note the root cause for task 03's `VERDICT.md`.
|
||||||
|
2. Run the suite in isolation — green (or pass-after-fix).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E (the task IS the test): the full browser wire, byte-exact via the existing echo oracle.
|
||||||
|
- Coverage: `--no-cov` suite; it exercises `app/` (chat endpoint, history mapping) for real — the `app/` >90% gate is unaffected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation (DB up) — both tests
|
||||||
|
- [ ] The regression pin holds: turn 2's echo shows `2 prior messages` + R1's exact tail; turn 3 shows `4 prior messages` + R2's exact tail (or the client bug is fixed + pinned)
|
||||||
|
- [ ] `tests/e2e/test_llm_history.py` (phase 74) still green in isolation (regression)
|
||||||
|
- [ ] The layer-3 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 03 — Verdict (fix or pin), full gate, atomic commit
|
||||||
|
|
||||||
|
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "Just check if there's a bug, there may not be and this was user error"; AGENTS.md rules 8/9 — the test gates are non-negotiable.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Record the phase's verdict with its evidence, run the complete quality gate, and land the single `--no-gpg-sign` commit — tests-only if no bug was found (the owner's expected outcome), fix + tests if one layer reproduced.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. **The verdict** — read the layer outcomes from tasks 01-02 (their test results + any fix notes):
|
||||||
|
- **All three layers green (no fix needed):** the wire is proven complete at the trimmer, the endpoint, and the full browser wire → verdict **NO BUG**: the owner's reported instance was model behavior/user error. The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3).
|
||||||
|
- **A layer reproduced (pass-after-fix):** the bug is fixed at that layer → verdict **BUG FOUND + FIXED at <layer>**, with the one-line root cause.
|
||||||
|
- Write `.agents/phases/todo/108_history_wire_check/VERDICT.md` BEFORE the commit: the three layer outcomes (pass / pass-after-fix + root cause / fail-should-not-occur), the verdict, and the evidence (which test names carry the pins). Keep it short — it is the durable record the owner asked for ("just check").
|
||||||
|
2. **The full gate** (DB up: `podman compose up -d db`; every command must pass before the commit):
|
||||||
|
- `uv run pytest` — green.
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90%.
|
||||||
|
- E2E in isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (NEW) and `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression).
|
||||||
|
- `uv run ruff check . && uv run pyright` — clean.
|
||||||
|
- No-regression spot check: `git diff --stat` shows ONLY the files this phase may touch — `tests/**`, `VERDICT.md`, `.agents/phases/**`, and (only if a bug was fixed) the single reproducing layer's file. If the diff shows anything else, stop and fix the scope before committing.
|
||||||
|
3. **The commit** (exactly one, `--no-gpg-sign`, per the 00_phase.md branch):
|
||||||
|
- no bug: `git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"`
|
||||||
|
- bug found: `git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"`
|
||||||
|
4. Move the phase directory: `mv .agents/phases/todo/108_history_wire_check .agents/phases/complete/` (the pipeline gate does this on success — do it only after the commit, and match how prior phases recorded the move).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task runs, not writes, the gate: every command above must pass before the commit exists.
|
||||||
|
- Coverage: **>90%** on `app/` (TOTAL line of the `term-missing` report).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `VERDICT.md` in the phase dir: layer outcomes + verdict (no bug / fixed-at-<layer> + root cause) + the pin test names
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||||
|
- [ ] `tests/e2e/test_history_wire_check.py` + `tests/e2e/test_llm_history.py` green in isolation (DB up)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean; the diff is scoped to this phase's allowed files
|
||||||
|
- [ ] Exactly one new commit with the verdict-branch message, `--no-gpg-sign`; `git status` clean afterwards (only gitignored runtime artifacts aside)
|
||||||
|
- [ ] Phase dir at `.agents/phases/complete/108_history_wire_check/`
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Phase 108 Verdict — History wire check (TODO.md L4)
|
||||||
|
|
||||||
|
**Verdict: NO BUG.** The follow-up history wire is proven complete at all
|
||||||
|
three layers, for the owner's exact scenario, byte-exact via the phase-74
|
||||||
|
`echo my history` oracle (D14 — no new marker). The reported instance
|
||||||
|
("What did I just ask you?" → "This is the first question you've asked")
|
||||||
|
was **model behavior / user error**, exactly the owner's own hypothesis
|
||||||
|
("Just check if there's a bug, there may not be"). No `app/` or
|
||||||
|
`frontend/` changes were needed — the phase ships tests-only (D13).
|
||||||
|
|
||||||
|
## Layer outcomes
|
||||||
|
|
||||||
|
| Layer | What it isolates | Outcome | Pins (evidence) |
|
||||||
|
|-------|------------------|---------|-----------------|
|
||||||
|
| 1 — unit: the trimmer | `history_to_messages` (`app/rag/prompts.py`) | **PASS** — the owner's 2-turn history (Q1 "What is my name?" / R1 "Your name is Reese.") under the DEFAULT budgets (40 turns / 24 000 chars, no overrides) survives whole: chronological, roles mapped, thinking → `reasoning_content` only when non-empty (A4) | `tests/unit/test_history.py::test_short_two_turn_history_kept_whole_and_chronological`, `::test_two_turn_history_thinking_mapping` |
|
||||||
|
| 2 — integration: the endpoint | `POST /api/chat` splice, both branches (`app/api/chat.py`) | **PASS** — the SAME 2-turn history through the real endpoint (follow-up "What did I just ask you?") makes the LLM request exactly `[system, user Q1, assistant R1, user Q2]`; SSE `done`; log line carries `history_msgs=2`. Branch-agnostic by design (phase 74 pinned the splice on both branches) | `tests/integration/test_chat_api.py::test_endpoint_two_turn_history_reaches_the_llm` |
|
||||||
|
| 3 — E2E: the full browser wire | client record → `conversation.slice(0, -1)` → body → LLM (`frontend/assets/app.js`) | **PASS** — the owner's verbatim 3-message scenario in a real browser: T2's echo = `history: 2 prior messages` + R1's exact 24-char tail; T3's echo = `history: 4 prior messages` + R2's exact tail (tails read from the persisted `bor.chat.v1` record — what the client saved is what it sends). The missing-first-turn bug would render T2 as `0 prior messages` / `last answer tail: none`; it does not | `tests/e2e/test_history_wire_check.py::test_cold_start_echo_shows_no_phantom_history`, `::test_owner_scenario_three_turns_carry_the_full_prior_history` |
|
||||||
|
|
||||||
|
## Why "no bug" is sound
|
||||||
|
|
||||||
|
- Layers 1 and 2 prove the server drops nothing: a short history is
|
||||||
|
orders of magnitude under both budgets, and the captured LLM request on
|
||||||
|
the real endpoint carries the complete prior exchange, chronological.
|
||||||
|
- Layer 3 proves the client sends what it saved: the echo's `N prior
|
||||||
|
messages` count and `last answer tail` are computed by the mock from
|
||||||
|
the request itself, and both matches hold byte-exactly for turns 2 and 3.
|
||||||
|
- The deterministic mock leaves no room for "the model forgot" at the
|
||||||
|
wire level — the model (turbo) simply receives the full history and
|
||||||
|
occasionally mis-reports it. The owner's third message working correctly
|
||||||
|
is consistent with model non-determinism over a complete wire.
|
||||||
|
|
||||||
|
## The pins stay as the permanent guard
|
||||||
|
|
||||||
|
A future regression that drops the first turn — in the trimmer, the
|
||||||
|
endpoint splice, or the client's record→history mapping (including the
|
||||||
|
phase-49 retry / phase-53 stale-regen pop paths) — fails layer 1, 2, or 3
|
||||||
|
immediately.
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# Phase 109 — Never-frozen turn: re-expanding thinking block + the persistent in-turn loader
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 (owner 2026-09-16): "Thinking can happen after the model starts responding. This sometimes results in a the chat appearing 'frozen' because the model responds, calls a tool, then continues thinking without re-expanding the thinking block. There should be a visual that the chat is still progressing regardless of what state it's in (some kind of loader will do)."
|
||||||
|
|
||||||
|
**Story:** n/a (owner request — extends the phase 17/48/87 thinking/tool feedback under the PLAN §7.4 never-stale contract; the phase's E2E proves the reported repro no longer freezes).
|
||||||
|
|
||||||
|
**Context (traced 2026-09-16):** the turn's visible feedback is state-driven in `frontend/assets/app.js`: the `UI_STATE` machine (L340-345: `idle`/`thinking`/`streaming`/`error`) is owned by `setUiState` (L1243) — the typing bubble (a `#typing-indicator` message with the animated `.bubble.typing` dots, `addTyping` L859, the 10s elapsed-seconds clock L1137) shows ONLY in `thinking`; `inFlight = thinking|streaming` drives the Stop button (L1251-1255); `#send-status` is the sole a11y live region (L347-353; visual elements are `aria-hidden` — the L1791 house pattern). The reported freeze, exactly as the owner described it: the `delta` handler (L2357-2370) runs `setUiState(streaming)` on the FIRST delta — which `removeTyping()`s the dots — then `closeThinkingBlock(wrap)` (L905-909: "auto-collapse; idempotent, **never reopens**"). A LATER `thinking` frame (the next agent round — the model answered, called a tool, then thinks again) hits the `thinking` handler (L2268-2301), which only appends to the collapsed block's `.thinking-text` — nothing is visible: the answer text is static, the dots are gone, the scratchpad is closed → the chat reads as frozen. Every OTHER state already has a cue: pre-delta thinking = live open block + dots; tool = the `.tool-call` line with the phase-87 `(Ns)` elapsed counter (plus relabeled dots pre-delta); streaming = growing text; retry = the status line. The post-delta thinking gap is the ONE uncovered state — and the owner wants a constant cue anyway ("regardless of what state it's in"). The CSS lives in `frontend/assets/styles.css` (the typing-dots rules there; `prefers-reduced-motion` is house law, §7.2). House patterns: unit pins read the assets as text (`tests/unit/test_frontend_tool_states.py` / `test_frontend_feedback.py` already pin the thinking-block + typing behavior); the mock LLM has `THINKING_TRIGGER = "think out loud"` (mock_llm.py L500 — streams ~700 chars of `reasoning_content` ahead of content) and the multi-round `tool_calls` markers (L77-191); the house rule is that a marker/regex change lands WITH its consuming task (PLAN §4).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The chat never reads as frozen: (1) a `thinking` frame RE-OPENS the thinking block after the answer has started (the block is open-while-thinking / closed-while-answering — the reported symptom, fixed at the handler), and (2) a compact persistent loader is visible for the ENTIRE active turn (send → terminal frame) in the composer status area — the constant progress cue the owner asked for, driven by the single `setUiState` owner so it can never go stale (§7.4).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `17_thinking_display` (complete) — the thinking block, the follow-the-tail pin contract (`THINKING_NEAR_BOTTOM_PX`), the restore-path collapsed rendering; the block's unit pins live in the `test_frontend_*` suites this phase extends.
|
||||||
|
- `48_stop_generation` (complete) — the `inFlight`/Stop-button state ownership the loader's single-owner toggle joins.
|
||||||
|
- `87_big_read_progress` (complete) — the `.tool-call` line + the `armToolLineClock`/`settleToolLine` `(Ns)` counter (the "at least one cue" inventory's tool entry).
|
||||||
|
- `06_loading_feedback` (complete) — the UI state machine + the never-stale feedback contract this phase extends (every state keeps a defined UI).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
- **D15 — The thinking block becomes a TOGGLE (task 01).** The `thinking` SSE handler gains `block.open = true` after `ensureThinkingBlock(wrap)` (idempotent — a no-op while already open, so the pre-delta live flow is byte-identical in behavior); the `delta` handler KEEPS its `closeThinkingBlock(wrap)`. The contract flips from "never reopens" to **open-while-thinking, closed-while-answering** — the block reflects the model's current activity in every agent round. `closeThinkingBlock`'s docstring/comment updates (the "never reopens" claim is gone — the delta handler closes, the thinking handler opens). The follow-the-tail pin logic (L2293: `block.open && isThinkingNearBottom(textEl)` measured BEFORE the re-render) is UNCHANGED — it already keys off `block.open`, so a re-opened block resumes pinned tail-following exactly like the live pre-delta block. The phase-14 RESTORE path (`renderStoredMessage` L1517-1518) still renders stored blocks collapsed — untouched.
|
||||||
|
- **D16 — The turn loader (task 02): a static shell element, single-owner visibility.**
|
||||||
|
- `frontend/index.html` — ONE static element in the composer's status row (next to the `#send-status` live region): `<div id="turn-loader" class="turn-loader" aria-hidden="true" hidden></div>` — static markup, hidden by default (no JS-built HTML — the createElement/textContent house rule; no document-derived data anywhere near it).
|
||||||
|
- `frontend/assets/app.js` — `setUiState` (L1243) is the SOLE owner, exactly like the existing `is-stop` toggle: `turnLoader.hidden = !inFlight` (shown iff `uiState ∈ {thinking, streaming}`). Every terminal path funnels through `setUiState` (done → `idle`, error → `error`, stop/timeout → `error`/`idle` per the existing handlers), so the loader CANNOT be left visible in a terminal state — the §7.4 never-stale guarantee comes from the single-owner pattern, not from per-handler cleanup.
|
||||||
|
- `frontend/assets/styles.css` — `.turn-loader` next to the typing-dots rules: compact, reuses the EXISTING typing-dot animation (same keyframes/dot styling — no new animation family), provenance comment citing phase 109 + `TODO.md` L3, and a `prefers-reduced-motion` variant mirroring the typing dots' treatment (static dots, no pulse). Contrast N/A (the dots are decorative — `aria-hidden` + the `#send-status` announcer carry meaning; §7.2 "text + color, never color alone" — the state TEXT stays in `#send-status`).
|
||||||
|
- **The invariant (unit + E2E):** while a turn is active, at least one visible progress cue is ALWAYS present — the loader (constant, D16), the open thinking block (thinking frames, D15), the tool-line `(Ns)` counter (tool frames, phase 87), or the growing answer text (streaming). D16 makes it true by construction; task 03 proves the reported repro (delta → tool → thinking-after-delta) no longer freezes.
|
||||||
|
- **NOT touched:** the `UI_STATE` set, the `SEND_STATUS` copy, the typing bubble's own lifecycle (it still shows only pre-delta, per the phase-17 contract — the loader is a SEPARATE constant cue, not a re-homing of the dots), `#send-status` (unchanged — still the sole a11y announcer), the mock's EXISTING markers, the server (this is a pure UI phase — `app/` is untouched, so `app/` coverage is a regression check only), `PLAN.md`, completed phases.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_thinking_block_reexpand.md` — the `thinking` handler re-opens the collapsed block; the delta handler keeps closing; the "never reopens" narrative updated; unit pins (read-the-assets pattern).
|
||||||
|
2. `02_turn_active_loader.md` — the static `#turn-loader` element, the `setUiState` single-owner toggle, the CSS (reused dot animation + reduced-motion + provenance), unit pins.
|
||||||
|
3. `03_e2e_and_gate.md` — the new mock marker forcing the reported `delta → tool → thinking-after-delta` sequence (lands WITH this task), the dedicated E2E `tests/e2e/test_turn_progress_loader.py` (isolation), the a11y pass, the full gate, the atomic commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_frontend_turn_loader.py` (NEW, the house read-the-assets-as-text pattern): task 01 pins — the `thinking` handler contains the `block.open = true` re-open (after `ensureThinkingBlock`), the `delta` handler still calls `closeThinkingBlock`, `closeThinkingBlock`'s docstring no longer claims "never reopens", the restore path (`renderStoredMessage`) still sets `block.open = false`; task 02 pins — `index.html` carries exactly one `#turn-loader` with `aria-hidden="true"` + `hidden`, `setUiState` is the SOLE writer of `turnLoader.hidden` (cross-file single-owner check: `turnLoader.hidden` appears nowhere else in `app.js`), the `.turn-loader` CSS rule exists next to the typing rules with the reduced-motion variant + the phase-109 provenance comment.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_turn_progress_loader.py` (task 03): `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` with the DB up.
|
||||||
|
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — unchanged by this UI phase; the gate is the regression check).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A `thinking` frame after the answer has started RE-OPENS the thinking block (the block is open-while-thinking / closed-while-answering; pre-delta flow + restore path unchanged — unit-pinned).
|
||||||
|
- [ ] `#turn-loader` is visible for the entire active turn and hidden in every terminal state — owned solely by `setUiState` (unit-pinned single-owner + the E2E's start/mid/end samples).
|
||||||
|
- [ ] E2E green in isolation: the reported repro (delta → tool → thinking-after-delta) shows the re-opened scratchpad with the new thinking text, the loader visible throughout, hidden after `done`; `#send-status` carries the state text (the loader is `aria-hidden`).
|
||||||
|
- [ ] The phase-17/48/87/6 regressions green in isolation: `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`, `tests/e2e/test_stop_generation.py -v --no-cov`, `tests/e2e/test_big_read_progress.py -v --no-cov`, `tests/e2e/test_loading_feedback.py -v --no-cov`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **D15 — The thinking block is a toggle, not a one-way door (owner-instructed: "continues thinking without re-expanding the thinking block" is THE reported defect).** `thinking` frames open the block (idempotent), `delta` frames close it; the follow-the-tail pin contract and the restore path are unchanged. No new block, no new state — the existing scratchpad reflects the model's current activity in every round.
|
||||||
|
- **D16 — The constant cue is a separate static loader, owned by `setUiState` (owner-instructed: "a visual that the chat is still progressing regardless of what state it's in — some kind of loader will do").** A static `#turn-loader` in the composer status row (reused typing-dot animation, `aria-hidden`, `#send-status` stays the sole announcer), shown iff `inFlight` — the single-owner pattern makes a stale loader impossible. The typing bubble's own pre-delta lifecycle is NOT re-homed (the phase-17 contract stands); the loader ADDS the constant the owner asked for.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/index.html frontend/assets/app.js frontend/assets/styles.css tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): never-frozen turn — re-expanding thinking block + the persistent in-turn loader"
|
||||||
|
```
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — The thinking block becomes a toggle: re-open on `thinking` frames, close on `delta`
|
||||||
|
|
||||||
|
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "the model responds, calls a tool, then continues thinking without re-expanding the thinking block."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Fix the reported freeze at its source: a `thinking` frame that arrives after the answer has started re-opens the collapsed thinking block (D15) — the scratchpad is visible exactly while the model is thinking, in every agent round — while the `delta` handler keeps closing it and nothing else about the block's lifecycle changes.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — the `thinking` SSE handler (L2268-2301): after `const block = ensureThinkingBlock(wrap);`, add `block.open = true;` (idempotent — while the block is already open (the pre-delta live flow) this is a no-op, so that flow's behavior is unchanged; after a `delta` closed it, this re-opens it for the new round's thinking). Update the handler's comment to state the toggle contract (D15): open-while-thinking, closed-while-answering — phase 109, `TODO.md` L3.
|
||||||
|
- The follow-the-tail logic below it (`const pinned = block.open && isThinkingNearBottom(textEl);`, L2293) is UNCHANGED — it already reads `block.open` before the re-render, so a re-opened block resumes pinned tail-following exactly like the live pre-delta block.
|
||||||
|
2. `frontend/assets/app.js` — `closeThinkingBlock` (L905-909): keep the function exactly as-is (the `delta` handler still calls it, L2369); update its docstring/comment — the "never reopens" claim is replaced by the toggle contract (the `thinking` handler re-opens; the `delta` handler closes).
|
||||||
|
- `renderStoredMessage` (L1517-1518 — the phase-14 restore path) is UNTOUCHED: stored blocks still render collapsed.
|
||||||
|
3. `tests/unit/test_frontend_turn_loader.py` (NEW — the house read-the-assets-as-text pattern; copy the file header/docstring conventions from `tests/unit/test_frontend_tool_states.py`):
|
||||||
|
- `test_thinking_handler_reopens_the_collapsed_block` — the `thinking` handler source contains `block.open = true` positioned AFTER the `ensureThinkingBlock(wrap)` line (order asserted — the block must exist before it opens).
|
||||||
|
- `test_delta_handler_still_closes_the_block` — the `delta` handler still calls `closeThinkingBlock(wrap)` (the close side of the toggle survives).
|
||||||
|
- `test_close_thinking_block_docstring_says_toggle_not_one_way` — `closeThinkingBlock`'s comment no longer contains "never reopens"; the toggle contract (open-while-thinking / closed-while-answering) is documented (assert the new wording, e.g. it names the `thinking` handler's re-open).
|
||||||
|
- `test_restore_path_still_collapses_stored_blocks` — `renderStoredMessage` still sets `block.open = false` (phase-14 contract regression).
|
||||||
|
- Cross-file: the `THINKING_NEAR_BOTTOM_PX` pin + `isThinkingNearBottom` logic are untouched (assert the pre-render `block.open &&` guard line still exists in the handler).
|
||||||
|
4. Run `uv run pytest tests/unit/test_frontend_turn_loader.py -v` + the existing frontend suites that pin this area (`tests/unit/test_frontend_tool_states.py tests/unit/test_frontend_feedback.py tests/unit/test_frontend_scroll.py tests/unit/test_big_read_progress.py`) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the toggle contract pinned at the source level (the house pattern for `app.js` behavior — no browser).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged — pure frontend task; the gate is the regression check).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The `thinking` handler re-opens the block (unit-pinned, correct order); the `delta` handler still closes it; the "never reopens" narrative is gone (unit-pinned)
|
||||||
|
- [ ] The restore path + the follow-the-tail pin logic are untouched (unit-pinned regressions)
|
||||||
|
- [ ] The existing frontend unit suites green; `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 02 — The persistent in-turn loader: static `#turn-loader`, single-owner visibility in `setUiState`
|
||||||
|
|
||||||
|
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "There should be a visual that the chat is still progressing regardless of what state it's in (some kind of loader will do)."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Add the constant progress cue (D16): a compact animated loader in the composer status row that is visible for the ENTIRE active turn (send → terminal frame), owned solely by `setUiState` so it can never be left stale — and hidden in every terminal state by construction.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — ONE static element in the composer's status row (the L290-300 region holding `#char-count` / `#send-btn` / `#send-status` — inspect the actual markup and place it adjacent to `#send-status` so it reads as the status line's companion):
|
||||||
|
```html
|
||||||
|
<div id="turn-loader" class="turn-loader" aria-hidden="true" hidden></div>
|
||||||
|
```
|
||||||
|
Static markup, `hidden` by default (idle on load). No JS-built HTML anywhere (the createElement/textContent house rule — this element is never constructed in JS).
|
||||||
|
2. `frontend/assets/app.js` — the single-owner toggle (D16):
|
||||||
|
- At the top of the module with the other element lookups (near `const sendStatus = document.querySelector("#send-status");` L307): `const turnLoader = document.querySelector("#turn-loader");`
|
||||||
|
- Inside `setUiState` (L1243), next to the existing `sendBtn.classList.toggle("is-stop", inFlight);` (L1251-1255): `turnLoader.hidden = !inFlight;` — shown iff `uiState ∈ {thinking, streaming}`. This is the SOLE writer of `turnLoader.hidden` in the file: every terminal path (done → `idle`, error → `error`, stop/timeout → the existing error/idle landings) funnels through `setUiState`, so the loader is hidden in every terminal state BY CONSTRUCTION — the §7.4 never-stale guarantee, no per-handler cleanup (that is the point; comment it that way).
|
||||||
|
- Do NOT touch the typing bubble's lifecycle (`addTyping`/`removeTyping` stay exactly as-is — the phase-17 pre-delta contract stands; the loader is a separate constant cue), the `SEND_STATUS` copy, or `#send-status` (still the sole a11y announcer — the loader is `aria-hidden` decoration, the L1791 house pattern).
|
||||||
|
3. `frontend/assets/styles.css` — the `.turn-loader` rule NEXT TO the typing-dots rules (find the `.bubble.typing` / dots animation block):
|
||||||
|
- Compact horizontal three-dot indicator, REUSING the existing typing-dot keyframes/dot styling (same animation name — no new animation family; sized down for the status row).
|
||||||
|
- Provenance comment: phase 109, `TODO.md` L3 — the constant in-turn progress cue; decorative (`aria-hidden`), `#send-status` carries the meaning.
|
||||||
|
- A `prefers-reduced-motion` variant mirroring the typing dots' treatment (static dots, no pulse — §7.2 house law).
|
||||||
|
4. `tests/unit/test_frontend_turn_loader.py` (EXTEND the file task 01 created):
|
||||||
|
- `test_index_html_carries_exactly_one_turn_loader` — `frontend/index.html` contains exactly ONE `id="turn-loader"`, with `aria-hidden="true"` and the `hidden` attribute (hidden by default).
|
||||||
|
- `test_set_ui_state_is_the_sole_owner_of_the_loader` — in `app.js`: `turnLoader.hidden` appears EXACTLY ONCE, inside `setUiState` (the cross-file single-owner check — grep the file text; any second write site fails the test, keeping the never-stale guarantee structural).
|
||||||
|
- `test_loader_css_reuses_the_typing_animation_and_reduced_motion` — the `.turn-loader` rule exists in `styles.css` after (or adjacent to) the typing-dots rules, references the SAME animation name as the typing dots, and a `prefers-reduced-motion` block covers it (static, no pulse); the provenance comment names phase 109.
|
||||||
|
- `test_loader_is_aria_hidden_and_status_untouched` — the loader element is `aria-hidden`; `#send-status`'s attributes are unchanged in `index.html` (still the live region — the house a11y split).
|
||||||
|
5. Run `uv run pytest tests/unit/test_frontend_turn_loader.py tests/unit/test_frontend_tool_states.py tests/unit/test_theme_frontend.py -v` (the theme suite parses `styles.css` — a new CSS rule must not break the built-in-theme pins) — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the single-owner invariant + the markup/CSS contract pinned at the source level (the house pattern).
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged — pure frontend task; the gate is the regression check).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `#turn-loader` exists exactly once in `index.html` (static, `aria-hidden`, hidden by default); `setUiState` is its sole visibility owner (unit-pinned)
|
||||||
|
- [ ] The CSS reuses the typing-dot animation, has the reduced-motion variant + the phase-109 provenance comment (unit-pinned); the theme CSS-parsing suites stay green
|
||||||
|
- [ ] The typing bubble's lifecycle, `SEND_STATUS`, and `#send-status` are byte-unchanged in behavior (the existing frontend suites green)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean; full `uv run pytest` green
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Task 03 — E2E: the reported repro (delta → tool → thinking-after-delta) + full gate + commit
|
||||||
|
|
||||||
|
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "the model responds, calls a tool, then continues thinking without re-expanding the thinking block" + "a visual that the chat is still progressing regardless of what state it's in"; AGENTS.md rules 4/8/9.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the fix end to end with a dedicated Playwright suite: a deterministic mock sequence that replays the owner's exact repro (answer starts, tool call, thinking AFTER the answer) must show the re-opened scratchpad, the loader visible throughout, and clean terminal states — then run the full gate and land the atomic commit.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/mock_llm.py` — ONE new marker (the house rule: marker/regex changes land WITH their consuming task — this task; document it in the module docstring next to the existing markers):
|
||||||
|
- A new `*_TRIGGER` constant + branch (checked like the other user-message markers, BEFORE the DEFLECT_MODE branch) whose question forces the reported sequence with BAKED-IN DELAYS (mid-turn windows of ≥1 s each, so Playwright assertions are deterministic — the `slow_llm.py` precedent for deliberate pacing):
|
||||||
|
- model call 1: ~2 s pre-delay (model latency — the loader's start-state window), then a short `content` delta (2-3 chunks; NO reasoning), then an `ls` `tool_calls` delta (synthetic id, no arguments — the L77-81 pattern), `finish_reason: "tool_calls"`.
|
||||||
|
- model call 2 (after the server's `tool_result`): `reasoning_content` chunks (~10 × ~0.3 s), then a `content` delta (2-3 chunks ending in a DISTINCTIVE final sentence the tests can match), then a FINAL `reasoning_content` chunk (3 × ~0.3 s), then finish.
|
||||||
|
- The server is position-independent over the wire (each `reasoning_content` chunk → a `thinking` SSE frame, each `content` chunk → a `delta` frame — `app/rag/llm.py` L572+), so the resulting SSE is exactly `delta → tool → tool_result → thinking → delta → thinking → done` — the owner's repro, deterministic.
|
||||||
|
2. `tests/e2e/test_turn_progress_loader.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_llm_history.py`: module-scoped mock-LLM app, fixture-docs import, `login`, per-test fresh conversation; module docstring: story n/a — owner request 2026-09-16, the isolation command, the marker contract, and what each test pins). Isolation: `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` (DB up). Each test sends the marker question in its OWN fresh conversation (one full turn per test):
|
||||||
|
- `test_loader_visible_from_send_through_the_tool_gap` — right after the send (inside call 1's 2 s pre-delay window): `#turn-loader` is VISIBLE (the thinking state, no frame yet); then wait for the `.tool-call` line to appear (the turn is provably in flight) → the loader is STILL visible, the tool line carries the phase-87 elapsed counter, and `#send-status` carries a state text (not empty).
|
||||||
|
- `test_thinking_block_reopens_after_delta_with_visible_loader` — wait until the thinking block is open with non-empty `.thinking-text` AND the answer bubble already carries call 1's content (i.e. the post-delta re-open — THE reported symptom's state): assert the block is `open`, the new thinking text is VISIBLE in it, and the loader is STILL visible (the frozen window is gone). Then wait for the terminal state (call 2's distinctive final sentence in the bubble, or the send button back to "Send"): the loader is HIDDEN, the block is still open (the LAST frame was thinking), `.thinking-text` is non-empty, the bubble contains BOTH call 1's and call 2's content, and the send button reads "Send" (not "Stop").
|
||||||
|
- `test_loader_a11y_and_reduced_motion` — after a full turn: the loader element is `aria-hidden="true"` in the DOM (the `#send-status` live region remains the sole announcer — assert its post-done text follows the `SEND_STATUS` idle shape, i.e. not stuck on a mid-turn label); then, in a context with `reducedMotion: "reduce"` (Playwright context option), send a second turn and assert the loader is still visible mid-turn (the reduced-motion variant renders the static dots — the CSS rule, not the visibility, is what changes).
|
||||||
|
3. Run the suite in isolation — all three tests green.
|
||||||
|
4. **The full gate** (DB up; every command passes before the commit):
|
||||||
|
- `uv run pytest` — green.
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90%.
|
||||||
|
- Regressions in isolation (the feedback-state history this phase extends): `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`, `tests/e2e/test_stop_generation.py -v --no-cov`, `tests/e2e/test_big_read_progress.py -v --no-cov`, `tests/e2e/test_loading_feedback.py -v --no-cov`.
|
||||||
|
- `uv run ruff check . && uv run pyright` — clean.
|
||||||
|
- Scope check: `git diff --stat` shows only `frontend/index.html`, `frontend/assets/app.js`, `frontend/assets/styles.css`, `tests/**`, `.agents/phases/**` (this is a UI phase — NO `app/` changes; if the diff shows any, stop and fix the scope).
|
||||||
|
5. **The commit** (exactly one, `--no-gpg-sign`):
|
||||||
|
```bash
|
||||||
|
git add frontend/index.html frontend/assets/app.js frontend/assets/styles.css tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): never-frozen turn — re-expanding thinking block + the persistent in-turn loader"
|
||||||
|
```
|
||||||
|
6. Move the phase directory: `mv .agents/phases/todo/109_turn_progress_loader .agents/phases/complete/` (the pipeline gate does this on success — do it only after the commit, matching how prior phases recorded the move).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the owner's repro replayed deterministically (the marker's baked delays make every window assertion race-free); the a11y split (visual loader + `#send-status` announcer) + the reduced-motion variant pinned.
|
||||||
|
- Coverage: **>90%** on `app/` (unchanged by this UI phase — the gate is the regression check).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` green in isolation (DB up) — all three tests
|
||||||
|
- [ ] The reported repro is pinned: post-delta thinking re-opens the block with visible text while the loader stays visible; terminal states are clean (loader hidden, button "Send", status not stuck)
|
||||||
|
- [ ] The regression suites (`test_thinking_display`, `test_stop_generation`, `test_big_read_progress`, `test_loading_feedback`) green in isolation; `uv run pytest` green; coverage TOTAL >90%; ruff + pyright clean
|
||||||
|
- [ ] Exactly one new commit with the phase message, `--no-gpg-sign`; the diff scoped to `frontend/` + `tests/` + `.agents/phases/`; `git status` clean afterwards
|
||||||
|
- [ ] Phase dir at `.agents/phases/complete/109_turn_progress_loader/`
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Phase 106 — Fix SSE chat stream DB connection pinning (SEC-14-04)
|
||||||
|
|
||||||
|
**Source:** `.agents/VULNS.md` — SEC-14-04 (Medium, open): "In-flight SSE chat stream pins a DB connection for the whole turn → pool-exhaustion DoS (~15 streams, one token user)" (CWE-770/400).
|
||||||
|
**Story:** n/a (security remediation).
|
||||||
|
**Context:** `app/api/chat.py` uses `db: Session = Depends(get_db)` which holds one DB connection for the entire SSE stream lifetime (embedding → retrieval → agent loop with tool calls → query_log write). The agent loop (`app/rag/agent.py::run_agent`) receives the same session and uses it for every tool round (ls/read/grep DB lookups). With the default implicit pool of 5+10=15, a single slow user can exhaust all connections by opening 15+ SSE streams. The sync/upload/git_sources house pattern (`SessionLocal()` → do work → `close()`) is the proven short-lived session model already used throughout the codebase.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Eliminate the SSE-stream DB-connection pinning that causes pool-exhaustion DoS (SEC-14-04): (1) make pool parameters explicit via env vars, (2) refactor `run_agent` and the chat endpoint to use short-lived DB sessions per DB step instead of one long-lived session, and (3) add an optional concurrency cap so the pool is never saturated even if individual steps take time. After this phase, the app remains functional under concurrent load and the pool is never exhausted by slow streams.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `99_kb_tree_table_and_back_nav` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches `app/db.py`, `app/api/chat.py`, `app/rag/agent.py`, `app/config.py`, `app/schemas.py` — none of which phase 99's files reach; its suites must stay green unchanged).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### Pool configuration (A1, task 01)
|
||||||
|
- `app/config.py`: add `db_pool_size: int = Field(default=5)` and `db_pool_max_overflow: int = Field(default=10)`. Add `db_pool_recycle: int = Field(default=3600)` (one hour, prevents stale connections). The `__post_init__` validator ensures `db_pool_size >= 1` and `db_pool_max_overflow >= 0`.
|
||||||
|
- `app/db.py`: pass `pool_size=settings.db_pool_size`, `max_overflow=settings.db_pool_max_overflow`, `pool_recycle=settings.db_pool_recycle` to `create_engine()`. The existing `pool_pre_ping=True` and `future=True` stay.
|
||||||
|
- Env vars: `BOR_DB_POOL_SIZE`, `BOR_DB_POOL_MAX_OVERFLOW`, `BOR_DB_POOL_RECYCLE`.
|
||||||
|
|
||||||
|
### Short-lived sessions in run_agent (A2, task 02)
|
||||||
|
- **The core problem:** `run_agent(llm, db, ...)` receives a `Session` from the chat endpoint and holds it for the entire agent loop (all rounds, all tool calls). Every tool call (`ls`, `read`, `grep`) executes SQL on this session, and the session is never closed until the stream ends.
|
||||||
|
- **The fix:** Change `run_agent` to accept a session factory (`Callable[[], Session]`) instead of a `Session`. For each DB operation (tool execution), create a short-lived session, execute the operation, close the session. The LLM chat history (messages list) is already in-memory and needs no DB.
|
||||||
|
- **Signature change:** `run_agent(llm, db_factory, ..., *, max_rounds, ...)` where `db_factory = lambda: SessionLocal()`. All internal DB accessors (`ls_top`, `ls_folder`, `find_document`, `all_documents`, `find_path_candidates`) already take `Session` — they are called inside the factory closure.
|
||||||
|
- **The chat endpoint:** remove `db: Session = Depends(get_db)`. Create `db_factory = lambda: SessionLocal()` at the top of the `stream()` generator. Pass `db_factory` to `run_agent`. For the retrieval steps (`load_steering_notes`, `load_kb_overview`, `retrieve`), create short-lived sessions inline (the existing pattern).
|
||||||
|
- **query_log write:** create a short-lived session, add + commit + close (the existing pattern in sync).
|
||||||
|
- **AgentHolder:** unchanged — it only tracks in-memory state (`read_docs`, `tool_calls`, `scaffold_stripped`).
|
||||||
|
|
||||||
|
### Chat concurrency cap (A3, task 03)
|
||||||
|
- `app/api/chat.py`: a module-level `asyncio.Semaphore` initialized to `max(1, settings.chat_max_concurrent)` (default 10). The `chat` endpoint acquires the semaphore before starting the stream and releases it when the stream ends (in a `finally` block). If the semaphore is exhausted, return 503 "Too many concurrent chat turns — try again."
|
||||||
|
- `app/config.py`: add `chat_max_concurrent: int = Field(default=10)`. Validator: `>= 1`.
|
||||||
|
- Env var: `BOR_CHAT_MAX_CONCURRENT`.
|
||||||
|
|
||||||
|
### NOT touched
|
||||||
|
- `app/rag/llm.py` — LLM client unchanged.
|
||||||
|
- `app/rag/retriever.py` — retrieval functions unchanged (they take `Session` as before).
|
||||||
|
- `app/rag/prompts.py` — prompt building unchanged.
|
||||||
|
- `app/rag/scaffolding.py` — scaffolding filter unchanged.
|
||||||
|
- `app/rag/suggestions.py` — suggestions unchanged.
|
||||||
|
- `app/models.py` — no model changes.
|
||||||
|
- `alembic/` — no migrations needed.
|
||||||
|
- Frontend — no UI changes.
|
||||||
|
- Completed phase E2E suites — behavior is preserved (same API contract, same SSE frames).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_pool_config_db.py` — explicit pool kwargs in `app/db.py` via `app/config.py` settings + tests.
|
||||||
|
2. `02_short_lived_sessions_agent.py` — refactor `run_agent` to use a session factory + short-lived sessions per DB step + chat endpoint DB refactor + integration tests.
|
||||||
|
3. `03_chat_concurrency_cap.py` — `asyncio.Semaphore` concurrency cap on `/api/chat` + config + integration tests + E2E.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_db_pool_config.py` (new, task 01): `create_engine` receives the correct pool kwargs from settings; `SessionLocal` is still callable. `tests/unit/test_agent_short_lived_sessions.py` (new, task 02): `run_agent` with a mock session factory — verify that each tool call creates a new session (the factory is called per-DB-operation), sessions are closed after use, and the agent loop completes correctly.
|
||||||
|
- Integration — `tests/integration/test_chat_db_sessions.py` (new, task 02): end-to-end chat turn (both deflected and grounded paths) — verify DB sessions are created and closed per step (not held across the stream); a grounded turn with tool calls uses separate sessions per tool round; query_log is written correctly. `tests/integration/test_chat_concurrency.py` (new, task 03): concurrent chat requests — verify the semaphore limits concurrent turns; excess requests get 503; released slots are reused.
|
||||||
|
- E2E (mandatory, A16) — `tests/e2e/test_chat_db_pool.py` (task 03), run in isolation with the DB up: `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov`.
|
||||||
|
- Regression: all existing `test_chat*.py` and `test_agent*.py` suites stay green (behavior preserved).
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `app/db.py::create_engine` receives explicit `pool_size`, `max_overflow`, `pool_recycle` from settings; default values match the previous implicit behavior (5+10).
|
||||||
|
- [ ] `run_agent` accepts a session factory (not a `Session`) and creates short-lived sessions for each DB operation; the chat endpoint creates `db_factory = lambda: SessionLocal()` and passes it.
|
||||||
|
- [ ] A grounded chat turn with tool calls: each tool round uses a separate DB session that is closed after the tool result is produced — no session is held across rounds.
|
||||||
|
- [ ] A concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent `/api/chat` turns; excess requests get 503.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` green in isolation (DB up); regression suites green; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — pool defaults match previous implicit behavior (owner-confirmed 2026-09-14).** `pool_size=5`, `max_overflow=10`, `pool_recycle=3600`. These are the SQLAlchemy defaults (5+10) plus a 1-hour recycle to prevent stale connections. The operator can override via env vars.
|
||||||
|
- **A2 — short-lived sessions per DB step, not per round (owner-confirmed 2026-09-14).** Each individual DB operation (one tool call, one retrieval query, one steering-notes load) gets its own session. This is the sync/upload house pattern already used throughout the codebase.
|
||||||
|
- **A3 — concurrency cap is configurable, defaults to 10 (owner-confirmed 2026-09-14).** The cap prevents pool exhaustion even if individual steps take longer than expected. The operator can increase it via `BOR_CHAT_MAX_CONCURRENT`.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): eliminate SSE chat stream DB connection pinning — short-lived sessions, explicit pool config, concurrency cap (SEC-14-04)"
|
||||||
|
```
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Task 01 — Explicit pool kwargs in `app/db.py` via settings
|
||||||
|
|
||||||
|
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): pool-exhaustion DoS via SSE stream DB connection pinning.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the database connection pool parameters explicit and configurable via environment variables, replacing the implicit SQLAlchemy defaults. This is a safe, additive change — no behavior change, just explicit configuration with the same defaults.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add three new settings fields to the `Settings` dataclass (after the existing DB-related fields, near `database_url`):
|
||||||
|
```python
|
||||||
|
#: Connection pool size for the primary Postgres engine (SEC-14-04).
|
||||||
|
#: Default 5 — matches SQLAlchemy's built-in default.
|
||||||
|
db_pool_size: int = Field(default=5)
|
||||||
|
#: Maximum overflow connections beyond pool_size (SEC-14-04).
|
||||||
|
#: Default 10 — matches SQLAlchemy's built-in default.
|
||||||
|
db_pool_max_overflow: int = Field(default=10)
|
||||||
|
#: Seconds before a pooled connection is recycled (SEC-14-04).
|
||||||
|
#: Default 3600 (1 hour) — prevents stale connections.
|
||||||
|
db_pool_recycle: int = Field(default=3600)
|
||||||
|
```
|
||||||
|
Add a `__post_init__` validator (or use Field validators) to ensure `db_pool_size >= 1` and `db_pool_max_overflow >= 0`. Raise `ValueError` with a descriptive message if violated.
|
||||||
|
|
||||||
|
2. `app/db.py` — update the `create_engine()` call to pass the pool kwargs:
|
||||||
|
```python
|
||||||
|
settings = get_settings()
|
||||||
|
engine = create_engine(
|
||||||
|
settings.database_url,
|
||||||
|
pool_pre_ping=True,
|
||||||
|
future=True,
|
||||||
|
pool_size=settings.db_pool_size,
|
||||||
|
max_overflow=settings.db_pool_max_overflow,
|
||||||
|
pool_recycle=settings.db_pool_recycle,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
(The `get_settings()` call was already happening indirectly via `get_settings().database_url`; make it explicit by assigning to a variable first.)
|
||||||
|
|
||||||
|
3. `app/config.py` — update the `.env.example` documentation (add the three new `BOR_DB_POOL_*` vars with their defaults and a comment about SEC-14-04).
|
||||||
|
|
||||||
|
4. Tests — `tests/unit/test_db_pool_config.py` (NEW):
|
||||||
|
- Default values: `Settings().db_pool_size == 5`, `db_pool_max_overflow == 10`, `db_pool_recycle == 3600`.
|
||||||
|
- Custom values: `Settings(db_pool_size=10, db_pool_max_overflow=20, db_pool_recycle=1800)` round-trips correctly.
|
||||||
|
- Validator: `db_pool_size=0` raises `ValueError`; `db_pool_max_overflow=-1` raises `ValueError`.
|
||||||
|
- Engine kwargs: `create_engine()` is called with the correct pool parameters (verify by inspecting the engine's pool configuration or by mocking `create_engine` and checking the call args).
|
||||||
|
|
||||||
|
5. Run `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov && uv run pytest tests/unit/ -q` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_db_pool_config.py` — defaults, custom values, validators, engine kwargs.
|
||||||
|
- Coverage: **>90%** on new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `app/config.py` has `db_pool_size`, `db_pool_max_overflow`, `db_pool_recycle` with correct defaults and validators
|
||||||
|
- [ ] `app/db.py::create_engine` receives explicit pool kwargs
|
||||||
|
- [ ] `tests/unit/test_db_pool_config.py` passes (defaults, custom values, validators)
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] no behavior change in existing tests
|
||||||
+90
@@ -0,0 +1,90 @@
|
|||||||
|
# Task 02 — Refactor `run_agent` and chat endpoint to short-lived sessions
|
||||||
|
|
||||||
|
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): SSE stream pins DB connection for the whole turn.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Replace the long-lived DB session (held via `Depends(get_db)` across the entire SSE stream) with short-lived sessions per DB operation. This eliminates the pool-exhaustion vulnerability by ensuring no DB connection is held longer than a single SQL operation.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — refactor `run_agent` to accept a session factory instead of a `Session`:
|
||||||
|
- Change the signature: replace `db: Session` with `db_factory: Callable[[], Session]`.
|
||||||
|
- The `db_factory` is a callable that returns a new session (e.g., `lambda: SessionLocal()`).
|
||||||
|
- In the agent loop, for each tool call that needs DB access, create a short-lived session:
|
||||||
|
```python
|
||||||
|
with db_factory() as tool_db:
|
||||||
|
result = execute_tool_call(tool_name, arguments, tool_db, ...)
|
||||||
|
```
|
||||||
|
- The `execute_tool_call` (or the inline tool dispatch) receives `tool_db` (a short-lived session), executes the tool's DB operations, and returns the result. The session is closed when the `with` block exits.
|
||||||
|
- The agent loop's message history (messages list) is in-memory and needs no DB — unchanged.
|
||||||
|
- `AgentHolder` is unchanged — it only tracks in-memory state.
|
||||||
|
- Update the module docstring to reflect the new signature.
|
||||||
|
|
||||||
|
2. `app/api/chat.py` — refactor the `chat` endpoint:
|
||||||
|
- Remove `db: Session = Depends(get_db)` from the function signature.
|
||||||
|
- Inside the `stream()` generator, create the session factory:
|
||||||
|
```python
|
||||||
|
from app.db import SessionLocal
|
||||||
|
db_factory = lambda: SessionLocal()
|
||||||
|
```
|
||||||
|
- For retrieval steps (`load_steering_notes`, `load_kb_overview`, `retrieve`), replace the direct `db` usage with short-lived sessions:
|
||||||
|
```python
|
||||||
|
with SessionLocal() as step_db:
|
||||||
|
steering_notes = load_steering_notes(step_db)
|
||||||
|
with SessionLocal() as step_db:
|
||||||
|
kb_overview = load_kb_overview(step_db)
|
||||||
|
with SessionLocal() as step_db:
|
||||||
|
chunks = retrieve(step_db, request.message, question_vec)
|
||||||
|
```
|
||||||
|
(Note: these can be separate sessions because they are independent reads. If they need to be in the same transaction, use one session — but they are all reads, so separate is fine and safer.)
|
||||||
|
- Pass `db_factory` to `run_agent` instead of `db`.
|
||||||
|
- For the `query_log` write (step 4), use a short-lived session:
|
||||||
|
```python
|
||||||
|
with SessionLocal() as log_db:
|
||||||
|
log_db.add(QueryLog(...))
|
||||||
|
log_db.commit()
|
||||||
|
```
|
||||||
|
- Update the module docstring to reflect the short-lived session pattern.
|
||||||
|
|
||||||
|
3. `app/rag/agent.py` — update all internal DB accessor calls inside the tool dispatch to use the session passed from the caller (which is now a short-lived session, not the long-lived one):
|
||||||
|
- The tool dispatch (inline in `run_agent` or in helper functions) receives the tool name and arguments, creates a session via `db_factory()`, calls the accessor, and closes the session.
|
||||||
|
- Example pattern for `read` tool:
|
||||||
|
```python
|
||||||
|
if tool_name == "read":
|
||||||
|
with db_factory() as tool_db:
|
||||||
|
doc = find_document(tool_db, source, path)
|
||||||
|
if doc:
|
||||||
|
result = doc.content[:settings.read_max_chars]
|
||||||
|
holder.read_docs.append(doc)
|
||||||
|
else:
|
||||||
|
result = _no_document_refusal(tool_db, combined)
|
||||||
|
```
|
||||||
|
- Same pattern for `ls` and `grep` tools.
|
||||||
|
- For `grep` on the whole KB (unscoped), use `all_documents(db_factory())` — one short-lived session for the bulk read.
|
||||||
|
|
||||||
|
4. Tests — `tests/unit/test_agent_short_lived_sessions.py` (NEW):
|
||||||
|
- Mock `db_factory` to track calls: verify that `db_factory()` is called for each tool execution (not just once at the start).
|
||||||
|
- Verify that sessions returned by `db_factory` are closed after use (use a mock that tracks `close()` calls).
|
||||||
|
- Verify that the agent loop completes correctly with a mock LLM and a mock DB factory.
|
||||||
|
- Test the deflected path (no tools, no DB factory usage beyond retrieval).
|
||||||
|
|
||||||
|
5. Tests — `tests/integration/test_chat_db_sessions.py` (NEW):
|
||||||
|
- Deflected turn: verify that retrieval uses short-lived sessions (steering notes, kb overview, retrieve each get their own session).
|
||||||
|
- Grounded turn with tool calls: verify that each tool round uses a separate session; sessions are closed after each tool result.
|
||||||
|
- Query log write: verify the query_log row is created correctly with a short-lived session.
|
||||||
|
- DB failure mid-stream: verify the error path works correctly with short-lived sessions.
|
||||||
|
|
||||||
|
6. Run `uv run pytest tests/unit/test_agent_short_lived_sessions.py tests/integration/test_chat_db_sessions.py -v --no-cov` — green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_agent_short_lived_sessions.py` — verify db_factory is called per-operation, sessions are closed, agent loop completes.
|
||||||
|
- Integration: `tests/integration/test_chat_db_sessions.py` — deflected and grounded turns use short-lived sessions; query_log writes correctly.
|
||||||
|
- Coverage: **>90%** on new/modified code.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `run_agent` accepts `db_factory: Callable[[], Session]` instead of `db: Session`
|
||||||
|
- [ ] Each tool call in the agent loop creates its own short-lived session via `db_factory()` and closes it after the tool result is produced
|
||||||
|
- [ ] The chat endpoint no longer uses `Depends(get_db)`; retrieval steps and query_log write use short-lived sessions
|
||||||
|
- [ ] `tests/unit/test_agent_short_lived_sessions.py` passes
|
||||||
|
- [ ] `tests/integration/test_chat_db_sessions.py` passes
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
|
- [ ] no behavior change in existing tests
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
# Task 03 — Chat concurrency cap + E2E test suite
|
||||||
|
|
||||||
|
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): pool-exhaustion DoS via SSE stream DB connection pinning.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Add a configurable concurrency cap on `/api/chat` to prevent pool exhaustion even if individual DB steps take longer than expected. Also write the dedicated E2E test suite and run all regression gates.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add the concurrency cap setting:
|
||||||
|
```python
|
||||||
|
#: Maximum concurrent chat turns allowed (SEC-14-04).
|
||||||
|
#: Default 10 — prevents pool saturation from too many simultaneous streams.
|
||||||
|
chat_max_concurrent: int = Field(default=10)
|
||||||
|
```
|
||||||
|
Add a validator: `chat_max_concurrent >= 1`, raise `ValueError` otherwise.
|
||||||
|
|
||||||
|
2. `app/api/chat.py` — add the semaphore:
|
||||||
|
- Module-level import: `import asyncio`.
|
||||||
|
- Module-level variable (initialized lazily to avoid import-time side effects):
|
||||||
|
```python
|
||||||
|
_chat_semaphore: asyncio.Semaphore | None = None
|
||||||
|
|
||||||
|
def _get_chat_semaphore() -> asyncio.Semaphore:
|
||||||
|
global _chat_semaphore
|
||||||
|
if _chat_semaphore is None:
|
||||||
|
settings = get_settings()
|
||||||
|
_chat_semaphore = asyncio.Semaphore(max(1, settings.chat_max_concurrent))
|
||||||
|
return _chat_semaphore
|
||||||
|
```
|
||||||
|
- In the `chat` endpoint (before the `stream()` generator definition), acquire the semaphore:
|
||||||
|
```python
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
```
|
||||||
|
- Inside the `stream()` generator, wrap the entire body in a semaphore acquire/release:
|
||||||
|
```python
|
||||||
|
async def stream() -> AsyncIterator[str]:
|
||||||
|
await sem.acquire()
|
||||||
|
try:
|
||||||
|
# ... existing stream body ...
|
||||||
|
finally:
|
||||||
|
sem.release()
|
||||||
|
```
|
||||||
|
- If the semaphore cannot be acquired immediately (all slots taken), return 503 before entering the generator:
|
||||||
|
```python
|
||||||
|
# At the top of the chat endpoint, before defining stream():
|
||||||
|
try:
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
# We can't do a non-blocking acquire in a sync function, so use a different approach:
|
||||||
|
# Check current semaphore value vs max, or use a try/except pattern
|
||||||
|
except Exception:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
Actually, since `chat()` is an `async def`, we can do a non-blocking acquire:
|
||||||
|
```python
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(sem.acquire(), timeout=0.001) # non-blocking check
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
return JSONResponse(status_code=503, content={"detail": "Too many concurrent chat turns — try again."})
|
||||||
|
```
|
||||||
|
|
||||||
|
Wait — this is racy (another request could slip in between the check and the actual acquire). Better approach: always acquire (blocking), but check if we're at the limit before starting:
|
||||||
|
|
||||||
|
Actually, the cleanest approach for FastAPI async endpoints:
|
||||||
|
```python
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
|
||||||
|
async def stream() -> AsyncIterator[str]:
|
||||||
|
await sem.acquire()
|
||||||
|
try:
|
||||||
|
# ... existing stream body ...
|
||||||
|
finally:
|
||||||
|
sem.release()
|
||||||
|
```
|
||||||
|
|
||||||
|
And at the top of the `chat` function (before `stream()` is defined), add a pre-check:
|
||||||
|
```python
|
||||||
|
# Pre-check: if the semaphore is fully occupied, reject immediately
|
||||||
|
# (This is a best-effort check; the semaphore inside stream() is the real gate.)
|
||||||
|
if sem._value == 0:
|
||||||
|
return JSONResponse(status_code=503, content={"detail": "Too many concurrent chat turns — try again."})
|
||||||
|
```
|
||||||
|
|
||||||
|
Hmm, `_value` is implementation-specific. Let me use a cleaner approach: use a counter instead of a semaphore for the pre-check, or just always acquire and let it block (the stream will start when a slot opens). Actually, the simplest correct approach:
|
||||||
|
|
||||||
|
```python
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
|
||||||
|
async def stream() -> AsyncIterator[str]:
|
||||||
|
await sem.acquire()
|
||||||
|
try:
|
||||||
|
# ... existing stream body ...
|
||||||
|
finally:
|
||||||
|
sem.release()
|
||||||
|
|
||||||
|
return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is correct: the semaphore blocks until a slot is available. The pre-check for 503 is optional — if we want to reject immediately, we can use `sem.acquire(blocking=False)` in a try/except:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# At the top of chat(), after sem = _get_chat_semaphore():
|
||||||
|
try:
|
||||||
|
sem.acquire(blocking=False) # non-blocking
|
||||||
|
except asyncio.InvalidStateError:
|
||||||
|
# Semaphore not ready yet (shouldn't happen, but be safe)
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
# Successfully acquired — we need to release it because stream() will acquire again
|
||||||
|
sem.release()
|
||||||
|
# Now let stream() acquire it properly
|
||||||
|
```
|
||||||
|
|
||||||
|
Actually this is getting complicated. Let me use the simplest correct approach: always acquire in the stream, and add a separate counter for the pre-check:
|
||||||
|
|
||||||
|
```python
|
||||||
|
_chat_semaphore: asyncio.Semaphore | None = None
|
||||||
|
_chat_active: int = 0 # thread-safe counter for pre-check
|
||||||
|
|
||||||
|
def _get_chat_semaphore() -> asyncio.Semaphore:
|
||||||
|
...
|
||||||
|
|
||||||
|
@router.post("/chat")
|
||||||
|
async def chat(...):
|
||||||
|
sem = _get_chat_semaphore()
|
||||||
|
max_concurrent = get_settings().chat_max_concurrent
|
||||||
|
|
||||||
|
# Pre-check: reject if we're already at capacity
|
||||||
|
if _chat_active >= max_concurrent:
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=503,
|
||||||
|
content={"detail": "Too many concurrent chat turns — try again."}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def stream() -> AsyncIterator[str]:
|
||||||
|
nonlocal _chat_active
|
||||||
|
_chat_active += 1
|
||||||
|
try:
|
||||||
|
await sem.acquire()
|
||||||
|
try:
|
||||||
|
# ... existing stream body ...
|
||||||
|
finally:
|
||||||
|
sem.release()
|
||||||
|
finally:
|
||||||
|
_chat_active -= 1
|
||||||
|
|
||||||
|
return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is clean: `_chat_active` is the pre-check counter (fast path), `sem` is the actual gate (ensures we never exceed the limit even under race conditions). The counter is incremented before the semaphore acquire and decremented in the outer `finally`.
|
||||||
|
|
||||||
|
3. `app/config.py` — update `.env.example` to document `BOR_CHAT_MAX_CONCURRENT`.
|
||||||
|
|
||||||
|
4. Tests — `tests/integration/test_chat_concurrency.py` (NEW):
|
||||||
|
- Two concurrent requests: both succeed (within the cap).
|
||||||
|
- N+1 concurrent requests where N = `chat_max_concurrent`: N succeed, 1 gets 503.
|
||||||
|
- After the first N complete, the (N+1)th request succeeds (slot freed).
|
||||||
|
- Use `httpx.AsyncClient` with `anyio` or `pytest-asyncio` for concurrency.
|
||||||
|
|
||||||
|
5. Tests — `tests/e2e/test_chat_db_pool.py` (NEW, Playwright E2E):
|
||||||
|
- Open multiple concurrent browser pages, each sending a chat request.
|
||||||
|
- Verify that at most `chat_max_concurrent` requests are active simultaneously.
|
||||||
|
- Verify that excess requests get a 503 response (or wait and eventually succeed).
|
||||||
|
- Use the mock LLM (`E2E_REAL_LLM=1` not set) for determinism.
|
||||||
|
- Run in isolation: `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov`.
|
||||||
|
|
||||||
|
6. Regression — run the full existing test suite:
|
||||||
|
- `uv run pytest tests/unit/ -q` — green.
|
||||||
|
- `uv run pytest tests/integration/ -q` — green.
|
||||||
|
- `uv run pytest tests/e2e/test_chat*.py tests/e2e/test_agent*.py -v --no-cov` — green in isolation.
|
||||||
|
|
||||||
|
7. Run the full gate:
|
||||||
|
```bash
|
||||||
|
uv run pytest --cov=app --cov-report=term-missing
|
||||||
|
uv run ruff check . && uv run pyright
|
||||||
|
```
|
||||||
|
All green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `tests/integration/test_chat_concurrency.py` — concurrent requests, 503 on excess, slot reuse.
|
||||||
|
- E2E: `tests/e2e/test_chat_db_pool.py` — Playwright suite for concurrency cap verification.
|
||||||
|
- Regression: all existing chat and agent test suites stay green.
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `app/config.py` has `chat_max_concurrent` with default 10 and validator `>= 1`
|
||||||
|
- [ ] `/api/chat` rejects with 503 when `chat_max_concurrent` turns are active
|
||||||
|
- [ ] Released slots are reused — a waiting request starts when a slot frees up
|
||||||
|
- [ ] `tests/integration/test_chat_concurrency.py` passes
|
||||||
|
- [ ] `tests/e2e/test_chat_db_pool.py` passes in isolation (DB up)
|
||||||
|
- [ ] All regression suites green
|
||||||
|
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Phase 111 — Chat error banner: a real Retry button (TODO L1)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3–21 — "L1 — Chat error banner: 'Try again' is plain text, not a button (2026-09-15, brain-of-reese interactive test)"
|
||||||
|
**Story:** n/a (interactive-test follow-up fix; extends the phase-06 loading-feedback and phase-49/53 retry assets).
|
||||||
|
**Context:** `frontend/index.html:112` renders `#kb-banner` (chat page only — the Sources/document pages do not render it) with `#kb-banner-text`; `frontend/assets/app.js` — `showErrorBanner(detail)` (L2084) writes `${detail} ${ERROR_HINT}` as **plain text**; `ERROR_HINT` (L362) begins "Try again — …" so "Try again" reads as a clickable action but is not. `retryLastTurn(wrap)` (L2131) re-asks the last question in place (phase 49); `#stale-regenerate` (index.html:134, handler at app.js ~L1898) is the existing banner-button → `retryLastTurn` pattern.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Give the chat-view error banner a real Retry control after a failed/dropped turn: the banner shows a visible Retry button that re-runs the last question without re-typing (reusing the phase-49 redo-in-place and the stale-banner button pattern). The banner text stops mimicking a button, and every existing `showErrorBanner` caller (share, save-doc, stale chat) keeps working text-only — the Retry button appears only on failed chat turns.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `110_fix_sse_db_pool_exhaustion` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches `frontend/index.html`, `frontend/assets/app.js`, `frontend/assets/styles.css`, and frontend unit tests).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Banner button (task 01):** add `<button type="button" class="banner-retry" id="banner-retry" hidden>` inside `#kb-banner` (after `#kb-banner-text`), mirroring the `#stale-regenerate` markup (same refresh SVG + visible "Retry" label). Hidden by default; `showErrorBanner(detail, opts)` gains an optional second arg — when the caller flags the error as a **failed chat turn** (the UI state-machine path at app.js:1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`), the button is revealed and wired to `retryLastTurn(lastBrainWrap)` — the same last-brain-bubble targeting the `#stale-regenerate` handler uses. No retryable brain bubble → no button.
|
||||||
|
- **Copy:** `ERROR_HINT` becomes "If this persists, check the LLM is reachable." — the "Try again —" prefix moves to the button (the text must no longer read as a fake control).
|
||||||
|
- **Non-turn callers** (share failures L1857/L1870/L1892, save-doc L735/L742, stale L1620/L1950/L1962, …) pass no opts → text-only banner, no button — no behavior change for them.
|
||||||
|
- **CSS:** `.banner-retry` in `styles.css` reuses the `.stale-regenerate` pill look (same component family); the banner keeps `role="alert"`.
|
||||||
|
- **NOT touched:** the per-answer Retry pill (phase 49), the stale banner, `retryLastTurn` itself, the server, and the RAG/document views (no `#kb-banner` there — no split needed, ASSUMPTION in task 01).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_banner_retry_button.md` — banner Retry button markup + handler + hint-copy fix.
|
||||||
|
2. `02_banner_retry_tests.md` — frontend unit tests for the button's presence/handler + stale hint-copy assertions updated.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_banner_retry.py` (new, task 02) — house-style source assertions: `#banner-retry` exists in the `#kb-banner` markup (hidden by default, `type="button"`); `showErrorBanner` wires the click → `retryLastTurn`; the button is revealed only on the turn-error path; `clearErrorBanner` re-hides it; `ERROR_HINT` no longer starts with "Try again".
|
||||||
|
- E2E: no new file — the turn-error path is exercised by the existing `tests/e2e/test_llm_retry.py` and `tests/e2e/test_smoke.py` suites, which must stay green (no banner behavior change for non-turn callers).
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate; the frontend JS is pinned by the source-assertion unit tests — no app/ code changes in this phase).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] After a dropped/failed chat turn, `#kb-banner` shows a visible Retry button; clicking it re-runs the last question without re-typing.
|
||||||
|
- [ ] Share/save-doc/stale-chat errors show a text-only banner (no button) — unchanged.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — Retry = re-ask the last question in place via the existing `retryLastTurn` (owner-confirmed 2026-09-14, roadmap confirmation).** No new retry mechanism; the phase-49 redo-in-place is reused.
|
||||||
|
- **A2 — the button is offered only on the UI state-machine's turn-error path (a dropped/failed chat turn); all other banner callers stay text-only (owner-confirmed 2026-09-14).** Matches L1's acceptance ("after a dropped/failed turn …").
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(ui): give the chat error banner a real Retry button that re-asks the last question"
|
||||||
|
```
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Task 01 — Banner Retry button: markup, handler, hint-copy fix
|
||||||
|
|
||||||
|
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:3–18` — "L1 — Chat error banner: 'Try again' is plain text, not a button … `showErrorBanner()` (`frontend/assets/app.js`, `ERROR_HINT` ~line 362) renders `${detail} ${ERROR_HINT}` as **plain text** into `#kb-banner-text` … Suggested fix: give the chat-view error banner a real Retry control that re-asks the last question (the stale-chat banner already has the pattern: `#stale-regenerate` → `retryLastTurn`; the phase-49 Retry pill asset exists). Keep `#kb-banner` dual-use working for the RAG view, or split the two banners if the RAG view's banner has different recovery semantics."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The chat error banner gets a real Retry button (turn failures only) that re-asks the last question through the existing `retryLastTurn` redo-in-place; the hint copy stops mimicking the button.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/index.html` — inside `#kb-banner` (L112), add after `<span id="kb-banner-text">`:
|
||||||
|
```html
|
||||||
|
<button type="button" class="banner-retry" id="banner-retry" hidden>
|
||||||
|
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
|
||||||
|
<span>Retry</span>
|
||||||
|
</button>
|
||||||
|
```
|
||||||
|
(the same refresh SVG the `#stale-regenerate` button at L134 uses). Add a comment block: revealed only for failed chat turns (task 01 of this phase); hidden for every other banner caller.
|
||||||
|
2. `frontend/assets/app.js`:
|
||||||
|
- `ERROR_HINT` (L362): change to `"If this persists, check the LLM is reachable."` (the action moves to the button).
|
||||||
|
- `showErrorBanner(detail, opts = {})` (L2084): keep the single-arg behavior byte-identical; when `opts.retryable` is true AND a retryable last brain bubble exists (the same lastBrainWrap lookup the `#stale-regenerate` handler at ~L1898 uses), unhide `#banner-retry` and bind its click **once** to `() => retryLastTurn(lastBrainWrap)`; re-binding on every reveal must be guarded (one listener per button lifetime). `clearErrorBanner()` re-hides the button.
|
||||||
|
- The turn-error path (L1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`): pass `{ retryable: true }`. Every other caller (L735, L742, L1620, L1857, L1870, L1892, L1950, L1962, …) is left unchanged.
|
||||||
|
- If `retryLastTurn` would no-op (no retryable bubble), do not reveal the button — reveal only when a bubble exists.
|
||||||
|
3. `frontend/assets/styles.css` — `.banner-retry`: same pill treatment as `.stale-regenerate` (color, border, hover, `focus-visible` ring per the theme), laid out inline after the banner text (the `.kb-banner` flex row + gap already handles spacing).
|
||||||
|
4. ASSUMPTION: no banner split — `#kb-banner` exists only in `frontend/index.html` (the chat page); `document.html`/the Sources pages do not render it, so "keep dual-use" is trivially satisfied and the RAG view is untouched.
|
||||||
|
5. ASSUMPTION: the button is revealed only on the UI state-machine's turn-error path (a dropped/failed chat turn) — non-turn errors (share, save-doc, stale chat) stay text-only (locked A2).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_frontend_banner_retry.py` (added by task 02 — this task ships the code, task 02 ships the pin).
|
||||||
|
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `#kb-banner` contains `#banner-retry` (hidden by default); a failed chat turn reveals it; clicking re-asks the last question without re-typing.
|
||||||
|
- [ ] `ERROR_HINT` no longer contains "Try again".
|
||||||
|
- [ ] No non-turn call site passes `retryable` (grep the call sites).
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 02 — Unit tests for the banner Retry button
|
||||||
|
|
||||||
|
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:19–21` — "Acceptance: after a dropped/failed turn, the banner shows a visible Retry button that re-runs the last question without re-typing; unit test for the banner's button presence/handler in the frontend test suite."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Pin the banner contract in the frontend unit suite: the button's presence, its handler, its reveal condition, and the new hint copy.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_frontend_banner_retry.py` (new) — house-style source assertions (pattern: `tests/unit/test_frontend_feedback.py`):
|
||||||
|
- `frontend/index.html`: `#banner-retry` exists inside the `#kb-banner` block, `hidden` by default, `type="button"`, with a visible "Retry" label.
|
||||||
|
- `frontend/assets/app.js`: `showErrorBanner` binds the click handler to `retryLastTurn`; the `UI_STATE.error` turn path passes the retryable flag; `clearErrorBanner` re-hides the button.
|
||||||
|
- `ERROR_HINT` does not start with "Try again".
|
||||||
|
2. Grep the whole `tests/` tree for the old hint copy (`Try again — if this persists`) and update any stale assertion (loading-feedback and banner-related frontend tests).
|
||||||
|
3. Run the full unit + integration suite — no regressions (in particular `tests/unit/test_frontend_*.py`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the new file above (≥4 assertions across markup/handler/reveal/copy).
|
||||||
|
- Coverage: **>90%** on `app/` unchanged (no app/ code touched by this phase).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/unit/test_frontend_banner_retry.py -v` green.
|
||||||
|
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No test anywhere asserts the old "Try again — …" hint.
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Phase 112 — Honesty gate: FTS hits need cosine corroboration (TODO L2a/b + README nit)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L23–86 — "L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)" — part a (L36–44), part b (L46–55) + the stochastic follow-up (L64–75), the README nit (L77–81), acceptance (L83–86). Part c (chips on non-grounded answers, L57–62) is delivered by phase 113.
|
||||||
|
**Story:** the completed story 04 (`04_story_honest_deflection`) — this phase fixes the A8 gate eagerness the live deflection test exposed.
|
||||||
|
**Context:** `app/api/chat.py::plan_turn` (L272–277): HIGH when `best_cosine >= settings.relevance_threshold (0.62) or fts_hits > 0` — a single weak FTS token hit (suspected: the token "capital" inside a quest file) promoted a non-KB question into grounded mode and injected two irrelevant docs into the HIGH prompt, which then tempted the model into a parametric answer. The follow-up one-tap re-run of the identical question produced a **clean, textbook deflection** with the same docs injected → HONESTY GATE compliance is stochastic across runs; the deterministic lever is the gate (don't inject irrelevant docs), not prompt copy alone.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Make the A8 honesty gate deterministic against weak lexical hits: an FTS hit flips the turn to HIGH (grounded) only when the vector signal corroborates it (best cosine clears a new `lexical_support_floor`); below the floor the turn stays LOW (deflected) even with FTS matches. The owner-confirmed locked-prompt contract decision (treat the stochastic disclosed-general-knowledge behavior as acceptable — documented, prompt text byte-identical) and the stale README deflection copy are fixed in the same phase.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `111_chat_banner_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Gate rule (task 01):** HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`). LOW otherwise — including the fts>0 / cosine<floor case (the Mongolia case). `lexical_support_floor` is a new setting (default 0.35, env `BOR_LEXICAL_SUPPORT_FLOOR`), validated `0 <= floor <= relevance_threshold`. Rationale: a real lexical match on a genuinely similar doc (cosine ≥ floor) still grounds; a single weak token match with vector-unsupported docs no longer promotes. `plan_turn`'s docstring (the A8 bullets) and the A8 entry in `.agents/PLAN.md` are updated with the revision note (house precedent: "A8 revised 2026-08-21" — an owner-confirmed change to a LOCKED decision is recorded in the plan, not silently deviated from).
|
||||||
|
- **No schema/API change:** `query_log.top_score` / `fts_hits` are recorded exactly as today (observability unchanged); `TurnPlan` shape unchanged; the LOW branch (deflect prompt, weak-hit titles, derived suggestions) unchanged.
|
||||||
|
- **Prompt contract (task 03):** owner decision (iii) — the model's stochastic disclosed-general-knowledge answer (when misleading docs are injected) is documented as acceptable; the `app/rag/prompts.py` module docstring (the house location for locked-prompt revision history, e.g. the 2026-08-22 note) gains a dated entry; the prompt strings stay byte-identical (LOCKED verbatim); the README deflection section notes the behavior.
|
||||||
|
- **README nit (task 04):** the quoted deflection opening *"I haven't done anything like that"* was removed in the 2026-08-22 locked-prompt revision — the two README spots (L11, L575) are updated to describe the current behavior (admit no notes + 2–3 concrete alternative questions).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_gate_meaningful_fts.md` — the `plan_turn` gate fix + `lexical_support_floor` setting + the A8 plan revision note.
|
||||||
|
2. `02_gate_tests.md` — unit pins for the HIGH/LOW quadrants on weak single-token FTS hits + the E2E deflection check.
|
||||||
|
3. `03_prompt_contract_documentation.md` — document the owner decision (iii) (prompts.py docstring + README); prompt text unchanged.
|
||||||
|
4. `04_readme_deflection_copy.md` — README stale deflection copy (L11, L575) updated.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_chat_gate.py` (existing — extend) + a new quadrant file (task 02): cosine ≥ threshold → HIGH regardless of FTS; fts>0 + cosine ≥ floor → HIGH; **fts>0 + cosine < floor → LOW** (the new behavior, the Mongolia regression pin); no hits → LOW.
|
||||||
|
- E2E: `tests/e2e/test_honest_deflection.py` (existing — extend, task 02): a known-out-of-KB question (mock LLM, so the test pins the gate not the model) → deflected, no citation chips, 2–3 alternative questions. Run in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov`.
|
||||||
|
- Regression: `tests/unit/test_chat_gate.py`, `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` stay green.
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A weak single-token FTS hit with vector-unsupported docs (cosine < floor) → LOW/deflected (unit-pinned).
|
||||||
|
- [ ] A known-out-of-KB question produces no false citations and 2–3 concrete alternative questions (E2E).
|
||||||
|
- [ ] `app/rag/prompts.py` prompt strings byte-identical to pre-phase (test-pinned); README deflection copy matches current behavior.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — lever: cosine corroboration; `lexical_support_floor` default 0.35, env-tunable via `BOR_LEXICAL_SUPPORT_FLOOR` (owner-confirmed 2026-09-14, roadmap confirmation).** Of the TODO's three options (rank threshold / stopword-short-token exclusion / cosine corroboration), cosine corroboration is the deterministic one; the tests pin the decision logic, not the default value.
|
||||||
|
- **A2 — locked-prompt contract: option (iii) — treat the stochastic disclosed-general-knowledge behavior as acceptable and document it (owner-confirmed 2026-09-14, roadmap confirmation).** The prompt text is unchanged (LOCKED verbatim); the deterministic protection is the gate (A1). Options (i) tighten copy / (ii) amend the prompt via the plan remain open to a future owner decision.
|
||||||
|
- **A3 — the A8 entry in `.agents/PLAN.md` is amended in this phase with a dated revision note** (owner-confirmed 2026-09-14) — house precedent for recording owner-confirmed LOCKED-decision changes.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ .agents/ README.md && git commit --no-gpg-sign -m "fix(rag): require cosine-corroborated FTS hits before the honesty gate flips HIGH — document the disclosed-answer behavior, refresh README deflection copy"
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 01 — Gate: FTS hits must be cosine-corroborated
|
||||||
|
|
||||||
|
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:36–44` — "a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised 2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR `fts_hits > 0`. A single weak FTS token hit (suspected: the token 'capital' inside a quest file) promotes a non-KB question into grounded mode and injects two irrelevant top-docs into the HIGH prompt — which then tempts the model into a parametric answer instead of deflection. Consider: require FTS hits to be *meaningful* (e.g. rank threshold, stopword/short-token exclusion, or cosine corroboration) before flipping to HIGH; or run the LOW prompt when top-docs score below a usefulness bar." (+ the follow-up, L64–75: "the deterministic lever is the gate (don't inject irrelevant docs — part a), not prompt copy alone")
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An FTS hit flips `plan_turn` to HIGH only when the best cosine clears the new `lexical_support_floor`; a weak single-token hit with vector-unsupported docs stays LOW (deflected).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add `lexical_support_floor: float = Field(default=0.35)` (env `BOR_LEXICAL_SUPPORT_FLOOR`) next to `relevance_threshold` (L119), with validation `0 <= lexical_support_floor <= relevance_threshold` (mirror the file's existing validator style); document it in `.env.example`.
|
||||||
|
2. `app/api/chat.py::plan_turn` — replace the gate line (`if best_cosine >= settings.relevance_threshold or fts_hits > 0:`):
|
||||||
|
```python
|
||||||
|
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
|
||||||
|
if best_cosine >= settings.relevance_threshold or lexical_supported:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
Update the `plan_turn` docstring's gate section (the A8 bullets): HIGH when `best_cosine >= threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); LOW otherwise — a lexical-only hit without vector support deflects (A8 revised 2026-09-14).
|
||||||
|
3. `.agents/PLAN.md` — amend the A8 locked-decision entry with a dated revision note (house precedent: "A8 revised 2026-08-21"): "A8 revised 2026-09-14 (owner-confirmed, TODO L2a): an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`); lexical-only hits without vector support deflect."
|
||||||
|
4. `app/rag/prompts.py` — prompt strings NOT touched (LOCKED verbatim).
|
||||||
|
5. ASSUMPTION: the chosen lever is cosine corroboration (one of the TODO's three listed options); floor default 0.35 (≈ half the 0.62 threshold) is tunable via env against the live KB — the unit/E2E tests pin the decision logic, not the default value (locked A1).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: covered by task 02 (`tests/unit/test_chat_gate.py` extension) — this task's code must keep it green.
|
||||||
|
- Coverage: **>90%** on `app/` including the new setting and the modified gate line.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `plan_turn`: fts_hits>0 + best_cosine < floor → LOW (`TurnPlan.deflected` True, LOW prompt, suggestions derived from weak-hit titles).
|
||||||
|
- [ ] fts_hits>0 + floor <= best_cosine < threshold → HIGH (the new "corroborated lexical" path).
|
||||||
|
- [ ] best_cosine >= threshold → HIGH regardless of FTS (unchanged); no hits → LOW (unchanged).
|
||||||
|
- [ ] `.env.example` documents `BOR_LEXICAL_SUPPORT_FLOOR`; the config validation rejects floor > threshold.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — Pin the gate's HIGH/LOW decision on weak FTS hits
|
||||||
|
|
||||||
|
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:83–86` — "Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports results) produces no false citations, follows the HONESTY GATE (or the amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a weak single-token FTS hit."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Unit tests pin all four gate quadrants — especially the new fts>0 / cosine<floor → LOW quadrant (the Mongolia/"capital" regression); the E2E deflection story asserts a known-out-of-KB question deflects with no false citations and 2–3 alternatives.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_chat_gate.py` (existing — extend; reuse its chunk-building helpers) — the quadrants:
|
||||||
|
- cosine ≥ threshold, fts=0 → HIGH (unchanged).
|
||||||
|
- fts>0, cosine ≥ floor (e.g. 0.50 with default settings) → HIGH (corroborated lexical — the new path).
|
||||||
|
- **fts>0, cosine < floor** (e.g. one `fts_hit=True` chunk with cosine 0.10 — the "capital" case) → LOW: `deflected=True`, LOW prompt, `suggestions` non-empty, the weak docs do not enter a HIGH prompt.
|
||||||
|
- no chunks → LOW (unchanged).
|
||||||
|
- boundary: cosine exactly at the floor → HIGH (`>=`, mirroring the threshold's convention); config with floor > threshold → validation error.
|
||||||
|
2. `tests/e2e/test_honest_deflection.py` (existing — extend): a test asking a known-out-of-KB question (e.g. "What is the capital of Mongolia?" — LLM-known, absent from the fixture KB; the mock LLM's deflection path keeps the test deterministic on the gate, not on model compliance) → the done frame is `deflected: true`, `sources` is empty (no false citations), `suggestions` has 2–3 items.
|
||||||
|
3. Run the E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` (DB up).
|
||||||
|
4. Regression: `tests/unit/test_chat_gate.py` (all), `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the quadrant table (the new quadrant is the regression pin for TODO L2a).
|
||||||
|
- E2E: the deflection story extension.
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All quadrants green, including fts>0 + cosine<floor → LOW.
|
||||||
|
- [ ] E2E: known-out-of-KB question → deflected, zero source chips, 2–3 alternatives.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
+21
@@ -0,0 +1,21 @@
|
|||||||
|
# Task 03 — Document the disclosed-answer contract (owner decision iii)
|
||||||
|
|
||||||
|
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:46–55, 64–75` — "b) **Model violates the locked prompt** (Rule 1 'Answer ONLY from the provided document context', Rule 3 HONESTY GATE): with irrelevant docs injected it answered from general knowledge. The disclosure is better UX than silence, but the contract says no pretending to know + 2-3 concrete alternative questions — decide whether to (i) tighten the prompt copy … (ii) amend the locked prompt via the plan to explicitly permit disclosed general-knowledge answers, or (iii) treat the observed behavior as acceptable and document it. Owner decision required — the prompt text is locked verbatim (change through the plan, not here)." + the follow-up: "HONESTY GATE compliance is **stochastic** across runs (run 1: parametric answer; run 2: perfect deflection). Implication for the fix direction: the deterministic lever is the gate …, not prompt copy alone; a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Record the owner-confirmed decision (iii): the stochastic disclosed-general-knowledge answer (when misleading docs are injected) is acceptable and documented — the prompt text stays byte-identical (LOCKED verbatim); the deterministic protection is the task-01 gate fix.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/prompts.py` — module docstring (the house location for the locked-prompt revision history, e.g. the 2026-08-22 note): add a dated entry recording the 2026-09-15 interactive-test finding (parametric "Ulaanbaatar" answer with a disclosure, on the Mongolia question with two irrelevant docs injected; the clean textbook deflection on the identical one-tap re-run — stochastic compliance) and the owner decision (2026-09-14, roadmap confirmation): treat the disclosed general-knowledge answer as acceptable; the deterministic lever is the gate (A8 revised, task 01); options (i)/(ii) remain open to a future plan amendment. Prompt strings: byte-identical.
|
||||||
|
2. `README.md` — in the deflection paragraph (the one task 04 rewrites), one sentence: with a small local model, a rare turn may answer from general knowledge with an explicit disclosure when retrieval was borderline — the gate (phase 112) minimizes this; the disclosure is surfaced, never silent.
|
||||||
|
3. Prompt-lock pin: if no existing test byte-pins the prompt text, add a small `tests/unit/test_prompt_lock.py` asserting the HIGH/LOW prompt constants against pre-phase anchor strings (the executor extracts the pre-phase values when writing the test — e.g. exact prefix/suffix + total length, so any byte change fails).
|
||||||
|
4. ASSUMPTION: the documentation lives in the prompts.py docstring (the existing revision-history location) + the README — no new docs file.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the prompt-lock pin (work item 3).
|
||||||
|
- Coverage: n/a (docs-only change) — the suite stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `app/rag/prompts.py` docstring carries the dated decision entry; the prompt strings are byte-identical (test-pinned).
|
||||||
|
- [ ] The README deflection section notes the rare disclosed-answer behavior.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Task 04 — README: fix the stale deflection copy
|
||||||
|
|
||||||
|
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:77–81` — "Plus a docs nit: the README still promises the exact deflection copy *'I haven't done anything like that'* — the mandated deflection opening was removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py` module docstring). Update the README's 'If it doesn't have notes…' paragraph to match current behavior."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The README describes the current deflection behavior (admit no notes + 2–3 concrete alternative questions) instead of the removed mandated opening.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `README.md` L11 ("If it doesn't have notes for your question, it admits it: *'I haven't done…*") — rewrite the quoted copy to match the current locked-prompt behavior: it admits it has no notes on that and offers 2–3 concrete alternative questions about things it DOES have notes on. No exact-copy promise (the opening is no longer mandated).
|
||||||
|
2. `README.md` L575 ("**Honest deflection** (the amber *'I haven't done anything like that'*…") — same update; keep the amber-banner description accurate.
|
||||||
|
3. Grep the README for any other occurrence of the old quoted opening and update it.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Docs-only: no code tests. Grep `tests/` for `haven't done anything` — if any test asserts the old copy, update it to the new behavior.
|
||||||
|
- Coverage: n/a.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `grep -rn "haven't done anything" README.md` → no hits (the quoted opening is gone).
|
||||||
|
- [ ] The deflection paragraph matches current behavior (admit + 2–3 concrete alternatives).
|
||||||
|
- [ ] `uv run pytest` green.
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# Phase 113 — Source chip quality: usefulness bar + related-docs tier (TODO L5 + L2c)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L101–147 — "L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)" — root-cause chain (L125–129), suggested directions (L131–142), acceptance (L144–146) — plus `TODO.md` L57–62 (L2 part c): "c) **Misleading chips on non-grounded answers** (`chat.py:252` — `done.sources` = weak hits when deflected; by design, but visually a citation). … At minimum: never render them as answer citations."
|
||||||
|
**Story:** n/a (interactive-test follow-up fix; extends the phase-09 retrieval-quality and phase-05 chip assets).
|
||||||
|
**Context:** `top_n_docs = 2` (`app/config.py:114`) forces two docs into `plan.docs`; `done.sources` (`app/api/chat.py` ~L796, `ChatDoneEvent.sources`) carries every entry and `appendSources` (`frontend/assets/app.js:1369`) chips them all with identical visual weight — "the answer used this" vs "this also scored" is indistinguishable. Deflected turns carry weak hits in `sources` "by design" but they render as citations. Owner-confirmed direction: the deterministic **usefulness bar** (server-side) + the **visual split** (UI) — cite-gated chips (parsing the model's cited paths) are rejected for now (stochastic with a small model).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A document earns a citation slot only when its retrieval signal is vector-corroborated (or the agent explicitly read it via a tool call); everything else that scored is demoted to a clearly secondary "nearby docs" row that never reads as a citation. For a single-document question the turn shows exactly one citation chip; a deflected turn shows no citation chips at all.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `112_honesty_gate_weak_hits` (todo) — the gate fix stops weak hits being injected into the HIGH prompt; this phase stops weak docs earning a `done.sources` slot. Same workstream, ordered after the gate.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Usefulness bar (task 01):** new settings `source_usefulness_floor: float = 0.35` (env `BOR_SOURCE_USEFULNESS_FLOOR`, validated `0 <= floor <= relevance_threshold`, mirroring phase 112's floor) and `related_max_docs: int = 2` (env `BOR_RELATED_MAX_DOCS`, validated `>= 0`). In `plan_turn`, retrieval docs are tiered: **cited** = distinct parent docs (best fused-score order, at most `top_n_docs`) whose best hit-chunk **cosine** clears the floor; **related** = the next scored distinct docs (at most `related_max_docs`) that did not clear it. Agent-read docs (`holder.read_docs`, the phase-37 agent tool reads) always stay cited — the model read them via tool calls, so they were used by definition. `TurnPlan` gains `related_docs: list[Document] = []`; the tiering is a new `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]` in `app/rag/retriever.py`, with `select_documents` becoming a thin wrapper (legacy behavior byte-identical for existing callers/tests). `query_log.sources` is unchanged (it records retrieval, not citations — locked A3).
|
||||||
|
- **Done frame (task 01):** `ChatDoneEvent` (`app/schemas.py`) gains `related: list[SourceRef] = []` — additive; old clients ignore unknown fields (house contract, PLAN §4). Built from `plan.related_docs` with the same (source, path) dedupe against the cited list as `cited_docs` already does.
|
||||||
|
- **UI split (task 02):** `frontend/assets/app.js` — new `appendRelated(wrap, related)` renders a `.msg-meta.related-docs` row under the bubble (only when `related` is non-empty): a small de-emphasized label "Nearby docs, in case:" + one link per doc with the class `related-doc` (NOT `source-chip`) — same `documentUrl(...)` href and left-click → `openDocumentModal` behavior as citation chips, visually secondary (reduced opacity/size/dashed border via theme variables; link contrast ≥4.5:1, WCAG 2.1 AA). The done-frame handler (~L2421) also calls `appendRelated(wrap, ev.related)`; the restored-chat path (~L1561) likewise when the stored payload carries `related` (pre-phase chats don't — graceful). Deflected turns: `ev.sources` is empty (the server change) → no chips; the weak hits arrive in `ev.related` → the row only.
|
||||||
|
- **NOT touched:** the citation-chip component (`.source-chip` / `appendSources`) for the cited tier; the suggestion chips; `top_n_docs` (ceiling, not quota); the phase-37 agent-read dedupe; the Sources/RAG pages.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_usefulness_bar_sources.md` — retriever tiering + `TurnPlan.related_docs` + `ChatDoneEvent.related` + the two settings.
|
||||||
|
2. `02_secondary_related_docs_ui.md` — the related-docs row in the chat UI (app.js + styles.css); deflected turns show no chips.
|
||||||
|
3. `03_chip_filter_tests.md` — unit pins on the four observed live shapes + E2E chip-count assertions.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_retriever.py` (extend — the tiering table), `tests/unit/test_source_chip_quality.py` (new, task 03 — the four observed shapes), the done-frame schema tests (`related` defaults `[]`; old payloads without the field still parse).
|
||||||
|
- E2E: `tests/e2e/test_source_chip_quality.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov`) — a known single-source question → exactly one citation chip; a deflected question → zero `.source-chip` elements (the row, if any, is `.related-doc`, never `.source-chip`).
|
||||||
|
- Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py`, `test_sources_midstream_bug.py` stay green.
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] For a single-document question, the turn shows one citation chip (E2E).
|
||||||
|
- [ ] A weak 2nd doc renders only in the de-emphasized related row, never as a `.source-chip` (unit + E2E).
|
||||||
|
- [ ] A deflected turn renders zero citation chips (the weak hits, if any, live in the related row).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — direction: usefulness bar (server, deterministic) + visual split (UI); cite-gated chips rejected for now (owner-confirmed 2026-09-14, roadmap confirmation).**
|
||||||
|
- **A2 — citation slot = vector-corroborated retrieval doc (best hit-chunk cosine >= `source_usefulness_floor`, default 0.35, env-tunable) OR agent-read doc; `top_n_docs` stays a ceiling, not a quota (owner-confirmed 2026-09-14).**
|
||||||
|
- **A3 — `query_log.sources` keeps recording the full retrieval (observability); `done.sources` records only the cited tier (owner-confirmed 2026-09-14).**
|
||||||
|
- **A4 — the related tier is capped at 2 docs (`related_max_docs`, env `BOR_RELATED_MAX_DOCS` — owner-confirmed 2026-09-14).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): tier sources by a usefulness bar — weak hits become a de-emphasized related-docs row, never citation chips"
|
||||||
|
```
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 01 — Usefulness bar: tier done.sources into cited + related
|
||||||
|
|
||||||
|
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:125–129, 137–140` — "Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval to return two documents, `done.sources` carries both (`app/api/chat.py` L252), and the UI chips every entry without distinguishing 'the answer used this' from 'this also scored'." + "**Usefulness bar on the 2nd doc** — only include a document in `done.sources` when its fused/cosine score clears a threshold (a single weak FTS token hit should not earn a citation slot); `top_n_docs` stays a ceiling, not a quota." + `TODO.md:57–62` (L2 part c: "`done.sources` = weak hits when deflected; by design, but visually a citation … At minimum: never render them as answer citations.")
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Retrieval docs are tiered at the honesty gate: cited (vector-corroborated, ≤ `top_n_docs`) vs related (scored but under the floor, ≤ `related_max_docs`); the SSE done frame carries both; agent-read docs always stay cited.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add `source_usefulness_floor: float = Field(default=0.35)` (env `BOR_SOURCE_USEFULNESS_FLOOR`) and `related_max_docs: int = Field(default=2)` (env `BOR_RELATED_MAX_DOCS`); validators: `0 <= source_usefulness_floor <= relevance_threshold`, `related_max_docs >= 0`; `.env.example` entries.
|
||||||
|
2. `app/rag/retriever.py` — add `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]`:
|
||||||
|
- rank distinct parent docs by best fused score (the existing `select_documents` ordering), tracking each doc's best hit-chunk cosine;
|
||||||
|
- **cited** = the docs whose best-chunk cosine >= `floor`, up to `n` (ceiling — a single strong doc yields one cited doc);
|
||||||
|
- **related** = the next docs in rank order (any cosine, including 0.0 lexical-only), up to `related_cap`, never overlapping the cited list.
|
||||||
|
- `select_documents` becomes a wrapper: `cited, _ = select_documents_tiered(chunks, n, 0.0, 0)` — floor 0.0 + cap 0 keeps the legacy "any score, top-N" behavior byte-identical for existing callers/tests.
|
||||||
|
3. `app/api/chat.py` — `TurnPlan` gains `related_docs: list[Document] = []` (after `docs`); `plan_turn` calls `select_documents_tiered(chunks, settings.top_n_docs, settings.source_usefulness_floor, settings.related_max_docs)` → `docs`, `related_docs` (both the HIGH and LOW branches — deflected turns: the weak hits fall to related, cited is usually empty). The done-frame build (~L796): add `related=[SourceRef(source=d.source, path=d.path, title=d.title) for d in <plan.related_docs deduped against cited_docs by (source, path)>]` — the same dedupe pattern `cited_docs` already uses.
|
||||||
|
4. `app/schemas.py` — `ChatDoneEvent` gains `related: list[SourceRef] = []` (additive; docstring note: old clients ignore unknown fields, PLAN §4).
|
||||||
|
5. `query_log.sources` — unchanged (locked A3).
|
||||||
|
6. ASSUMPTION: the bar is on the **cosine** of the doc's best hit chunk, not the RRF fused score — the fused `score` is a rank key, not a similarity; a lexical-only hit has cosine 0.0 and is vector-unsupported by definition (consistent with the phase-112 gate; locked A2).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_retriever.py` (extend) — the tiering table: both clear → both cited; strong + weak → 1 cited + 1 related; both weak → 0 cited + 2 related; related cap respected; `select_documents` wrapper legacy behavior unchanged (existing tests stay green without edits).
|
||||||
|
- Unit: the existing done-frame/schema tests — `related` defaults to `[]`; a payload without the field still parses (back-compat).
|
||||||
|
- Coverage: **>90%** on `app/` including the new function.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `plan_turn` returns tiered docs; HIGH and LOW branches both populated correctly.
|
||||||
|
- [ ] The done frame carries `related` (≤ `related_max_docs`, deduped against cited); old frames (no field) parse.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Task 02 — UI: the related-docs row (never a citation chip)
|
||||||
|
|
||||||
|
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:141–142` — "**Visual split** — keep both, but render uncited/weak docs as a clearly secondary 'related docs' row, not citation chips." + `TODO.md:59–62` — "When the answer is disclosed general knowledge (or a deflected turn), the chips should be visually de-emphasized / labeled (e.g. 'nearby docs I have, in case'), or omitted when the brain says it didn't use them. At minimum: never render them as answer citations."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The chat bubble renders the cited tier exactly as today (`.source-chip` via `appendSources`) and the related tier as a clearly secondary labeled row; a deflected turn renders zero citation chips.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js`:
|
||||||
|
- new `appendRelated(wrap, related)` next to `appendSources` (L1369): early-return when empty; a `.msg-meta.related-docs` row (`role="list"`, `aria-label="Nearby docs, in case"`) + a small `<span class="related-docs-label">Nearby docs, in case:</span>` + one link per doc — class `related-doc` (NOT `source-chip`), the same `documentUrl(s.source, s.path, "/")` href and left-click → `openDocumentModal(s.source, s.path, link)` behavior, `title`/`aria-label` carrying the full path.
|
||||||
|
- the done-frame handler (~L2421, next to `appendSources(wrap, ev.sources)`): also `appendRelated(wrap, ev.related)`.
|
||||||
|
- the restored-chat path (~L1561): same, when the stored payload carries `related` (pre-phase saved chats don't — the row is simply absent, graceful).
|
||||||
|
- deflected turns: `ev.sources` is empty (the task-01 server change) → `appendSources` no-ops; the weak hits arrive in `ev.related` → row only.
|
||||||
|
2. `frontend/assets/styles.css` — `.related-docs` (muted row: smaller font, theme-variable color — link text contrast ≥4.5:1, WCAG 2.1 AA), `.related-doc` (dashed border, no hover elevation of the citation chips; `focus-visible` ring), `.related-docs-label` (small caps or muted small text); the row stacks below the citation `.msg-meta` row with the existing gap.
|
||||||
|
3. Frontend unit test (house source-assertion style — lives in `tests/unit/test_source_chip_quality.py`, extended by task 03): `appendRelated` exists and uses `related-doc` (assert `source-chip` is NOT in the `appendRelated` body); the row renders only when related is non-empty; the label text is present; the done handler calls `appendRelated`.
|
||||||
|
4. ASSUMPTION: label copy "Nearby docs, in case:" (the TODO's suggested wording, trimmed).
|
||||||
|
5. ASSUMPTION: pre-phase saved chats (stored payload without `related`) restore exactly as today — no related row.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the frontend source-assertion tests (task 03's file).
|
||||||
|
- E2E: pinned by task 03.
|
||||||
|
- Coverage: n/a (frontend) — the `app/` gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A related doc renders only as `.related-doc` in the labeled row — never as `.source-chip`.
|
||||||
|
- [ ] A deflected turn (mock) renders zero `.source-chip` elements under the bubble.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 03 — Pin the chip contract: the four observed shapes + E2E chip counts
|
||||||
|
|
||||||
|
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:110–123, 144–146` — the four observed live cases (1. "What is the capital of Mongolia?" → `Trooper_Nagraz.pl` + `Trooper_Begzei.pl`, both unrelated; 2. phase-gate question answered from `brain-of-reese/.agents/validate.sh` → second chip `ServMon/README.md` unused; 3. Trooper_Nagraz question answered from `Trooper_Nagraz.pl` → second chip `Trooper_Byzin.pl` uncited; 4. meta question about the conversation's own history → chips `app/api/suggestions.py` + `108_history_wire_check/00_phase.md`, neither used) + "Acceptance: for a single-document question, the turn shows one citation chip; a unit test pins `done.sources` filtering (or the chip renderer's cite-gate) on the four observed shapes; E2E asserts chip count for a known single-source question."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A unit test pins the cited/related tiering on the four live shapes; the E2E suite asserts the visible chip counts.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_source_chip_quality.py` (new) — model each of the four observed shapes as a `plan_turn`/done-frame fixture (retrieval chunks with controlled cosine/`fts_hit`/fused `score`) and assert the tiering:
|
||||||
|
1. **both docs weak** (cosine < floor, the Mongolia case) → `sources` empty, `related` ≤ 2.
|
||||||
|
2. **one strong + one weak** (the validate.sh case: `validate.sh` cosine ≥ floor, `ServMon/README.md` below) → exactly 1 in `sources`, the weak one in `related`.
|
||||||
|
3. **the Nagraz case** — same shape, different fixtures (`Trooper_Nagraz.pl` strong, `Trooper_Byzin.pl` weak) → 1 cited, 1 related.
|
||||||
|
4. **the meta/history question** (no doc clears the floor, the agent reads nothing) → `sources` empty, `related` ≤ 2; assert the frame shape that the UI renders as row-only (the rendering is pinned by task 02's source tests + the E2E).
|
||||||
|
5. **agent-read exemption**: a doc under the floor that is in `holder.read_docs` (agent tool read) still lands in `sources` (cited).
|
||||||
|
- plus the frontend source-assertion tests from task 02 work item 3 (same file).
|
||||||
|
2. `tests/e2e/test_source_chip_quality.py` (new; conftest/mock-LLM/fixture-KB pattern per `tests/e2e/test_retrieval_quality.py`):
|
||||||
|
- a known single-source question (a fixture-KB question whose answer comes from one doc) → the done bubble has **exactly one** `.source-chip`.
|
||||||
|
- a deflected question (known-out-of-KB) → **zero** `.source-chip`; if a `.related-docs` row exists, its links are `.related-doc`, never `.source-chip`.
|
||||||
|
- if the fixture KB cannot produce a strong+weak two-tier shape, say so in the test docstring and rely on the unit table for that shape.
|
||||||
|
3. Run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (DB up).
|
||||||
|
4. Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py` green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the four-shape table (the acceptance pin) + the agent-read exemption.
|
||||||
|
- E2E: the chip-count assertions (the acceptance pin).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The four observed shapes are unit-pinned (plus the agent-read exemption).
|
||||||
|
- [ ] E2E: single-source question → exactly one citation chip; deflected turn → zero.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Phase 114 — Embed question length: truncation + accurate error (TODO L6)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L149–181 — "L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading 'couldn't reach the embedding model' error (2026-09-15, brain-of-reese interactive test)"
|
||||||
|
**Story:** n/a (interactive-test follow-up fix; extends the phase-67 LLM-retry and phase-06 loading-feedback assets).
|
||||||
|
**Context:** The composer clamps at 4,000 chars; `app/api/chat.py:423` embeds the **full** question via `llm.embed_one`; aipi's litellm rejects the ~903-token input with **HTTP 500**: "input (903 tokens) is too large to process. increase the physical batch size (current batch size: 512)". The single-text path in `app/rag/llm.py` (`_embed_batch` → `_TooLarge`, ~L279–284) turns that into `EmbeddingError("a single …-char chunk exceeded the endpoint's per-request input token cap — lower BOR_CHUNK_TARGET_CHARS and re-import")` — an **import-oriented** message — and the chat endpoint's catch-all (~L428–444) maps EVERY `EmbeddingError` to "I couldn't reach the embedding model — please try again." Both diagnoses are wrong (reachability is fine; the chunker constant is irrelevant to a question). The chunker's own `HARD_MAX_CHARS = 1200` (`app/rag/chunker.py:51`, ~1024 tokens at ~1.4 chars/token) shows the question path never got the same treatment.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every legal question (≤ the UI clamp) is embeddable: the embed step gets a bounded prefix of the question (the chunker's 1200-char budget) while the full question still reaches the LLM prompt; and if the input is still too large (a smaller-cap model, a misconfiguration), the turn fails with an accurate "question too long" error — no false reachability diagnosis, no wasted retries — and the banner carries the phase-111 Retry button.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `111_chat_banner_retry` (todo) — L6's acceptance: "the L1 'Try again' button fix should also apply to this banner" — the too-long error flows through the same turn-error state machine, so the phase-111 Retry button is offered on it.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Truncation (task 01):** new setting `embed_question_max_chars: int = 1200` (env `BOR_EMBED_QUESTION_MAX_CHARS`, default = the chunker's `HARD_MAX_CHARS` budget, validated `> 0`). The chat embed step (chat.py:423) embeds `request.message[:settings.embed_question_max_chars]`; the LLM prompt build is unchanged (the full question still reaches the model). Questions shorter than the budget are byte-identical to today.
|
||||||
|
- **Error mapping (task 02):** `app/rag/llm.py` — new `EmbeddingInputTooLargeError(EmbeddingError)` subclass; the single-text `_TooLarge` branch of `_embed_batch` raises it (same message text — the importer path is byte-identical, it still catches `EmbeddingError`). The chat endpoint catches `EmbeddingInputTooLargeError` **before** `EmbeddingError` inside the phase-67 retry loop → no retry (a deterministic failure — locked A3) → terminal `ChatErrorEvent` with `detail="Question too long — trim it and re-ask."` and a new optional `hint` field: `hint="The app reached the embedding model fine — only the question length is the problem."` `ChatErrorEvent` gains `hint: str | None = None` (additive; PLAN §4 old-client ignore contract). The frontend's phase-111 reworked `showErrorBanner(detail, opts)` shows `opts.hint` when the frame carries one, else the default `ERROR_HINT`.
|
||||||
|
- **Retry:** the too-long frame flows through the turn-error state machine → the phase-111 banner Retry button is offered (re-asking is the user's call after trimming; the composer clamp still applies).
|
||||||
|
- **NOT touched:** the importer's embed path and its batch-halving `_TooLarge` behavior/error copy, the 4,000-char composer clamp (locked A2 — truncation, not a lower clamp), the reachability-failure retry semantics (phase 67 — byte-identical).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_embed_truncation.md` — the bounded-prefix embed + the setting.
|
||||||
|
2. `02_too_long_error_mapping.md` — `EmbeddingInputTooLargeError`, the chat-path mapping, `ChatErrorEvent.hint`, the frontend hint support.
|
||||||
|
3. `03_embed_length_tests.md` — the unit pins + the 4,000-char E2E.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — truncation (long → prefix embedded, LLM prompt carries the full text; short → byte-identical), error mapping (too-large failure → exact detail + hint, no retry frame, one attempt; transport failure → legacy reachability path with retries — the regression pin), the config validator.
|
||||||
|
- E2E: `tests/e2e/test_embed_question_length.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov`) — a 4,000-char question (the composer clamp) streams to done (mock LLM), no error banner.
|
||||||
|
- Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A 4,000-char question embeds (bounded prefix) and the turn succeeds; the LLM prompt carries the full question.
|
||||||
|
- [ ] A too-large embed failure (forced in a unit test) → the accurate "Question too long" frame + the reachability-fine hint; the banner offers the phase-111 Retry button.
|
||||||
|
- [ ] A reachability embed failure behaves byte-identically to pre-phase (retries + old copy).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — both fixes combined: 1200-char embed truncation (default = the chunker budget, env-tunable) + the precise too-long error mapping (owner-confirmed 2026-09-14, roadmap confirmation).**
|
||||||
|
- **A2 — the 4,000-char composer clamp stays (owner-confirmed 2026-09-14) — truncation, not a lower clamp.**
|
||||||
|
- **A3 — a too-large embed failure is NOT retried (deterministic failure) — it short-circuits the phase-67 retry loop (owner-confirmed 2026-09-14).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): embed a bounded question prefix (1200-char budget) and map the embed too-large failure to an accurate too-long error with a reachability-fine hint"
|
||||||
|
```
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Task 01 — Embed a bounded question prefix
|
||||||
|
|
||||||
|
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:151–164, 168–170` — "Repro: type/paste a question to the UI maximum (the composer clamps at 4,000 chars — char counter shows '4000/4000 — character limit') and send. Result, **100% reproducible**: the turn dies pre-token with the banner 'I couldn't reach the embedding model — please try again.' … a short question embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets **HTTP 500** from aipi … So the maximum legal question length exceeds the embed model's maximum legal input — and the chunker's own 1200-char cap (set to stay under the ~1024-token per-request cap) shows the question path never got the same treatment." + "**Truncate for embedding** — embed a bounded prefix of the question (e.g. the same 1200-char budget as chunks) while the full question still reaches the LLM prompt."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The chat embed step embeds at most `embed_question_max_chars` (default 1200 — the chunker's `HARD_MAX_CHARS` budget) of the question; the full question still reaches the LLM prompt.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — add `embed_question_max_chars: int = Field(default=1200)` (env `BOR_EMBED_QUESTION_MAX_CHARS`), validator `> 0`; `.env.example` entry with a comment citing the chunker rationale (`app/rag/chunker.py:30–51` — ~1.4 chars/token, stays under the ~1024-token per-request cap).
|
||||||
|
2. `app/api/chat.py` — the embed step (~L423): `question_vec = await llm.embed_one(request.message[: settings.embed_question_max_chars])`. Everything downstream is unchanged: retrieval runs on the prefix vector (intended — the prefix is the question's head); the LLM prompt build (`hist` + the full `request.message`) is untouched; the per-turn log line is untouched (`question=%r` logs the full text).
|
||||||
|
3. One-line comment at the call site: the prefix is bounded to the embed model's input cap (the chunker budget); the full question still reaches the LLM prompt (TODO L6).
|
||||||
|
4. ASSUMPTION: the budget is a setting (env-tunable), default 1200 — not a hard-coded constant — so a model with a larger/smaller cap is accommodated without a code change (locked A1).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — a question > the budget → `embed_one` receives exactly the prefix (mock LLM client); the LLM request messages carry the full question; a question ≤ the budget → byte-identical call.
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A 4,000-char question → `embed_one` called with the 1200-char prefix; the LLM request carries the full 4,000-char message.
|
||||||
|
- [ ] A short question → no behavior change (byte-identical call).
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 02 — Map the too-large embed failure to an accurate error
|
||||||
|
|
||||||
|
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:171–173` — "**Map the 500 to a precise error** — detect the 'too large' embed failure and surface 'question too long — trim it' (and fix the ERROR_HINT for this case: reachability is fine)." + `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A deterministic "input too large" embed failure surfaces as a precise "question too long" terminal error with a hint that reachability is fine — no false reachability diagnosis, no wasted retries; the error banner carries the phase-111 Retry button (the turn-error path).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/llm.py` — add `class EmbeddingInputTooLargeError(EmbeddingError)` (near `EmbeddingError`, L42), with a docstring: the single-text input exceeded the endpoint's token cap — deterministic, not a reachability failure. In the single-text `_TooLarge` branch of `_embed_batch` (~L279–284): raise `EmbeddingInputTooLargeError(<the existing import-oriented message>)` instead of plain `EmbeddingError` — the message text is **identical** (the importer path is byte-identical; it still catches `EmbeddingError`, and the subclass is a drop-in).
|
||||||
|
2. `app/schemas.py` — `ChatErrorEvent` gains `hint: str | None = None` (additive; docstring: the client shows the hint in place of its default reachability hint when present; old clients ignore the field — PLAN §4).
|
||||||
|
3. `app/api/chat.py` — the embed-failure handling (~L428–444, inside the phase-67 retry `while` loop): catch `EmbeddingInputTooLargeError` **before** `EmbeddingError` → do NOT restart (locked A3 — deterministic) → `settled = True`, log an error line (the existing format plus a `too-large` marker), and yield:
|
||||||
|
```python
|
||||||
|
ChatErrorEvent(
|
||||||
|
detail="Question too long — trim it and re-ask.",
|
||||||
|
hint="The app reached the embedding model fine — only the question length is the problem.",
|
||||||
|
).model_dump()
|
||||||
|
```
|
||||||
|
The existing `EmbeddingError` branch (reachability) is unchanged, including the retry semantics and the old copy.
|
||||||
|
4. `frontend/assets/app.js` — the phase-111 reworked `showErrorBanner(detail, opts)`: honor `opts.hint` — `bannerText.textContent = detail ? \`${detail} ${opts.hint ?? ERROR_HINT}\` : (opts.hint ?? ERROR_HINT)`. The SSE error-frame handler in the stream state machine (~L1281): pass `{ retryable: true, hint: ev.hint }` when the frame carries a hint.
|
||||||
|
5. ASSUMPTION: detail copy "Question too long — trim it and re-ask." (the TODO's "question too long — trim it", phrased as a banner sentence); hint copy as in work item 3.
|
||||||
|
6. ASSUMPTION: no retry on too-large (locked A3) — the phase-67 retry loop is for transient failures; a size failure is guaranteed to repeat.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — force the `_TooLarge` branch (a fake httpx response: HTTP 500 + a "too large to process" body) → the chat SSE stream yields exactly one error frame with the precise detail + hint and **no** retry frame; a transport failure (no "too large" signature) → the legacy reachability path with the retry loop and old copy (the regression pin).
|
||||||
|
- Coverage: **>90%** on `app/` including the new exception class and branch.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A too-large embed failure → the frame `{type: "error", detail: "Question too long — trim it and re-ask.", hint: "…fine…"}` — no "couldn't reach" copy, no retry frame.
|
||||||
|
- [ ] A reachability embed failure → byte-identical to pre-phase (retries + old copy).
|
||||||
|
- [ ] The frontend shows the frame's hint when present; the banner offers the phase-111 Retry button on this error.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Task 03 — Unit + E2E pins for the 4,000-char question
|
||||||
|
|
||||||
|
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The acceptance is pinned: a full-clamp (4,000-char) question succeeds end-to-end (truncated embed), and the too-long mapping is unit-pinned.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/unit/test_embed_question_length.py` (new):
|
||||||
|
- **truncation:** a mock `LLMClient` records the `embed_one` input; a 4,000-char question → exactly the prefix (default budget); the chat request to the LLM carries the full question; a 100-char question → byte-identical call.
|
||||||
|
- **error mapping:** a fake embed transport returning HTTP 500 + "too large to process" body for the input → the chat SSE stream yields exactly one error frame with the precise detail + the reachability-fine hint and no retry frame; short input + a 500 WITHOUT the "too large" signature → the legacy reachability path (retry frames + old copy) — the regression pin.
|
||||||
|
- **config:** the `embed_question_max_chars` default (1200) and validator.
|
||||||
|
2. `tests/e2e/test_embed_question_length.py` (new; the `tests/e2e/` conftest + mock-LLM pattern): type a 4,000-char question into the composer (the counter shows "4000/4000") → send → the turn streams to done (mock LLM) — no error banner.
|
||||||
|
3. Run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (DB up).
|
||||||
|
4. Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: as above (the acceptance pin: the error mapping).
|
||||||
|
- E2E: the 4,000-char success path (the acceptance pin: the truncated embed).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A 4,000-char question → a successful turn (E2E); the embed input was the prefix (unit).
|
||||||
|
- [ ] The too-long mapping is unit-pinned (exact frame, no retry).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Phase 115 — Doc drafts: Discard + DELETE route + title fix (TODO L7)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L183–209 — "L7 — 'Save as doc' has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test)"
|
||||||
|
**Story:** n/a (interactive-test follow-up fix; extends the phase-59/75 doc-draft and save-as-doc assets).
|
||||||
|
**Context:** "Save as doc" (`frontend/assets/app.js::saveAsDoc`, L720) POSTs `/api/doc-drafts` (201) → `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **"Push to docs branch"**. The drafts router (`app/api/doc_drafts.py`) has POST/GET/PUT/POST-push only — no DELETE, no TTL/pruning; a draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo. Side observation: the draft's default title comes from `defaultDocTitle()` (app.js:619) — the **last user record** in the conversation — and after a Retry redo-in-place (phase 49) the redone answer sits at the end, so its title came from an unrelated trailing question (a junk 4,000-char test question, not the question the answer answered). The body (full session transcript) is correct by design; only the title derivation mismatches.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
An orphaned doc draft can be discarded from the edit screen (a new admin-gated `DELETE /api/doc-drafts/{token}` + a Discard control), and the draft's default title is the question the saved answer actually answered (its paired user record) — fixing the retry-redo mismatch.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `114_embed_question_length` (todo) — pipeline predecessor (execution order) only; no code dependency.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **DELETE route (task 01):** `DELETE /api/doc-drafts/{token}` → **204**. The whole router already sits behind `require_admin` (phase 59, `dependencies=[Depends(require_admin)]` at L63) — the new route inherits it; the uuid4 token is the screen's credential (same trust model as GET/PUT/push). Unknown token → 404 via the existing `_get_draft_or_404` helper. No migration (a row delete); no push-side state (the git push happens only on push).
|
||||||
|
- **Discard UI (task 02):** the doc-edit screen — a "Discard draft" control next to "Push to docs branch" (secondary/danger treatment per the theme). `frontend/assets/doc-edit.js`: `confirm()` (destructive + irreversible — no undo exists), `DELETE /api/doc-drafts/${token}` (the same token the screen already uses for GET/PUT) → 204 → `location.assign("/")` (back to the chat page). Non-204 → the page's existing inline-error pattern, no navigation.
|
||||||
|
- **Title fix (task 03):** `defaultDocTitle(wrap)` — takes the saved brain bubble's wrap (the `.save-as-doc-btn`'s bubble); the title is the text of the user bubble **paired** with that brain bubble (the nearest preceding user message in the DOM conversation flow), falling back to the current last-user-record-in-`conversation` logic when no wrap is given or no paired user bubble is found (first-turn edge / DOM mismatch). The `DOC_TITLE_MAX` slice + "Note" fallback are unchanged.
|
||||||
|
- **NOT touched:** the push flow (byte-identical), the draft body (`buildSessionTranscript` — full session, correct by design), the drafts schema (no field change), no TTL/pruning (locked A1 — out of scope).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_delete_draft_route.md` — `DELETE /api/doc-drafts/{token}` (204 / 404 / admin-gated).
|
||||||
|
2. `02_discard_ui.md` — the Discard control on the doc-edit screen wired to the route.
|
||||||
|
3. `03_draft_title_fix.md` — the title from the answer's own question (the paired user record).
|
||||||
|
4. `04_draft_discard_tests.md` — the API + frontend + E2E pins.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: `tests/integration/test_doc_drafts_api.py` (existing — extend, task 04): DELETE removes the row (204; subsequent GET 404); unknown token → 404; the admin gate applies (same assertions the sibling routes use).
|
||||||
|
- Unit (frontend, house source-assertion style, task 04): the Discard control's presence/handler (confirm → DELETE → 204 → redirect; non-204 → inline error, no navigation); the `defaultDocTitle` pairing logic + the call site.
|
||||||
|
- E2E: `tests/e2e/test_save_doc_session.py` (existing — extend, task 04): the discard flow (save → edit screen → discard → confirm → back on the chat, draft gone); a save-as-doc after a Retry redo-in-place → the title matches the redone answer's own question. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov`.
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An orphaned draft can be discarded from the edit screen; the draft row is gone afterward (integration).
|
||||||
|
- [ ] The title of a save-as-doc after a retry redo matches the redone answer's own question (E2E).
|
||||||
|
- [ ] The push flow is byte-identical (regression green).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — scope: the Discard control + the DELETE route + the title fix; NO TTL/pruning (owner-confirmed 2026-09-14, roadmap confirmation — the discard covers the acceptance; pruning can be a future phase).**
|
||||||
|
- **A2 — the DELETE route is admin-gated by the router-level `require_admin` (phase 59) with the uuid4 token as the credential — same trust model as the sibling routes (owner-confirmed 2026-09-14).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(docs): discard doc drafts from the edit screen (DELETE /api/doc-drafts/{token}) + derive the draft title from the answer's own question"
|
||||||
|
```
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 01 — DELETE /api/doc-drafts/{token}
|
||||||
|
|
||||||
|
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:185–199, 201–204` — "Clicking 'Save as doc' on an answer POSTs `/api/doc-drafts` (201) and navigates to `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **'Push to docs branch'**. There is no Discard/cancel control, the drafts API has no DELETE route (`app/api/doc_drafts.py`: POST, GET, PUT, POST /push only) and no TTL/pruning. A draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo." + "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential) …"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A new admin-gated `DELETE /api/doc-drafts/{token}` removes a draft row (204) so an orphaned draft can be discarded instead of only pushed.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/api/doc_drafts.py` — add (after the `update_draft` PUT route, before the push route — or at the file's route-order idiom):
|
||||||
|
```python
|
||||||
|
@router.delete("/{token}", status_code=status.HTTP_204_NO_CONTENT)
|
||||||
|
def delete_draft(
|
||||||
|
token: uuid.UUID,
|
||||||
|
db: Session = Depends(get_db), # noqa: B008
|
||||||
|
) -> None:
|
||||||
|
"""Discard a draft (the edit screen's Discard control, phase 115).
|
||||||
|
|
||||||
|
Admin-gated like the whole router (phase 59); the uuid4 token is
|
||||||
|
the screen's credential — after a successful discard, GET/PUT/push
|
||||||
|
all 404.
|
||||||
|
"""
|
||||||
|
row = _get_draft_or_404(db, token)
|
||||||
|
db.delete(row)
|
||||||
|
db.commit()
|
||||||
|
```
|
||||||
|
Match the file's existing import style (check whether `status` from `fastapi` is already imported; use the file's idiom for the 204 response).
|
||||||
|
2. No migration, no schema change, no other route touched.
|
||||||
|
3. ASSUMPTION: 204 No Content (no body) — the token is a one-way credential; nothing else references the row (no FK targets, no push-side state).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: extended in task 04 (`tests/integration/test_doc_drafts_api.py`).
|
||||||
|
- Coverage: **>90%** on `app/` including the new route.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `DELETE /api/doc-drafts/{token}` → 204; the subsequent `GET` → 404.
|
||||||
|
- [ ] Unknown token → 404; the admin gate applies exactly like the sibling routes.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 02 — Discard control on the doc-edit screen
|
||||||
|
|
||||||
|
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:201–204` — "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential), or a TTL/prune for stale drafts; …" + `TODO.md:207–208` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test) …"
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The doc-edit screen offers a Discard control: confirm → DELETE → back to the chat.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. The doc-edit template (find the exact HTML file — the edit screen that renders "Push to docs branch"; `frontend/doc-edit.html` or the template it uses): add next to the push control:
|
||||||
|
```html
|
||||||
|
<button type="button" id="discard-draft" class="discard-draft"
|
||||||
|
title="Delete this draft permanently — this cannot be undone">Discard draft</button>
|
||||||
|
```
|
||||||
|
visually secondary to the push button (the theme's muted/danger treatment).
|
||||||
|
2. `frontend/assets/doc-edit.js` — a handler near the push handler:
|
||||||
|
- `if (!confirm("Discard this draft? This cannot be undone.")) return;`
|
||||||
|
- `fetch(\`/api/doc-drafts/${token}\`, { method: "DELETE" })` (the same token the screen already uses for GET/PUT, per its existing load code ~L136–143);
|
||||||
|
- 204 → `location.assign("/")` (back to the chat page);
|
||||||
|
- non-204 → the page's existing inline-error pattern (message + no navigation, no crash).
|
||||||
|
3. `frontend/assets/styles.css` — `.discard-draft`: the secondary/danger button style (contrast ≥4.5:1, `focus-visible` ring per the theme), laid out next to the push button.
|
||||||
|
4. Frontend unit tests (house source-assertion style, shipped in task 04's `tests/unit/test_frontend_doc_draft_discard.py`): the button's presence; `confirm(...)` before the DELETE; 204 → redirect; non-204 → inline error, no navigation.
|
||||||
|
5. ASSUMPTION: after a successful discard the user lands on the chat page (`/`) — the draft has no other home (no drafts list exists).
|
||||||
|
6. ASSUMPTION: a native `confirm()` is acceptable for this one destructive action (the codebase has no custom dialog asset — if the executor finds one in the theme, use it instead).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the frontend source-assertion tests (task 04).
|
||||||
|
- E2E: the discard flow (task 04, `tests/e2e/test_save_doc_session.py`).
|
||||||
|
- Coverage: n/a (frontend) — the `app/` gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The edit screen shows "Discard draft" next to the push control; the confirm dialog appears; a 204 returns the user to `/`.
|
||||||
|
- [ ] A failed DELETE (e.g. a 404 race) shows the inline error; no navigation; no crash.
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 03 — Draft title from the answer's own question
|
||||||
|
|
||||||
|
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:193–199` — "Side observation (edge case, same test): the draft's default title is the user record *immediately preceding* the saved answer. After a Retry redo-in-place (L-see app.js `retryLastTurn`), the redone answer sits at the end of the conversation, so its save-as-doc title came from an unrelated trailing question (the doc was titled with a junk 4,000-char test question, not the question the answer answered). Body is the full session transcript (correct by design); title derivation is just the mismatch." + `TODO.md:203–204` — "consider deriving the default title from the question the answer actually answered (its paired user record) rather than the preceding record."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The draft's default title is the user question paired with the saved brain bubble (its own question) — after a Retry redo-in-place the title matches the redone answer's question.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `defaultDocTitle()` (L619) → `defaultDocTitle(wrap)`:
|
||||||
|
- when *wrap* (the brain bubble) is given, walk the DOM conversation flow backwards from *wrap* to the **nearest user message bubble** (confirm the exact user-bubble class — the `.msg` variant used for user turns) and use its text;
|
||||||
|
- fall back to the current last-user-record-in-`conversation` logic when *wrap* is absent or no paired user bubble is found (first-turn edge / DOM mismatch);
|
||||||
|
- the `DOC_TITLE_MAX` slice + whitespace collapse + "Note" fallback are unchanged.
|
||||||
|
2. `saveAsDoc(btn)` (L720) — pass the bubble: `const title = defaultDocTitle(btn.closest(<the bubble class>))` — the save button lives in the bubble's meta (the `addSaveAsDocButton` code ~L704–710 shows the exact ancestor; use the same class `lastBrainWrap` uses at L577).
|
||||||
|
3. The `docSlug`/path logic is unchanged (it derives from the title).
|
||||||
|
4. Frontend unit tests (house source-assertion style, task 04's file): `defaultDocTitle` takes a wrap arg and prefers the paired user bubble over the last conversation record; the `saveAsDoc` call site passes the bubble ancestor.
|
||||||
|
5. ASSUMPTION: the pairing is DOM-structural (nearest preceding user bubble), not index-based — the redo-in-place reorders the DOM, and the structural pair IS the answer's question by construction.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the frontend source-assertion tests (task 04).
|
||||||
|
- E2E: the title-after-retry pin (task 04, `tests/e2e/test_save_doc_session.py`).
|
||||||
|
- Coverage: n/a (frontend) — the `app/` gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A save-as-doc on a normal (non-redone) answer → the same title as pre-phase (no regression).
|
||||||
|
- [ ] A save-as-doc after a Retry redo-in-place → the title is the redone answer's own question (E2E).
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 04 — API, frontend, and E2E pins for the discard + title
|
||||||
|
|
||||||
|
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:207–209` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test); title of a save-as-doc after a retry redo matches the redone answer's own question."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The acceptance is pinned at all three layers: the API (the row is gone), the frontend (the control + handler + the pairing), and the E2E (the full discard flow + the title after a retry redo).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/integration/test_doc_drafts_api.py` (existing — extend):
|
||||||
|
- DELETE an existing draft → 204; the subsequent GET → 404.
|
||||||
|
- DELETE an unknown token → 404.
|
||||||
|
- the admin gate: the same assertions the sibling routes use (mirror the existing test's auth fixtures — anonymous → 401 / non-admin → 403 per the router's `require_admin`).
|
||||||
|
2. `tests/unit/test_frontend_doc_draft_discard.py` (new, house source-assertion style):
|
||||||
|
- the doc-edit template carries `#discard-draft`; `doc-edit.js` calls `confirm(...)` before the `DELETE /api/doc-drafts/` fetch; 204 → the redirect; non-204 → the inline error, no navigation.
|
||||||
|
- `app.js`: `defaultDocTitle` takes a wrap arg and prefers the paired user bubble; the `saveAsDoc` call site passes the bubble ancestor.
|
||||||
|
3. `tests/e2e/test_save_doc_session.py` (existing — extend):
|
||||||
|
- **the discard flow:** ask (mock LLM) → save as doc → the edit screen → click Discard → confirm → back on the chat page; an API check (test client) confirms the draft row is gone (GET 404).
|
||||||
|
- **the title after a retry:** ask → the answer → Retry (redo-in-place) → save as doc on the redone answer → the edit screen's title field value == the redone question (not an unrelated trailing question).
|
||||||
|
4. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` (DB up).
|
||||||
|
5. Regression: `tests/integration/test_doc_drafts_api.py` (all), the existing `tests/e2e/test_save_doc_session.py` tests green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Integration: the DELETE contract (the acceptance: "the draft row is gone afterward (API test)").
|
||||||
|
- Unit: the frontend pins.
|
||||||
|
- E2E: the flow + the title (the acceptance).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All the acceptance pins green (API row gone; title after a retry).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Phase 116 — Document modal: themed code-block scrollbar (TODO L3)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L88–99 — "L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test)"
|
||||||
|
**Story:** `document-viewer.md` — the modal belongs to the document-viewer story (the same-page chip viewer, phase 26).
|
||||||
|
**Context:** The document modal (the almost-fullscreen chip viewer, `.doc-modal` — `frontend/assets/styles.css:4117+`) shows the raw content in a code block; long lines (e.g. a `quest::say(...)` line in a quest `.pl` file) overflow horizontally and reveal the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The modal's code block scrolls horizontally with a themed scrollbar (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), consistent with the dark theme; long lines stay unwrapped (code stays code) and scroll instead of clipping.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `115_doc_draft_discard` (todo) — pipeline predecessor (execution order) only; no code dependency (CSS + one E2E file).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Theme tokens (task 01):** two CSS custom properties in the theme's token block: `--scrollbar-thumb` (a muted theme color, ≥3:1 against the track — a scrollbar is a UI component, non-text contrast AA) and `--scrollbar-track` (near the code-block background). Scope: the modal's raw-content code element **only** (confirm the exact selector from `frontend/assets/document-modal.js` / the `.doc-modal` rules) — no global scrollbar restyle (out of scope; the TODO asks only about the modal).
|
||||||
|
- **Rules (task 01):** on the modal code block: `overflow-x: auto` (confirm present — add if the element relies on an ancestor), `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)` (Firefox), and the Chromium pair: `::-webkit-scrollbar { height: 8px }`, `::-webkit-scrollbar-track { background: var(--scrollbar-track) }`, `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
|
||||||
|
- **E2E (task 02):** the document-viewer story gains a check: open the modal on a document with a long line → the code block is horizontally scrollable (`scrollWidth > clientWidth`; a scroll action moves it) → a screenshot to `.agents/screenshots/` (house convention) as the visual record.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_themed_scrollbar.md` — the theme tokens + the scoped scrollbar rules.
|
||||||
|
2. `02_scrollbar_e2e_check.md` — the E2E scroll check + screenshot.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `tests/e2e/test_document_viewer.py` (existing — extend, task 02); run in isolation: `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov`.
|
||||||
|
- CSS: no unit layer for CSS — the E2E check + screenshot are the gate; the dark-theme suite (`tests/e2e/test_dark_tech_theme.py`) must stay green (no token collision).
|
||||||
|
- Coverage: n/a (no app/ code change) — the validate.sh gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The modal's code block scrolls horizontally; the scrollbar is themed — the screenshot in `.agents/screenshots/` shows no native light-gray bar.
|
||||||
|
- [ ] No global scrollbar change (the new selector is scoped under `.doc-modal` — grep); the other pages are visually unchanged (regression suites green).
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — keep horizontal scroll (NO line-wrap) for the code content (owner-confirmed 2026-09-14, roadmap confirmation — the TODO's "consider wrapping long lines per content type" is rejected for code: wrapping breaks code readability; the themed scrollbar is the fix).**
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "style(ui): theme the document modal's code-block horizontal scrollbar (scrollbar-color + webkit pseudos)"
|
||||||
|
```
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 01 — Themed scrollbar for the modal code block
|
||||||
|
|
||||||
|
**Phase:** `116_modal_scrollbar_theme` · **Source:** `TODO.md:90–96` — "In the document modal (the almost-fullscreen chip viewer), the raw-content code block overflows horizontally and reveals the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge (observed on a quest `.pl` file whose `quest::say(...)` line exceeds the modal width). Style the scrollbar to match the theme (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), or consider wrapping long lines per content type."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The modal's raw-content code block gets a themed horizontal scrollbar (both engine families), scoped to the modal.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — find the modal code block's exact selector (the `.doc-modal` raw-content `<pre>`/code element — how `frontend/assets/document-modal.js` renders the content; the `.doc-modal-panel` rules start ~L4135):
|
||||||
|
- ensure `overflow-x: auto` on the scrolling element (add it if the element relies on an ancestor for the overflow);
|
||||||
|
- add the scoped rules: `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)`, and `::-webkit-scrollbar { height: 8px }` / `::-webkit-scrollbar-track { background: var(--scrollbar-track) }` / `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
|
||||||
|
2. `frontend/assets/styles.css` — define `--scrollbar-thumb` / `--scrollbar-track` in the theme's token block (derive from existing theme colors — the thumb must be visibly distinct from the track, ≥3:1 non-text contrast).
|
||||||
|
3. Do NOT restyle scrollbars elsewhere — the new selector stays scoped under `.doc-modal` (the TODO's scope).
|
||||||
|
4. ASSUMPTION: no line-wrap (locked A1, phase level) — the code stays unwrapped and scrolls.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the scroll check (task 02).
|
||||||
|
- Coverage: n/a (CSS) — the suite stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The modal code block's horizontal scrollbar is themed for Chromium (webkit pseudos) and Firefox (`scrollbar-color`) — the task-02 screenshot shows no native light-gray bar.
|
||||||
|
- [ ] No other page's scrollbar changes (the new selector is scoped under `.doc-modal`).
|
||||||
|
- [ ] `uv run pytest` green (in particular `tests/e2e/test_dark_tech_theme.py`).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Task 02 — E2E: the modal scrollbar check + screenshot
|
||||||
|
|
||||||
|
**Phase:** `116_modal_scrollbar_theme` · **Source:** `TODO.md:98–99` — "Acceptance: the modal's code block scrolls horizontally with a themed scrollbar; screenshot check in the document-viewer E2E story."
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The document-viewer E2E story asserts the modal code block scrolls horizontally and records a screenshot of the themed scrollbar.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_document_viewer.py` (existing — extend) — a test:
|
||||||
|
- open the document modal on a document with a line longer than the modal width (check the fixture KB's contents first; if no fixture doc qualifies, pick the longest existing line and assert `scrollWidth > clientWidth` against it — note the choice in the test docstring);
|
||||||
|
- the code block element: `scrollWidth > clientWidth` (the overflow is real);
|
||||||
|
- a horizontal scroll action (`scrollLeft` / `scrollBy`) actually moves it (the scroll works, no clipping);
|
||||||
|
- the themed rule is in effect: `getComputedStyle` exposes `scrollbar-color` (the webkit pseudos are not exposed via computed style — that half is covered by the screenshot);
|
||||||
|
- a screenshot to `.agents/screenshots/` (house convention — check the exact path other screenshot tests use).
|
||||||
|
2. Run in isolation: `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` (DB up).
|
||||||
|
3. Regression: the existing `test_document_viewer.py` tests stay green.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: the scroll behavior + the screenshot (the acceptance).
|
||||||
|
- Coverage: n/a.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The new E2E test green in isolation; the screenshot shows the themed scrollbar (no native light-gray bar).
|
||||||
|
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# Phase 117 — Compact, well-wrapped tool-call lines
|
||||||
|
|
||||||
|
**Source:** owner visual-glitch report (live chat, 2026-09-15) — "how much space the tool calls take up, and the tool call text is spit and wrapped poorly."
|
||||||
|
**Story:** n/a (owner bug report — mobile viewport, `https://brain.reeseapps.com`, reproduced 2026-09-15)
|
||||||
|
**Context:** `frontend/assets/app.js` (`appendToolLine` — renders one `.tool-call` line per `tool` SSE frame into a `.tool-calls` list above the answer bubble; `ensureThinkingBlock` — places the Thinking `<details>` above the `.tool-calls` list via `anchor = body.querySelector(".tool-calls") ?? body.querySelector(".bubble")`; the delta/done/stop handlers each call `closeThinkingBlock(wrap)`; `renderStoredMessage` re-renders persisted tools through the SAME `appendToolLine`); `frontend/assets/shared.js` (`addToolLines` — the shared page's local copy of the same renderer, pinned byte-parity with `appendToolLine`); `frontend/assets/styles.css` (`.tool-calls` = flex column + gap; `.tool-call` = `display:flex; align-items:baseline` + a full card: `background` + `border` + `border-left:3px var(--accent-line)` + `border-radius` + `padding`; `.tool-call code` = a chip: `var(--brand-soft)` background + padding + radius on top of `var(--mono)`/`var(--ink)`); `tests/unit/test_frontend_tool_states.py` (pins the exact tool-line template literals + the `.tool-calls`/`.tool-call` DOM shape + the `.tool-call` CSS — `test_tool_call_style_is_accent_and_contrast_safe` asserts `display: flex` + `var(--accent-line)` on `.tool-call`); `tests/unit/test_big_read_progress.py` (pins phase 87's `.tool-elapsed` clock, which queries `.tool-calls .tool-call:last-child` and appends a SIBLING suffix — the line's own text/literals stay byte-identical); `tests/e2e/test_agent_document_tools.py` + `tests/e2e/test_big_read_progress.py` (the tool-line E2E — the former asserts `.tool-call` count + `to_contain_text` after a completed turn, the latter asserts the FIRST `.tool-call` line and the `.tool-elapsed` suffix are `to_be_visible` DURING the live frameless gap, before the answer's delta).
|
||||||
|
|
||||||
|
## Bug basis (code-traced + reproduced live, 2026-09-15)
|
||||||
|
Reproduced at 390×844 (mobile) on the live site. One answer to "Generate a change log … last 5 phases" rendered **6+ stacked full-width cards** — one per `tool` frame — each a complete bordered card (accent left border + surface background + radius) holding a mono path *chip* inside it. Two distinct defects:
|
||||||
|
- **Space:** each `tool` frame appends a full-width bordered `.tool-call` card spanning the whole chat column. The agent loop is round-capped and fires several calls per turn (`ls → read → read …`), so a single answer stacks N cards above it. On a phone the tool process visually swamps the answer.
|
||||||
|
- **Wrapping:** `.tool-call` is `display:flex; align-items:baseline`, so the label text node (`"📄 Reading "`) and the `<code>` path are **two separate flex items**. The long mono path squeezes the label flex-item, and `overflow-wrap: anywhere` breaks the label **mid-word** (`Reading` → `Rea` / `ding`); the path wraps to 3 lines indented to the right of that narrow broken label. The "spit and wrapped poorly" look.
|
||||||
|
- **The live-feedback that must be preserved:** the "calling tool" live state lives in `#send-status` (aria-live) + the typing-indicator aria-label (pinned by `test_calling_tool_label_strings`), NOT the tool lines — so folding the tool lines does not remove any live feedback. Phase 87's ticking `.tool-elapsed` suffix, however, IS on a `.tool-call` line and its E2E asserts it is **visible during the live gap** — so the lines must stay visible while the turn is in flight and fold only once the answer begins.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Turn the per-call tool "cards" into a single collapsible disclosure — one compact "Tool calls (N)" line by default, expandable to the individual calls — and restyle the individual lines as deboxed, inline-flowing text so the path wraps to the left edge like a normal sentence and the label never breaks mid-word. Collapsed by default for completed/restored turns (the space fix), open during a live turn (keeps phase 87's live suffix visible), with the "calling tool" live state unchanged.
|
||||||
|
|
||||||
|
## Owner decisions (chat, 2026-09-15 — recorded per AGENTS.md rule 3)
|
||||||
|
- **D1 — Frontend-only.** No `app/` change, no new SSE frame, no persistence-format change. The stored record stays `{name, argument}` (+ `truncated`); only the *rendering* changes. The server and the SSE event set are byte-identical.
|
||||||
|
- **D2 — Reuse the Thinking-block convention.** The tool calls become a native `<details>/<summary>` disclosure in the same `.msg-body` wrap, mirroring `details.thinking` (open-while-active, closed-when-done). No new component, no new JS dependency.
|
||||||
|
- **D3 — Open live, folded at rest.** The disclosure is created **open** on the first live `tool` frame (the calls + phase 87's `.tool-elapsed` suffix stay visible during the turn); it folds when the answer's first `delta` arrives, on `done`, and on stop/abort (`closeToolCalls`, idempotent — the exact sites that call `closeThinkingBlock`). The restore path (`renderStoredMessage`) and the shared page (`addToolLines`) render it **closed**. This is what makes the user's screenshot (a completed turn) collapse to one line while keeping the live behavior green.
|
||||||
|
- **D4 — Debox + inline flow.** The `.tool-call` line loses its card (no `display:flex` / background / border / left border / radius / padding) so the label + inline `<code>` flow as one continuous run (fixes the mid-word label break + indented wrap); the `<code>` loses its chip background but keeps `var(--mono)` + `var(--ink)`. The accent is carried by the line's `color` (`var(--accent-ink)`), not a border.
|
||||||
|
- **D5 — Plain-text summary, no emoji.** The summary reads `Tool call (1)` / `Tool calls (N)` — plain text, matching the emoji-free chrome (the Thinking summary is just "Thinking") and the phase-08 emoji-free-chrome lean. The existing `📄`/`🔎` line glyphs stay (they are pinned literals + the emoji-guard strip set).
|
||||||
|
- **D6 — Keep the pinned literals + DOM shape.** The four `line.textContent = "…"` label literals, the `.tool-calls` list (role=list, aria-label "Tool calls"), the `.tool-call` listitems, and the `<code>` `textContent` arguments stay **byte-identical** — so `test_frontend_tool_states.py` (all but the one CSS assert), `test_big_read_progress.py`, the emoji guard, and phase 87's `.tool-calls .tool-call:last-child` query all stay green unchanged.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
### `frontend/assets/app.js`
|
||||||
|
- **`appendToolLine(wrap, name, argument)`** — wrap the list in a disclosure; keep everything else byte-identical:
|
||||||
|
- `let container = body.querySelector(".tool-calls-disclosure");` (idempotent per wrap — was `.tool-calls`).
|
||||||
|
- On first frame: `container = document.createElement("details"); container.className = "tool-calls-disclosure"; container.open = true;` then a `<summary class="tool-calls-summary">` (createElement, textContent only) appended first, then the existing list — `const listEl = document.createElement("div"); listEl.className = "tool-calls"; listEl.setAttribute("role","list"); listEl.setAttribute("aria-label","Tool calls");` — appended second; then `body.insertBefore(container, body.querySelector(".bubble"))` (the disclosure goes where the list did: above the answer, below an existing Thinking block).
|
||||||
|
- `const list = container.querySelector(".tool-calls");` — build the line EXACTLY as today (`line.className="tool-call"`, `role="listitem"`, the four pinned `line.textContent` label literals, the `<code>` with `code.textContent = argument`), `list.appendChild(line)`.
|
||||||
|
- Update the summary count on every append (live + restore): `const n = list.children.length; container.querySelector("summary").textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
|
||||||
|
- **No `innerHTML` anywhere in the function** (house rule; pinned). The function stays flat (no nested function declaration) so the `js[fn : js.find("\n}\n", fn)]` body-extraction pins keep working.
|
||||||
|
- **`closeToolCalls(wrap)`** (new, next to `closeThinkingBlock`): `const disc = wrap?.querySelector?.(".tool-calls-disclosure"); if (disc) disc.open = false;` — idempotent, no-op without a disclosure. Called at **every** existing `closeThinkingBlock(wrap)` site: the `delta` handler (≈L2514), the `done` handler (≈L2519), and the stop/abort settle (≈L2646).
|
||||||
|
- **`renderStoredMessage`** — after the existing `if (Array.isArray(m.tools)) { …appendToolLine(wrap, t.name, arg)… }` loop, add `closeToolCalls(wrap);` so a restored turn renders folded (the space fix; mirrors the thinking restore rendering collapsed).
|
||||||
|
- **`ensureThinkingBlock`** — the anchor becomes `body.querySelector(".tool-calls-disclosure") ?? body.querySelector(".tool-calls") ?? body.querySelector(".bubble")` so the Thinking block lands above the WHOLE disclosure (not inside it). `block.open = true` unchanged.
|
||||||
|
|
||||||
|
### `frontend/assets/shared.js`
|
||||||
|
- **`addToolLines(wrap, tools)`** — parity with `appendToolLine`, but **created closed** (pure render, always folded): build the `details.tool-calls-disclosure` (`container.open = false`) + `summary.tool-calls-summary` + the existing `.tool-calls` list (role=list, aria-label) exactly as today; render each line byte-identical (the same four label literals + `<code>` + the phase-95 `truncated-note`); set the summary count once at the end (`Tool call (N)` / `Tool calls (N)`). Keep the `code.textContent = argument` count at 3 and no `innerHTML` (the `test_shared_page_tool_lines_…` / `test_frontend_tool_states.py` shared pins).
|
||||||
|
|
||||||
|
### `frontend/assets/styles.css`
|
||||||
|
- **`.tool-call`** — DEBOX: remove `display:flex`, `align-items:baseline`, `gap`, `background`, `border`, `border-left`, `border-radius`, `padding`. KEEP `color: var(--accent-ink)`, `font-size: 0.8rem`, `line-height: 1.4`, `overflow-wrap: anywhere`. As a flex item of the `.tool-calls` column it stays block-level per line, but its label + inline `<code>` now flow as one continuous run → the path wraps to the left edge and the label no longer breaks mid-word.
|
||||||
|
- **`.tool-call code`** — DECHIP: remove `background: var(--brand-soft)`, `padding`, `border-radius`. KEEP `font-family: var(--mono)`, `font-size: 0.95em`, `color: var(--ink)` (≈11.5:1, AA).
|
||||||
|
- **NEW `.tool-calls-disclosure` + `.tool-calls-summary`** — model the disclosure on the existing `details.thinking` styling (house AA palette, no new hue): the summary is a native focusable toggle that carries the house 3px `:focus-visible` ring (the `details.thinking summary` already has it — reuse that language), small status font, `color: var(--accent-ink)` (≈10.4:1 on the surface — same pairing the deboxed line uses). No new color literal (phase-92 zero-literal invariant; B5 text+color, never color alone — the count is text).
|
||||||
|
- **`.tool-calls`** (the list) — UNCHANGED: stays `display:flex; flex-direction:column; gap: 0.25rem` (spacing between the now-deboxed lines).
|
||||||
|
- **`.tool-elapsed` / `.truncated-note`** — UNCHANGED (phase 87 / phase 95).
|
||||||
|
|
||||||
|
### `tests/unit/test_tool_call_compact.py` (NEW — source-level house pattern)
|
||||||
|
- **app.js:** `appendToolLine` body contains `document.createElement("details")` + `className = "tool-calls-disclosure"` + `document.createElement("summary")`; `container.open = true` (the live default, D3); the count literal `Tool call${n === 1 ? "" : "s"} (${n})` (or the equivalent template) is built with `textContent` (no `innerHTML`); `closeToolCalls` is defined and called at **3** handler sites (delta/done/stop) + once in `renderStoredMessage` (pin: `closeToolCalls(wrap)` appears ≥4×); `ensureThinkingBlock`'s anchor includes `.tool-calls-disclosure`; the four pinned `line.textContent = "…"` label literals are STILL present (mirror the guard); `innerHTML` is NOT in the `appendToolLine` body.
|
||||||
|
- **shared.js:** `addToolLines` body contains `document.createElement("details")` + `className = "tool-calls-disclosure"` + `document.createElement("summary")` + `open = false` (closed on the shared page, D3); the four label literals are still present; `body.count("code.textContent = argument") == 3`; no `innerHTML`.
|
||||||
|
- **styles.css:** the `.tool-call` rule has **NO** `display: flex` and **NO** `var(--accent-line)` (deboxed, D4) but still has `var(--accent-ink)`; the `.tool-call code` rule still has `var(--mono)` + `var(--ink)` and has **NO** `background`; `.tool-calls-disclosure` and `.tool-calls-summary` rules exist.
|
||||||
|
|
||||||
|
### `tests/unit/test_frontend_tool_states.py` (UPDATE — the only existing test that changes)
|
||||||
|
- **`test_tool_call_style_is_accent_and_contrast_safe`** — the `.tool-call` rule is deboxed: REMOVE the `assert "display: flex" in row` and `assert "var(--accent-line)" in row` lines; KEEP `assert "var(--accent-ink)" in row` (the accent is now the line's color) and the `.tool-call code` `var(--mono)` + `var(--ink)` asserts. Update the docstring to describe the deboxed, inline-flow line (phase 117). Every OTHER test in this file stays green unchanged (the DOM-shape, literal, branch-order, persist, restore, shared-parity, and no-CDN pins are all preserved by D6).
|
||||||
|
|
||||||
|
### `tests/e2e/test_tool_call_compact.py` (NEW story suite — mock "use your tools" flow, 3 tool calls)
|
||||||
|
The conftest `app_server`/`page` fixtures + the phase-37 admin login + the marker question that drives the mock's `ls → ls(scoped) → read` flow (mirror `test_agent_document_tools.py`). Four tests:
|
||||||
|
1. **Folds at rest:** after the turn completes, the `.tool-calls-summary` is visible with text `Tool calls (3)`; the disclosure is **not** open (`.tool-calls-disclosure` has no `[open]`); the `.tool-call` lines are present in the DOM (count 3) but hidden; the answer bubble is present.
|
||||||
|
2. **Expands on tap:** clicking the summary opens the disclosure; the three `.tool-call` lines become visible with the correct text (nth 0 "Listing documents", nth 1 "Listing documents in", nth 2 "Reading ").
|
||||||
|
3. **Deboxed inline flow:** on a visible (expanded) `.tool-call` line, `getComputedStyle(line).display !== "flex"` (the label + path are one inline run, not two flex items) and the line `to_contain_text("Reading ")` (label immediately followed by the path in the same run).
|
||||||
|
4. **Restored folded:** RELOAD (same context — the phase-14/50 persisted conversation restores); the restored brain message's `.tool-calls-disclosure` is present, closed, summary `Tool calls (3)`, `.tool-call` count 3 in the DOM — no auto-expand on load.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `95_read_truncation_cap`, `87_big_read_progress`, `70_harness_aligned_tools`, `37` (the tool-line rendering this restyles — complete). NO code dependency beyond the shared frontend files; the only pipeline predecessor is execution order (todo/ is empty — this is the next phase).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_disclosure_wrapper.md` — the `<details>/<summary>` wrapper + count + open-live/close-at-rest + restore-closed + shared.js parity (app.js + shared.js + the new unit module's JS pins).
|
||||||
|
2. `02_debox_and_inline_flow.md` — the CSS debox/dechip + disclosure/summary rules, and the one `test_frontend_tool_states.py` CSS-pin update (styles.css + unit CSS pins + the existing CSS test).
|
||||||
|
3. `03_e2e_story_suite.md` — `tests/e2e/test_tool_call_compact.py` (fold / expand / deboxed-flow / restored-folded).
|
||||||
|
4. `04_verify_and_commit.md` — full gate (unit + integration, coverage >90%, the new E2E story in isolation, `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + smoke in isolation, ruff + pyright) + atomic commit + move to complete/.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit — `tests/unit/test_tool_call_compact.py` (new): the JS/CSS pins listed above (disclosure + summary + count + open-live, close-at-4-sites, restore-closed, shared parity, deboxed CSS, disclosure/summary rules, the pinned literals still present, no innerHTML).
|
||||||
|
- Existing unit suites MUST stay green: `tests/unit/test_big_read_progress.py` (phase-87 clock + the byte-identical tool-line literals), `tests/unit/test_shared_page.py` (shared-page tool-line parity), and `tests/unit/test_frontend_tool_states.py` (all tests except the ONE deboxed CSS assert updated in task 02).
|
||||||
|
- E2E — `tests/e2e/test_tool_call_compact.py` (new; isolation gate per AGENTS.md rule 9): the four tests above, mock LLM (no slow proxy needed — the fold is about a completed turn).
|
||||||
|
- Regression E2E (run in isolation by task 04): `test_agent_document_tools.py` (tool-line count + text after a completed turn — the lines stay in the DOM inside the folded disclosure), `test_big_read_progress.py` (the FIRST `.tool-call` line + the `.tool-elapsed` suffix are visible DURING the live gap — the disclosure is still open then), `test_thinking_display.py` (the Thinking block ordering vs. the disclosure), `test_smoke.py`.
|
||||||
|
- Coverage: **>90%** on `app/` — no `app/` code changes (the floor is held by the untouched suite).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A completed tool turn renders as ONE collapsed "Tool calls (N)" line (E2E pin 1); tapping it reveals the individual calls (E2E pin 2); the restored/shared view is folded on load (E2E pin 4). The 6+ stacked cards from the screenshot are gone.
|
||||||
|
- [ ] An expanded `.tool-call` line is deboxed inline-flow text (E2E pin 3 + the CSS unit pins): the path wraps to the left edge and the label never breaks mid-word — the "spit and wrapped poorly" glitch is fixed at 390×844 and at desktop.
|
||||||
|
- [ ] Live behavior unchanged: during a live frameless gap the calls + phase-87 `.tool-elapsed` suffix stay visible (the disclosure is open until the first delta) — `test_big_read_progress.py` green in isolation; the "calling tool" `#send-status`/aria state is byte-identical (`test_calling_tool_label_strings` green).
|
||||||
|
- [ ] `uv run pytest` green (including `test_big_read_progress.py`, `test_shared_page.py`, and the updated `test_frontend_tool_states.py`); `uv run pytest --cov=app --cov-report=term-missing` >90%; the new E2E story + `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `git diff --stat` limited to `frontend/assets/app.js`, `frontend/assets/shared.js`, `frontend/assets/styles.css`, `tests/unit/test_tool_call_compact.py`, `tests/unit/test_frontend_tool_states.py`, `tests/e2e/test_tool_call_compact.py`, and the phase files — nothing in `app/`.
|
||||||
|
- [ ] One atomic `--no-gpg-sign` commit (e.g. `feat(ui): fold tool calls into a compact collapsible disclosure`); phase dir moved to `.agents/phases/complete/`.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **Frontend-only (D1)** — `app/`, the SSE event set, and the persistence format are byte-identical; only the rendering changes.
|
||||||
|
- **Reuse the native `<details>` Thinking-block convention (D2/D3)** — open while the turn is live, folded at rest and on restore; no new component or dependency. This is what keeps phase 87's live-suffix E2E and the tool-line count/text E2E green.
|
||||||
|
- **Pinned literals + DOM shape stay (D6)** — the four `line.textContent` labels, the `.tool-calls` list, the `.tool-call` listitems, and the `<code>` `textContent` arguments are byte-identical; the emoji guard and phase 87's `.tool-calls .tool-call:last-child` query are unaffected. Only the ONE CSS assert in `test_frontend_tool_states.py` changes (the debox).
|
||||||
|
- **Debox, don't restyle the accent away (D4)** — the line keeps `var(--accent-ink)` as its text color (≈10.4:1 AA on the surface); the accent moves off the border onto the text, so the line still reads as distinct from the brand-ink Thinking summary.
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Task 01 — The `<details>/<summary>` disclosure wrapper + count + open-live/close-at-rest
|
||||||
|
|
||||||
|
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15) — tool-call cards take too much space and wrap poorly.
|
||||||
|
**Story:** n/a (owner bug report)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Wrap the existing `.tool-calls` list in a native `details.tool-calls-disclosure` with a `summary.tool-calls-summary` that carries a `Tool call (N)` / `Tool calls (N)` count. The disclosure is created **open** on the first live `tool` frame, **folds** when the answer starts / the turn ends / is stopped (`closeToolCalls`), and renders **closed** on the restore path and the shared page. The `.tool-calls` list, the `.tool-call` lines, the four label literals, and the `<code>` arguments stay byte-identical (D6) — only the wrapper is added.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — `appendToolLine` (house comment style, citing phase 117):
|
||||||
|
- `let container = body.querySelector(".tool-calls-disclosure");` (was `.tool-calls`).
|
||||||
|
- First-frame branch (the `if (!container)` block): create the disclosure + summary + the existing list, in this order:
|
||||||
|
```js
|
||||||
|
container = document.createElement("details");
|
||||||
|
container.className = "tool-calls-disclosure";
|
||||||
|
container.open = true; // D3: open while the turn is live; closeToolCalls folds it
|
||||||
|
const summary = document.createElement("summary");
|
||||||
|
summary.className = "tool-calls-summary";
|
||||||
|
container.appendChild(summary);
|
||||||
|
const listEl = document.createElement("div");
|
||||||
|
listEl.className = "tool-calls";
|
||||||
|
listEl.setAttribute("role", "list");
|
||||||
|
listEl.setAttribute("aria-label", "Tool calls");
|
||||||
|
container.appendChild(listEl);
|
||||||
|
body.insertBefore(container, body.querySelector(".bubble"));
|
||||||
|
```
|
||||||
|
- After the `if`: `const list = container.querySelector(".tool-calls");` then build the line EXACTLY as today (`line.className="tool-call"`, `role="listitem"`, the four pinned `line.textContent` label literals, the `<code>` with `code.textContent = argument`) and `list.appendChild(line)`.
|
||||||
|
- Summary count on every append (live + restore): `const n = list.children.length; container.querySelector("summary").textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
|
||||||
|
- **No `innerHTML`** anywhere in the function; keep it flat (no nested function declaration) so the `js[fn : js.find("\n}\n", fn)]` body-extraction pins keep working.
|
||||||
|
2. `frontend/assets/app.js` — `closeToolCalls(wrap)` (new, defined next to `closeThinkingBlock`):
|
||||||
|
```js
|
||||||
|
function closeToolCalls(wrap) {
|
||||||
|
const disc = wrap?.querySelector?.(".tool-calls-disclosure");
|
||||||
|
if (disc) disc.open = false; // idempotent; no-op without a disclosure
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Add `closeToolCalls(wrap);` on the line next to each existing `closeThinkingBlock(wrap);` — the `delta` handler (≈L2514), the `done` handler (≈L2519), and the stop/abort settle (≈L2646). Comment each: the answer began / the turn ended / the turn was stopped — fold the record (mirrors the Thinking block settling closed).
|
||||||
|
3. `frontend/assets/app.js` — `renderStoredMessage`: immediately after the existing `if (Array.isArray(m.tools)) { …appendToolLine(wrap, t.name, arg)… }` loop, add `closeToolCalls(wrap);` (a restored turn renders folded — the space fix; mirrors the thinking restore rendering collapsed).
|
||||||
|
4. `frontend/assets/app.js` — `ensureThinkingBlock`: change the anchor to
|
||||||
|
`const anchor = body.querySelector(".tool-calls-disclosure") ?? body.querySelector(".tool-calls") ?? body.querySelector(".bubble");`
|
||||||
|
so the Thinking block lands above the whole disclosure. `block.open = true` and everything else unchanged.
|
||||||
|
5. `frontend/assets/shared.js` — `addToolLines` (parity, but **created closed** — pure render, always folded):
|
||||||
|
- Build the same `details.tool-calls-disclosure` with `container.open = false`, a `summary.tool-calls-summary` (first child), and the existing `.tool-calls` list (role=list, aria-label "Tool calls") — appended to the same `.msg-body` anchor as today.
|
||||||
|
- Render each line byte-identical (the same four label literals + `<code>` + the phase-95 `truncated-note`), appending to the list.
|
||||||
|
- Set the summary count once at the end: `const n = list.children.length; summary.textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
|
||||||
|
- Keep `body.count("code.textContent = argument") == 3` and no `innerHTML` (the shared-parity pins).
|
||||||
|
6. `tests/unit/test_tool_call_compact.py` (NEW module) — the source-level JS pins (house pattern, `_js()` / `_shared_js()` readers like `test_frontend_tool_states.py`):
|
||||||
|
- **app.js `appendToolLine`:** body contains `document.createElement("details")`, `className = "tool-calls-disclosure"`, `document.createElement("summary")`; `container.open = true`; the count template `Tool call${n === 1 ? "" : "s"} (${n})` (assert the template fragment, not the rendered value); `innerHTML` NOT in the body; the four pinned `line.textContent = "…"` label literals are still present.
|
||||||
|
- **app.js `closeToolCalls`:** defined (`function closeToolCalls`); `closeToolCalls(wrap)` appears **≥4×** (the 3 handler sites + `renderStoredMessage`).
|
||||||
|
- **app.js `ensureThinkingBlock`:** the body contains `.tool-calls-disclosure` (the new anchor term) AND still contains `querySelector(".tool-calls")` + `querySelector(".bubble")` + `block.open = true`.
|
||||||
|
- **shared.js `addToolLines`:** body contains `document.createElement("details")`, `className = "tool-calls-disclosure"`, `document.createElement("summary")`, `open = false` (closed on the shared page); the four label literals present; `body.count("code.textContent = argument") == 3`; no `innerHTML`.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `uv run pytest tests/unit/test_tool_call_compact.py -v` green (all JS pins). `uv run pytest tests/unit/test_frontend_tool_states.py tests/unit/test_big_read_progress.py tests/unit/test_shared_page.py -v` green **unchanged** (the CSS pin is task 02; the JS/DOM pins are preserved by D6).
|
||||||
|
- Quick live sanity (session log): dev server, trigger a tool turn — during the turn the disclosure is open (calls + any `.tool-elapsed` suffix visible); when the answer starts it folds to "Tool calls (N)"; a reload shows it folded. (The deterministic E2E is task 03 — this is a wiring smoke only.)
|
||||||
|
- Coverage: **>90%** on `app/` unaffected (no `app/` change).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `git diff frontend/assets/app.js` shows: the disclosure/summary/list creation in `appendToolLine`, the summary count line, the new `closeToolCalls`, the three handler call sites + the `renderStoredMessage` call, and the `ensureThinkingBlock` anchor — and NOTHING else; the four `line.textContent` label literals and the `<code>` `textContent` bytes byte-identical.
|
||||||
|
- [ ] `git diff frontend/assets/shared.js` shows the parallel closed-disclosure build + count + the unchanged line/literal bytes.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_tool_call_compact.py -v` green; `tests/unit/test_frontend_tool_states.py` + `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py` green unchanged (CSS test still asserts the OLD flex/border — that flips in task 02, so run it here to confirm only that one assert is the pending delta).
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change in completed work (the fold/expand proof is task 03; the live-suffix regression is task 04).
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Task 02 — Debox + inline-flow CSS, and the one CSS-pin update
|
||||||
|
|
||||||
|
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15) — the tool-call "cards" are too heavy and the label breaks mid-word.
|
||||||
|
**Story:** n/a (owner bug report)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Debox the `.tool-call` line (no card, no flex) so the label + inline `<code>` flow as one continuous run (fixes the `Rea`/`ding` mid-word break and the indented 3-line wrap), dechip the `.tool-call code` (no background chip, keep mono + ink), and add the `.tool-calls-disclosure` / `.tool-calls-summary` rules modeled on the Thinking disclosure. Update the ONE existing CSS assert in `test_frontend_tool_states.py` to the deboxed contract, and add the CSS pins to the new unit module.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/styles.css` — **`.tool-call`** (the debox, D4). Remove `display: flex;`, `align-items: baseline;`, `gap: 0.45rem;`, `background: var(--surface);`, `border: 1px solid var(--line);`, `border-left: 3px solid var(--accent-line);`, `border-radius: var(--radius-sm);`, `padding: 0.3rem 0.75rem;`. KEEP `color: var(--accent-ink);`, `font-size: 0.8rem;`, `line-height: 1.4;`, `overflow-wrap: anywhere;`. Update the rule's comment: phase 117 deboxed the line — the accent now rides the text color (≈10.4:1 on the surface) instead of a border; the label + inline `code` flow as one run so the path wraps to the left edge (no more mid-word label break).
|
||||||
|
2. `frontend/assets/styles.css` — **`.tool-call code`** (the dechip). Remove `background: var(--brand-soft);`, `padding: 0.05em 0.35em;`, `border-radius: 5px;`. KEEP `font-family: var(--mono);`, `font-size: 0.95em;`, `color: var(--ink);`, `overflow-wrap: anywhere;`. Comment: the path keeps the mono + ink treatment (≈11.5:1 AA) but loses the chip background — it is inline text in the line's run now.
|
||||||
|
3. `frontend/assets/styles.css` — **NEW `.tool-calls-disclosure` + `.tool-calls-summary`** rules, placed next to the `details.thinking` rules and modeled on them (house AA palette, no new hue — the phase-92 zero-literal invariant):
|
||||||
|
```css
|
||||||
|
/* Phase 117: the tool-call record folds into a single disclosure line,
|
||||||
|
reusing the Thinking block's native <details> convention. Open while
|
||||||
|
the turn is live (phase 87's elapsed suffix stays visible), folded at
|
||||||
|
rest / on restore. The summary is a native focusable toggle — it
|
||||||
|
inherits the house 3px :focus-visible ring (see details.thinking
|
||||||
|
summary). Accent rides the TEXT (var(--accent-ink), ≈10.4:1 on the
|
||||||
|
surface) — text + color, never color alone (B5). No new literal. */
|
||||||
|
.tool-calls-disclosure { margin: 0.25rem 0; }
|
||||||
|
.tool-calls-summary {
|
||||||
|
cursor: pointer;
|
||||||
|
color: var(--accent-ink);
|
||||||
|
font-size: 0.8rem;
|
||||||
|
line-height: 1.4;
|
||||||
|
padding: 0.15rem 0;
|
||||||
|
list-style: none; /* the native marker is redundant with the count text */
|
||||||
|
}
|
||||||
|
.tool-calls-summary::-webkit-details-marker { display: none; }
|
||||||
|
```
|
||||||
|
(If the `details.thinking summary` uses a chevron/marker rather than the native one, match that treatment so the two disclosures read as one family — the count text is the accessible label either way.)
|
||||||
|
4. `frontend/assets/styles.css` — **`.tool-calls`** (the list) — UNCHANGED: keep `display: flex; flex-direction: column; gap: 0.25rem;` (spacing between the now-deboxed lines). **`.tool-elapsed`** and **`.tool-call .truncated-note`** — UNCHANGED (phase 87 / phase 95).
|
||||||
|
5. `tests/unit/test_tool_call_compact.py` (extend the module from task 01) — the CSS pins:
|
||||||
|
- the `.tool-call` rule (regex `\.tool-call \{([^}]*)\}`) has **NO** `display: flex` and **NO** `var(--accent-line)` (deboxed), but still has `var(--accent-ink)`;
|
||||||
|
- the `.tool-call code` rule still has `var(--mono)` + `var(--ink)` and has **NO** `background`;
|
||||||
|
- `.tool-calls-disclosure` and `.tool-calls-summary` rules exist (the summary carries `var(--accent-ink)`).
|
||||||
|
6. `tests/unit/test_frontend_tool_states.py` — **`test_tool_call_style_is_accent_and_contrast_safe`** (the ONE existing test that changes, D4/D6):
|
||||||
|
- REMOVE the two now-false asserts: `assert "display: flex" in row` and `assert "var(--accent-line)" in row`.
|
||||||
|
- KEEP `assert "var(--accent-ink)" in row` (the accent is now the line's color) and the `.tool-call code` `var(--mono)` + `var(--ink)` asserts and the `.tool-calls` `gap` assert.
|
||||||
|
- Update the docstring: the line is now a deboxed inline-flow row (phase 117) — the accent rides the text color; the `code` is inline mono text (no chip). Everything else in this file is untouched.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `uv run pytest tests/unit/test_tool_call_compact.py -v` green (JS pins from task 01 + the new CSS pins); `uv run pytest tests/unit/test_frontend_tool_states.py -v` green with the updated CSS test; `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py` green unchanged.
|
||||||
|
- Quick live sanity (session log): dev server, expand a completed tool turn at 390×844 — the lines are plain text rows (no card border/background), the path wraps to the left edge, and "Reading" never breaks as "Rea/ding". (The deterministic E2E is task 03.)
|
||||||
|
- Coverage: **>90%** on `app/` unaffected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `git diff frontend/assets/styles.css` shows: the deboxed `.tool-call`, the dechipped `.tool-call code`, the two new disclosure/summary rules — and NOTHING else (`.tool-calls` gap, `.tool-elapsed`, `.truncated-note` unchanged).
|
||||||
|
- [ ] `git diff tests/unit/test_frontend_tool_states.py` shows ONLY the two removed asserts + the docstring in `test_tool_call_style_is_accent_and_contrast_safe`.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_tool_call_compact.py tests/unit/test_frontend_tool_states.py tests/unit/test_big_read_progress.py tests/unit/test_shared_page.py -v` all green.
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change in completed work (E2E proof in task 03/04).
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# Task 03 — The E2E story suite (fold / expand / deboxed-flow / restored-folded)
|
||||||
|
|
||||||
|
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15).
|
||||||
|
**Story:** n/a (owner bug report)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Prove the new behavior end-to-end against the deterministic mock: a completed 3-call tool turn folds to one "Tool calls (3)" line, expands on tap to the deboxed inline-flow lines, and renders folded on reload. Reuses the phase-37 admin login + the mock "use your tools" flow (exactly the 3 tool calls `test_agent_document_tools.py` drives: `ls` → `ls(scoped)` → `read`).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_tool_call_compact.py` (NEW module) — header docstring cites phase 117 + the owner report; the conftest `app_url` + `page` fixtures; `from e2e.auth_helpers import login`. Constants:
|
||||||
|
- `MARKER_QUESTION = "Use your tools: what is the exact JSON shape of reeselink.json for my aws route53 hosted zone?"` (the SAME marker that drives the mock's 3-tool flow — mirror `test_agent_document_tools.py` so the flow is deterministic).
|
||||||
|
- `SUMMARY = "#messages .msg.brain .tool-calls-summary"` (the disclosure toggle).
|
||||||
|
- `DISCLOSURE = "#messages .msg.brain .tool-calls-disclosure"`.
|
||||||
|
- `LINES = "#messages .msg.brain .tool-call"`.
|
||||||
|
- A `_submit_tools_turn(page, app_url)` helper: `login(page, app_url, next="/")`; `page.fill("#message-input", MARKER_QUESTION)`; `page.click("#send-btn")`; wait for the answer bubble (`MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"`) — i.e. the turn is COMPLETE (the disclosure has already folded on the first delta).
|
||||||
|
2. **`test_tool_calls_fold_to_one_line_after_turn`** (pin 1): after `_submit_tools_turn`:
|
||||||
|
- `expect(page.locator(SUMMARY)).to_be_visible()` and `to_contain_text("Tool calls (3)")`.
|
||||||
|
- the disclosure is NOT open: `page.get_attribute(DISCLOSURE, "open")` is `None` (a closed `<details>` has no `open` attribute).
|
||||||
|
- the lines are present in the DOM but folded: `expect(page.locator(LINES)).to_have_count(3)` (DOM count) and the FIRST line is NOT visible (`expect(page.locator(LINES).first).not_to_be_visible()`).
|
||||||
|
- the answer is present: `expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)`.
|
||||||
|
3. **`test_summary_click_expands_the_calls`** (pin 2): after `_submit_tools_turn`:
|
||||||
|
- `page.locator(SUMMARY).click()` (native `<summary>` — a real focusable toggle, AA).
|
||||||
|
- `page.get_attribute(DISCLOSURE, "open")` is now `"true"` (or the attribute present).
|
||||||
|
- `expect(page.locator(LINES)).to_have_count(3)` and every line is now visible; assert the text (mirror `test_agent_document_tools.py`): nth 0 `to_contain_text("Listing documents")`, nth 1 `to_contain_text("Listing documents in")`, nth 2 `to_contain_text("Reading ")`.
|
||||||
|
4. **`test_expanded_line_is_deboxed_inline_flow`** (pin 3): after expanding (reuse the click from pin 2 in a fresh turn):
|
||||||
|
- take the visible `.tool-call` "Reading" line; assert `page.evaluate` on its computed style: `getComputedStyle(el).display !== "flex"` (the label + path are one inline run, not two flex items — the debox).
|
||||||
|
- `expect(line).to_contain_text("Reading ")` (the label is immediately followed by the path in the same run — no separate indented code column).
|
||||||
|
5. **`test_restored_turn_renders_folded`** (pin 4): after `_submit_tools_turn`:
|
||||||
|
- `page.reload()` (same context — the phase-14/50 persisted conversation restores).
|
||||||
|
- wait for the restored brain message's answer; then `expect(page.locator(SUMMARY)).to_be_visible()` + `to_contain_text("Tool calls (3)")`; the disclosure is NOT open (`get_attribute(...) is None`); `expect(page.locator(LINES)).to_have_count(3)` in the DOM; the FIRST line is NOT visible (folded on load — no auto-expand).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- E2E: `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`). Mock LLM by default (no slow proxy — the fold is about a completed turn, not a live gap).
|
||||||
|
- No `app/` change — the conftest app-server + mock flow are used as-is.
|
||||||
|
- Coverage: **>90%** on `app/` unaffected.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` green in isolation (all four pins).
|
||||||
|
- [ ] The suite proves: folded-after-turn (pin 1), expand-on-tap (pin 2), deboxed inline-flow (pin 3), restored-folded (pin 4).
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] No behavior change in completed work (the adjacent-suite regression is task 04).
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 04 — Full gate + atomic commit
|
||||||
|
|
||||||
|
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15).
|
||||||
|
**Story:** n/a (owner bug report)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Run the complete phase gate, land the phase as one atomic commit, and move the phase directory to `complete/`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. **Full regression gate** (AGENTS.md rule 9):
|
||||||
|
- `uv run pytest` — unit + integration green (including `tests/unit/test_tool_call_compact.py`, the updated `tests/unit/test_frontend_tool_states.py`, and the untouched `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py`).
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage **>90%** (no `app/` change this phase — confirm the floor is held).
|
||||||
|
- `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` — green **in isolation** (this phase's E2E story — fold / expand / deboxed-flow / restored-folded).
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — green in isolation (the tool-line count + text after a completed turn — the lines stay in the DOM inside the folded disclosure).
|
||||||
|
- `uv run pytest tests/e2e/test_big_read_progress.py -v --no-cov` — green in isolation (the FIRST `.tool-call` line + the `.tool-elapsed` suffix are visible DURING the live gap — the disclosure is still open then; this is the key live-behavior regression proof for D3).
|
||||||
|
- `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov` — green in isolation (the Thinking block ordering vs. the new disclosure anchor).
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` — green in isolation.
|
||||||
|
- `uv run ruff check . && uv run pyright` — clean.
|
||||||
|
2. **Manual live check** (keep the output in the session log): the live/dev app at a **mobile viewport (390×844)** — trigger a multi-call tool turn (e.g. "Generate a change log … last 5 phases"). Confirm: the 6+ stacked cards from the original report are now ONE collapsed "Tool calls (N)" line; tapping it reveals deboxed text rows whose paths wrap to the left edge with the label never breaking mid-word (`Reading` stays intact); during a live frameless gap the calls + any "(Ns)" suffix are visible (the disclosure is open); on reload the turn is folded.
|
||||||
|
3. **Commit** (AGENTS.md rule 8 — one atomic, Conventional-Commits commit, always `--no-gpg-sign`), staging `frontend/assets/app.js`, `frontend/assets/shared.js`, `frontend/assets/styles.css`, `tests/unit/test_tool_call_compact.py`, `tests/unit/test_frontend_tool_states.py`, `tests/e2e/test_tool_call_compact.py`, and the phase files:
|
||||||
|
`feat(ui): fold tool calls into a compact collapsible disclosure`
|
||||||
|
— body: owner visual-glitch report (2026-09-15) — each tool call rendered as a full-width bordered card, so a multi-call turn stacked N cards over the answer, and the flex label+code layout broke the label mid-word ("Reading" → "Rea/ding") with the path indented to 3 lines. Two changes, frontend-only: (1) the tool-call record now folds into a single native `<details>` "Tool calls (N)" line reusing the Thinking-block convention — open while the turn is live (phase 87's elapsed suffix stays visible), folded when the answer starts / on done / on stop / on restore, closed by default on the shared page; (2) the individual lines are deboxed (no card, no flex) and dechipped (no code background) so the label + path flow as one inline run that wraps to the left edge. The four pinned line literals, the `.tool-calls` list, the `<code>` textContent arguments, the emoji-guard set, the persistence format, and the server are byte-identical; only the ONE deboxed CSS assert in `test_frontend_tool_states.py` changes. New source-level unit pins + a dedicated fold/expand/debox/restore E2E story.
|
||||||
|
4. Move the phase directory: `mv .agents/phases/todo/117_tool_call_compact .agents/phases/complete/` and include the move in the same commit.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the phase-level gate — the commands above are the completion evidence.
|
||||||
|
- Coverage: >90% held.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All seven gate commands green (unit + integration, coverage >90%, the new E2E story + `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + smoke in isolation, ruff + pyright).
|
||||||
|
- [ ] The live mobile check shows the stacked cards replaced by one collapsed "Tool calls (N)" line that expands to deboxed inline-flow rows (path wraps to the left edge, label intact), live calls still visible during a gap, and restored turns folded (output kept in the session log).
|
||||||
|
- [ ] `git show --stat HEAD` lists the staged files above + the phase files (todo → complete move) — nothing else (in particular `app/`, `pyproject.toml` / `uv.lock`, `conftest.py`, `mock_llm.py`, `slow_llm.py` untouched).
|
||||||
|
- [ ] `.agents/phases/complete/117_tool_call_compact/` exists; `todo/` no longer contains it.
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Phase 118 — Summary seed context: top-5 "start here" suggestions (TODO L3)
|
||||||
|
|
||||||
|
**Source:** `TODO.md` L3 — "I want to change the way the initial chat context works. Right now we feed two entire documents into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day."
|
||||||
|
**Story:** n/a (owner-directed RAG behavior change; extends the phase-30 summaries, the phase-37/70/94 agent tools, and the phase-113 tiering assets).
|
||||||
|
**Context:** A grounded (HIGH) turn currently seeds `build_high_prompt` with the **full text** of the top-2 cited documents (`select_documents_tiered`'s cited tier — floor `BOR_SOURCE_USEFULNESS_FLOOR` 0.35, ceiling `BOR_TOP_N_DOCS` 2, `app/api/chat.py:plan_turn`), and `run_agent` receives them as `seed_docs` whose `read` is refused (`ALREADY_IN_CONTEXT` — the full text is already in the prompt). Summaries exist on `Document.summary` (phase-30 lite digest — currently **non-markdown only**; NULL for markdown docs, pre-30 rows and fail-soft misses) and are embedded as one `is_summary` chunk (position −1) per document. LOCKED A7 (full text of the top-2, never truncated, on the retrieval path) is **re-revised by the owner directive in this TODO**: full text enters the context only through the capped `read` tool (locked A6 below).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Grounded turns seed the top 5 related documents as **summary** blocks — a "start here if these summaries seem right to you" starting point — and the LLM extends its context by `read`-ing only what it needs; every document (markdown included) has a stored summary (generated for all new/changed docs at import, backfilled on the next sync for existing NULL rows). Answers get faster and never start by ingesting massive possibly-irrelevant full texts.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `117_tool_call_compact` (complete) — the last completed phase; this phase rewrites the `read`-tool copy the battery gate pins, so the gate (task 07) runs against its final state.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
- **Suggested tier (task 02):** new pure `select_suggested(chunks, n=None)` in `app/rag/retriever.py` — the top-N DISTINCT parent documents in fused rank order (the same stable score-descending walk as `select_documents_tiered` — a document's rank is fixed by its FIRST seen chunk), **no cosine floor** (suggestions are opt-in starting points, not citations — the floor stays for the `done`-event related tier only). `n` defaults to the new setting `suggested_docs` (`BOR_SUGGESTED_DOCS`, default **5**, validator `>= 1`).
|
||||||
|
- **Suggestion blocks (task 03):** the HIGH prompt keeps the `<documents>` section tag and the per-block `<document …>` markup (the E2E mock keys off both), but each block's BODY becomes the document's **summary**: the identity attributes (`source`/`path`/`title`/`date` — the phase-106 D5 `date` stays, always present) + `doc.summary` (stripped). Defensive fallback ONLY when `doc.summary` is None/whitespace (a fail-soft miss — locked A5): the first `suggestion_preview_chars` (`BOR_SUGGESTION_PREVIEW_CHARS`, default **400**) content characters + the shared `TRUNCATION_MARKER` — no LLM call at chat time. A new pinned intro line (a module constant in `app/rag/prompts.py`) precedes the blocks inside the section: the start-here framing — the blocks are summaries of the top-ranked documents; if one seems right, `read` that document (combined `source/path`) to add its full content; answer from what you have and what you read; cite the document(s) you used by path.
|
||||||
|
- **Agent `read` (task 04):** with summary seeds, `read` on a suggested document **adds its full text** — the existing mechanics are all unchanged (the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `ToolResultPiece`/SSE `tool_result` frame, the phase-106 D5 `date:` line, `holder.read_docs` append, `tool_calls` increment). The dedupe set in `_execute_tool`'s read branch becomes **`holder.read_docs` only** (the seeds drop out — a suggested doc is a summary, not full text), so `ALREADY_IN_CONTEXT` fires only for a document ALREADY READ into full-text context (its copy stays accurate: "the full text is already in your prompt"). The `read` tool description in `AGENT_TOOLS` (`app/rag/agent.py`) and the `read` clause of `TOOLS_SECTION` (`app/rag/prompts.py`) are rewritten for the summary-seed mode — SINGLE mode (no kill switch, locked A1): the `<documents>` section shows summaries, `read` adds a document's full text, and re-reading an already-read document is refused. The `ls`/`grep` clauses and the discipline rules (one call per reply, never repeat) stay byte-identical. `tests/unit/test_prompt_lock.py` byte-pins `TOOLS_SECTION` (sha256 + length + prefix/suffix): re-cut its anchors in the same change; `PERSONA` and the LOW `DEFLECT_MODE` body MUST stay byte-identical (the lock pins those too — the E2E mock keys on the markers' presence).
|
||||||
|
- **Turn wiring (task 05):** `plan_turn` computes BOTH tiers once, for BOTH branches: `suggested = select_suggested(chunks)` and `related = select_related(chunks, excluded={suggested ids}, cap=settings.related_max_docs)` (new pure helper in `app/rag/retriever.py` — the same rank walk, skipping the excluded ids, no floor; with a 5-doc suggestion the related tier is rank order 6..n, the phase-113 UI "nearby docs" row). The HIGH branch: `build_high_prompt(suggested, …)` and `run_agent(..., seed_docs=suggested)`; the LOW branch: prompt BYTE-IDENTICAL (weak-hit titles only, `DEFLECT_MODE`, no tools, chips — A8 gate untouched) but the TurnPlan still carries suggested + related for the durable record. `TurnPlan.docs` is RENAMED `suggested_docs` (same dataclass position); `summary_hits` is redefined as the hit chunks with `is_summary` whose parent document is in the SUGGESTED set. Citations: `done.sources` = suggested + agent-read, deduped by `(source, path)` order-preserving (locked A4 — a grounded turn always shows chips; the deflected turn keeps `[]`). Durable record (LOCKED A3 — the log records retrieval, not citations): `query_log.sources` + the per-turn log line = suggested + related + read, deduped. The per-turn log line gains `suggested=N` after `summary_hits=N` (PLAN §9 extension). `source_usefulness_floor` stops being a seeding input (it may stay for other consumers — the executor checks; the SETTING itself is never removed).
|
||||||
|
- **Summaries for every document (task 01):** the importer's summary step covers **every** new/changed A9 document — the markdown exclusion (`_import_one_file`'s `if suffix in (".md", ".markdown"): return`) is removed. **Backfill:** on the UNCHANGED branch (same `content_hash`), a document whose `summary is None` gets the same best-effort `_store_summary` pass — summary text stored on `doc.summary` + one new embedded `is_summary` chunk — with a `backfill=True` flag that counts success in a NEW `ImportSummary.summary_backfilled` counter (instead of `summaries`). The backfill runs BEFORE the `created_at_manual` early-return and the date refresh (the manual flag protects the DATE only, phase 106 D1 — a manually-dated doc still gets its summary), is still fail-soft (`LLMError`/`EmbeddingError` → session rollback + `summary_errors += 1`, the doc row untouched, the date refresh still runs), and NEVER counts `added`/`updated`/`pruned` (so no `sources_meta` bump, no KB-overview/folder-summary regeneration — the content is unchanged, locked A2). Non-NULL summaries (owner-edited via phase 57 included) are never touched. The import log line (PLAN §9) gains `summary_backfilled=N` after `summary_errors=N`.
|
||||||
|
- **Old-contract retirement (task 06):** `tests/e2e/test_whole_document_context.py` (the phase-24 full-text sentinel E2E) is DELETED — its contract no longer exists (locked A1/A6). Every test that pins the retired behavior is updated in place to pin the new one: markers to audit across `tests/unit/`, `tests/integration/`, `tests/e2e/` — the `ALREADY_IN_CONTEXT` seed-read flows (`test_agent.py`, `test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`), the `<documents>` full-text pins (`test_prompts.py`, `test_prompts_dates.py`, `test_chat_gate.py`), the markdown-summary pin (`test_importer.py::test_markdown_file_never_gets_summary` — inverted), the per-turn log-line pins (`test_chat_gate.py` — `summary_hits=`). A pin of UNCHANGED behavior stays untouched.
|
||||||
|
- **Copy gate (task 07):** the `read`-tool copy is the lever the gate telemetry proved matters (the phase-70/72/94 precedent) — the real-model fixture battery re-runs before the phase completes and its verdict is recorded in `TOOL_CALLING_TESTING.md` (gate semantics: locked A7 below).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_all_doc_summaries.md` — importer: every document gets a summary (markdown included) + the NULL-summary backfill on the unchanged path.
|
||||||
|
2. `02_suggested_tier_selection.md` — retriever: `select_suggested` (top-N, no floor) + `BOR_SUGGESTED_DOCS`.
|
||||||
|
3. `03_summary_document_blocks.md` — prompts: the "start here" suggestion blocks in the HIGH prompt (+ the preview fallback).
|
||||||
|
4. `04_read_suggested_docs.md` — agent: `read` on a suggested doc adds its full text; the `read` copy rewritten (both surfaces); the prompt-lock re-cut.
|
||||||
|
5. `05_chat_turn_wiring.md` — `plan_turn`: suggested seeding, `select_related`, citations, the durable record, `suggested=N`.
|
||||||
|
6. `06_e2e_summary_context.md` — new Playwright E2E; the phase-24 E2E retired; the old pins updated.
|
||||||
|
7. `07_fixture_battery_gate.md` — the real-model fixture battery re-run + the verdict recorded.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_importer.py` (markdown summarized; backfill success/failure; the manual-date case; the log-line counter), `tests/unit/test_retriever.py` (`select_suggested` / `select_related` rank order, cap, no floor, dedupe), `tests/unit/test_prompts.py` + `test_prompts_dates.py` (summary blocks, the `date` attribute, the preview fallback, the intro line; the LOW prompt byte-identical), `tests/unit/test_prompt_lock.py` (the re-cut `TOOLS_SECTION` anchors; `PERSONA` + LOW body untouched), `tests/unit/test_agent.py` (read-of-suggested success + re-read refusal + cap path), `tests/unit/test_chat_gate.py` (HIGH seeds 5 summaries — no full text in the prompt; related = rank 6+; both branches' durable record; `suggested=N`), `tests/unit/test_config.py` (the two new settings).
|
||||||
|
- E2E: `tests/e2e/test_summary_seed_context.py` (new — run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`).
|
||||||
|
- Regression: the remaining E2E suites green (the tool-flow suites re-targeted in task 06), and the battery gate (task 07).
|
||||||
|
- Coverage: **>90%** on `app/` (the validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A grounded turn seeds the top-5 suggested documents as SUMMARY blocks (no full text in the prompt); the LLM `read`s what it needs and answers faster.
|
||||||
|
- [ ] Every imported document (markdown included) has a `summary`; an existing NULL-summary doc backfills on the next sync (no content re-embed, no `sources_meta` bump, fail-soft).
|
||||||
|
- [ ] `read` on a suggested doc adds its full text (cap/truncation/date line unchanged); a re-read of an already-read doc is refused.
|
||||||
|
- [ ] `done.sources` = suggested + read (deduped); `query_log` + the log line record suggested + related + read; the log line carries `suggested=N`.
|
||||||
|
- [ ] The deflection path and every SSE frame shape are byte-identical; `PERSONA` + the LOW body stay byte-locked.
|
||||||
|
- [ ] `uv run pytest` green; coverage >90%; the new E2E green in isolation; the fixture battery gate PASS (task 07); `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — no kill switch (owner-confirmed 2026-09-15, roadmap confirmation):** the summary-seed context IS the behavior; the legacy full-text top-2 seeding is gone and the phase-24 E2E is retired (not re-scoped).
|
||||||
|
- **A2 — every document gets a summary, no exceptions, including markdown (owner-confirmed 2026-09-15):** generated for all new/changed docs at import; existing NULL rows backfill on the next sync as a summary-only pass (no content re-chunk/re-embed, no `sources_meta` bump); owner-edited (non-NULL) summaries are never touched.
|
||||||
|
- **A3 — the suggestion set = the top 5 distinct documents in fused rank order, NO cosine floor** (`BOR_SUGGESTED_DOCS` default 5); the floor stays for the `done`-event related tier only (owner-confirmed 2026-09-15).
|
||||||
|
- **A4 — `done.sources` (the citation surface) = suggested + agent-read, deduped** — a grounded turn always shows chips (owner-confirmed 2026-09-15).
|
||||||
|
- **A5 — a NULL summary at prompt time (a fail-soft miss) falls back to a 400-char content preview** (`BOR_SUGGESTION_PREVIEW_CHARS`) — deterministic, no LLM call at chat time (owner-confirmed 2026-09-15).
|
||||||
|
- **A6 — A7 re-revised (owner directive, TODO L3, 2026-09-15):** the retrieval path never seeds full texts; full text enters the context only through the capped `read` tool. `PLAN.md` is untouched (Protocol B) — the owner folds it in at the next redo.
|
||||||
|
- **A7 — the battery-gate semantics under summary seeding (owner-confirmed 2026-09-15):** conditions 1 (every turn answers) and 2 (zero round-cap hits) still gate; condition 4 (fixture-mode contract accuracy ≥ 0.90) still gates; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The full telemetry is recorded in `TOOL_CALLING_TESTING.md` regardless.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): seed grounded turns with top-5 document summaries instead of full texts — every document summarized, read extends the context"
|
||||||
|
```
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 01 — Summaries for every document + the NULL-summary backfill
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…now we have faster ways for the LLM to identify what information it needs… with their summaries loaded into context rather than their entire content…" (locked A2: every document gets a summary, no exceptions, including markdown.)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every A9 document has a stored lite summary: the markdown exclusion is dropped (all new/changed docs get one), and an existing unchanged doc whose `summary` is NULL gets a summary-only backfill on the next sync.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — `_import_one_file` (~L539–544): delete the markdown early-return (`if full_path.suffix.lower() in (".md", ".markdown"): return`) so EVERY new/changed document falls through to `_store_summary`. Update the adjacent comment (phase 30's "markdown is already natural language" rationale is retired — locked A2, 2026-09-15).
|
||||||
|
2. `app/rag/importer.py` — `_store_summary` (~L548–592): add a `backfill: bool = False` keyword; on success count `summary.summary_backfilled += 1` when `backfill` else `summary.summaries += 1` (the two counters are disjoint — the log line keeps both). Everything else (fail-soft rollback, the `is_summary` chunk replacement, the embed) is unchanged.
|
||||||
|
3. `app/rag/importer.py` — `_import_one_file`'s UNCHANGED branch (~L442–464): after `summary.unchanged += 1` and its log line, BEFORE the `created_at_manual` early-return and the date refresh, add: `if doc.summary is None: await _store_summary(session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary, backfill=True)`. Order matters: a backfill failure rolls back its own session work (nothing else is pending), and the date refresh still runs afterwards. Strict `is None` check — an empty-string summary is owner-set (phase 57) and is never overwritten. A backfill NEVER touches `content_hash`, the content chunks, `created_at` semantics, or the added/updated/pruned counters — so no `sources_meta` bump, no KB-overview/folder-summary regeneration (locked A2).
|
||||||
|
4. `app/rag/importer.py` — `ImportSummary` (~L95–117): new `summary_backfilled: int = 0` field (docstring: unchanged docs whose NULL summary was backfilled — one `is_summary` chunk per success); update the `summaries`/`summary_errors` docstrings ("Non-markdown files" → "Files" — every A9 format now, locked A2); `log()` gains `summary_backfilled=%d` placed after `summary_errors=%d` (PLAN §9 line extension).
|
||||||
|
5. `app/rag/summarizer.py` — NO copy change: `SUMMARY_INSTRUCTION` is already format-agnostic and the `SUMMARY_MODE` marker is the E2E mock's key (byte-stable).
|
||||||
|
6. ASSUMPTION: the backfill is NOT a KB change (locked A2) — no `sources_meta` bump even though a prompt-facing summary appears: the doc content is unchanged, and saved-chat staleness tracks content generations (the phase-53 invariant).
|
||||||
|
7. ASSUMPTION: `created_at_manual` rows still get the backfill — the manual flag protects the DATE only (phase 106 D1), not the summary.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `tests/unit/test_importer.py` — update/extend:
|
||||||
|
- `test_markdown_file_never_gets_summary` (~L574) → INVERT: a markdown file gets a stored summary + one embedded `is_summary` chunk (rename, e.g. `test_markdown_file_gets_stored_summary`).
|
||||||
|
- new: unchanged doc with NULL summary → backfilled (summary stored, one `is_summary` chunk with embedding, `unchanged` counter, `summary_backfilled=1`, `summaries` untouched, `added/updated/pruned` all 0).
|
||||||
|
- new: unchanged doc WITH a stored summary → no summary LLM call at all (the fake `chat` is never invoked) and no new chunk.
|
||||||
|
- new: unchanged doc with `created_at_manual=True` and NULL summary → summary backfilled, `created_at` byte-untouched.
|
||||||
|
- new: backfill failure (fake `chat` raises `LLMError`) → fail-soft: `summary_errors=1`, `summary_backfilled=0`, the doc row + date refresh unaffected (the date still refreshes when it differs).
|
||||||
|
- `test_import_summary_log_line_includes_summary_counters` (~L719) → the log line carries `summary_backfilled=N`.
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A sync of a KB containing markdown docs stores a summary on every doc (mock LLM `SUMMARY_MODE` path).
|
||||||
|
- [ ] A second sync (all unchanged) backfills exactly the NULL-summary docs; the third sync makes no summary calls at all.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_importer.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 02 — `select_suggested`: the top-N, no-floor suggestion tier
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pure `select_suggested(chunks, n)` in the retriever returns the top-N distinct parent documents in fused rank order, with NO cosine floor (locked A3), plus the `BOR_SUGGESTED_DOCS` setting (default 5).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/retriever.py` — new pure function `select_suggested(chunks: Sequence[RetrievedChunk], n: int | None = None) -> list[Document]`: walk `chunks` in the SAME stable score-descending order as `select_documents_tiered` (a document's rank is fixed by its FIRST seen chunk; dedupe by `document.id`), return at most *n* documents — **no floor** (a lexical-only hit with cosine 0.0 is suggested when ranked, unlike the phase-113 cited tier). `n` defaults to `get_settings().suggested_docs`. Docstring: the phase-118 "start here" suggestion tier — opt-in starting points, not citations (locked A3); the returned rows carry the full document content, byte-identical (the content is what `read` serves later — never truncated, A6).
|
||||||
|
2. `app/config.py` — new `suggested_docs: int = 5` (env `BOR_SUGGESTED_DOCS`), validator `>= 1` (the `related_max_docs` non-negative style, `app/config.py` ~L381–387), docstring citing the phase-118 "start here" seeding; `.env.example` entry with the same comment (near the `BOR_TOP_N_DOCS` / `BOR_RELATED_MAX_DOCS` block).
|
||||||
|
3. `tests/unit/test_config.py` — the setting default + the validator (0 and negative rejected, 1 accepted).
|
||||||
|
4. `tests/unit/test_retriever.py` — `select_suggested`:
|
||||||
|
- rank order = best fused score per document, first-seen-chunk rank (the same walk — a doc whose best chunk appears later in the list still ranks by first-seen position, matching `select_documents_tiered`'s order semantics);
|
||||||
|
- dedupe (multiple chunks of one doc → one row);
|
||||||
|
- the cap (`n` and the settings default);
|
||||||
|
- NO floor: a lexical-only 0.0-cosine document is included when ranked (contrast pin against `select_documents_tiered`'s floored cited tier on the same input);
|
||||||
|
- deterministic tie-break inherited from the fused order.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `select_suggested` returns the top-5 distinct docs in fused rank order for a typical fused list; the floor never filters.
|
||||||
|
- [ ] `BOR_SUGGESTED_DOCS` is env-tunable (default 5) and validated.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 03 — The "start here" suggestion blocks in the HIGH prompt
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach."
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The HIGH prompt's `<documents>` section seeds each suggested document as an identity + SUMMARY block (never the full content), with a pinned intro line carrying the start-here framing; a NULL summary (fail-soft miss) falls back to a 400-char content preview (locked A5).
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/prompts.py` — `build_high_prompt`: keep the `<documents>` section tag and the per-block `<document source=… path=… title=… date=…>` markup BYTE-STABLE (the E2E mock's `_document_block` parser and the `read`-tool copy key off both) — only the block BODY changes: `doc.summary.strip()` instead of `doc.content`. The `date` attribute stays (phase 106 D5, always present).
|
||||||
|
2. `app/rag/prompts.py` — the preview fallback (locked A5): when `doc.summary` is None or whitespace-only, the body is `doc.content[:limit]` + the shared `TRUNCATION_MARKER` on its own line — `limit` from `get_settings().suggestion_preview_chars`, read ONLY on that fallback path (the house pattern: `build_steering_section`/`build_kb_section` read settings inside the function; `build_high_prompt`'s signature is unchanged, every existing caller works, and a prompt built from summary-bearing docs makes no settings call for the cap). New setting in `app/config.py`: `suggestion_preview_chars: int = 400` (env `BOR_SUGGESTION_PREVIEW_CHARS`, validator `> 0`) + `.env.example` entry.
|
||||||
|
3. `app/rag/prompts.py` — new module constant `SUGGEST_INTRO` (pinned copy, the phase-15 `_STEERING_INTRO`/phase-31 `_KB_INTRO` precedent) rendered inside the `<documents>` section, BEFORE the first block, when at least one block is present. Copy (adjust wording freely, keep the three contracts below, pin the final text in `test_prompt_lock.py` or `test_prompts.py`): the blocks are SUMMARIES of the top-ranked documents for the question — start here if one seems right; call `read` with the document's combined `source/path` to add its FULL content before answering specifics; a document's full text is NOT in the prompt until you read it; cite the document(s) you used by path. The empty-body fallback line (`"(no documents matched — do not invent specifics)"`) stays.
|
||||||
|
4. `app/rag/prompts.py` — docstrings: the module docstring's HIGH-mode description ("full top-document texts under `<documents>`") → the summary-seed contract (A6 re-revised); `build_high_prompt`'s docstring updated likewise; the phase-106 D5 note (the `date` attribute) stays.
|
||||||
|
5. `app/rag/retriever.py` — module docstring: the A7-revised paragraph ("the LLM receives the entire relevant document… the full text of the top-N documents is always fed through, never truncated") → the A6 re-revised contract: the retrieval path seeds SUMMARIES (the suggestion tier, locked A3); full text enters the context only through the capped `read` tool; `select_documents`/`select_documents_tiered` docstrings get the same one-line pointer (the functions themselves are untouched — they still serve the related tier + any remaining callers).
|
||||||
|
6. `build_deflect_prompt` is UNTOUCHED (the LOW prompt + `DEFLECT_MODE` body stay byte-identical — the prompt lock pins it).
|
||||||
|
7. ASSUMPTION: the section keeps the name `<documents>` (mock + copy contract) rather than a new `<suggested>` tag — the tag is a stability surface, the intro line carries the semantic change.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `tests/unit/test_prompts.py` — a HIGH prompt built from docs with summaries: the block bodies are the summaries (sentinel: a full-content string that appears NOWHERE in the prompt; the summary text appears inside the `<document>` block), the intro line is present before the first block; the identity + `date` attributes byte-identical to today's format.
|
||||||
|
- `tests/unit/test_prompts_dates.py` — the `date` attribute pins survive the body change (update bodies, keep the attribute assertions).
|
||||||
|
- new: the preview fallback — a doc with `summary=None` → body = first 400 chars + `TRUNCATION_MARKER`; `summary=" "` → same; a short content (< 400) → the whole content, no marker; `BOR_SUGGESTION_PREVIEW_CHARS` env override honored.
|
||||||
|
- new: the LOW prompt is byte-identical to pre-task on the same inputs (sha-pin via `build_deflect_prompt` output or the existing lock).
|
||||||
|
- `tests/unit/test_prompt_lock.py` — if `SUGGEST_INTRO` is added as a constant, pin it there (sha + prefix); `PERSONA`/LOW-body anchors untouched.
|
||||||
|
- `tests/unit/test_config.py` — the new setting.
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A grounded prompt built from 5 summary-bearing docs contains all 5 summaries + the intro, and ZERO full-content characters beyond the preview cap for NULL-summary docs.
|
||||||
|
- [ ] The LOW prompt output is byte-identical to pre-phase for identical inputs.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompts_dates.py tests/unit/test_prompt_lock.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 04 — `read` on a suggested doc adds its full text + the rewritten `read` copy
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content…" (suggested = "for reading" — the model reads what its summaries point at.)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`read` on a suggested document is allowed and adds the document's full text (all existing read mechanics unchanged); the dedupe refusal fires only for an already-READ document; the `read` tool description and the `TOOLS_SECTION` `read` clause are rewritten for the summary-seed mode; the prompt-lock anchors are re-cut.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — `_execute_tool`'s `read` branch (~L1131–1136): `known = {(doc.source, doc.path) for doc in holder.read_docs}` — the `seed_docs` fall out of the dedupe set (a suggested doc is a summary in the prompt, NOT its full text). A first `read` of a suggested doc now succeeds through the EXISTING path unchanged: `holder.read_docs.append(doc)`, `holder.tool_calls += 1`, the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `holder.read_truncations` (→ the SSE `tool_result` frame), the `Document {source}/{path}:` header + phase-106 D5 `date:` line. A second `read` of the same (now full-text-in-context) document still gets `ALREADY_IN_CONTEXT` — its copy stays byte-identical (it already names the correct action: answer from the text in the prompt). The `seed_docs` parameter of `_execute_tool`/`run_agent` stays (callers unchanged) but its docstring + the `run_agent` docstring paragraph "``seed_docs`` are the documents the retrieval already put in context… re-reading one of them is rejected" are rewritten: the seeds are SUMMARY blocks in the prompt; reading one adds its full text; only an already-READ doc is refused.
|
||||||
|
2. `app/rag/agent.py` — `AGENT_TOOLS` `read` function description: rewrite the leading contract sentences — the `<documents>` section shows the SUMMARIES of the top-ranked documents (their full texts are NOT in the prompt); use `read` to add one of them (or any other document) by its combined `source/path` string, exactly as shown in the `ls` output / the `<documents>` blocks; do not re-read a document you have already read — its full text is already in your prompt. KEEP byte-identical: the combined-`source/path` identity contract, the bare-path-will-not-resolve sentence, the truncation-notice paragraph, the "call one tool at a time" sentence, and the `path` parameter description (minus its "Only pass a document NOT already shown in the `<documents>` section" clause → "Do not re-read a document you have already read").
|
||||||
|
3. `app/rag/prompts.py` — `TOOLS_SECTION`: rewrite ONLY the `read` clause the same way (today's "Do not call `read` for a document already shown in the `<documents>` section, even when the user asks you to open or read it — its full text is already in your prompt; answer directly from it." becomes: the `<documents>` section holds SUMMARIES — the full text of a suggested document is not in the prompt until you `read` it; do not re-read a document you have already read — its full text is already in your prompt; answer directly from it. If the user asks you to open or read a suggested document, `read` it — that is the point of the section.) KEEP byte-identical: the `ls` clause (the phase-94 drill-down contract), the `grep` clause, and the discipline rules (one call per reply, never repeat a refused/succeeded call, answer as soon as you have what you need).
|
||||||
|
4. `tests/unit/test_prompt_lock.py` — re-cut the `TOOLS_SECTION` anchors for the new copy: `TOOLS_SECTION_SHA256` + `TOOLS_SECTION_LEN` ALWAYS change; `TOOLS_SECTION_PREFIX`/`SUFFIX` only if the new copy touches those exact boundaries (the prefix is the `ls`-clause opening and the suffix is the discipline-rules ending — both stay, so they should survive; verify, don't assume). Update the "Pre-phase-112 anchors" comment to cite the phase-118 re-cut (A6, owner directive 2026-09-15). `PERSONA` and the LOW-body anchors stay untouched — if they fail, the change leaked into locked text and the task is wrong.
|
||||||
|
5. `app/rag/agent.py` — module docstring: the "A7 scope clarification" paragraph (~L120, "the top-2 seed documents stay whole — 'this should never happen'") → the A6 re-revised contract (seeds are summaries; the `read` path is the only full-text path and keeps its cap); point 3's seed-read refusal teaching → the already-read refusal.
|
||||||
|
6. ASSUMPTION: the `ALREADY_IN_CONTEXT` constant text is unchanged (it is accurate for the already-read case and the E2E mock/telemetry key on stable copy).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `tests/unit/test_agent.py` — update/extend:
|
||||||
|
- a `read` of a `seed_docs` document SUCCEEDS: the full-content result (header + date line), `holder.read_docs` length 1, `holder.tool_calls` 1;
|
||||||
|
- a second `read` of the same document → `ALREADY_IN_CONTEXT` (byte-identical line), counters untouched;
|
||||||
|
- the cap path on a suggested doc (content > `read_max_chars`) → truncated result + notice + `read_truncations` entry (existing pins re-targeted from "seed" to "suggested");
|
||||||
|
- the `AGENT_TOOLS` `read` description pins: the new summary contract present, the identity/truncation/one-call sentences byte-present;
|
||||||
|
- `ls` and `grep` behaviors byte-identical (their existing pins untouched).
|
||||||
|
- `tests/unit/test_prompts.py` — the `TOOLS_SECTION` inside a built HIGH prompt carries the new `read` clause; `ls`/`grep` clauses byte-identical (sha-pin the unchanged clauses if the file's style allows).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `read('source/path')` on a suggested doc adds its full text to the context and counts as an executed call; a repeat is refused.
|
||||||
|
- [ ] `PERSONA` + the LOW `DEFLECT_MODE` body byte-locks pass without re-cutting; only the `TOOLS_SECTION` anchors moved.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Task 05 — `plan_turn` wiring: suggested seeding, citations, the durable record
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content… That way the LLM responds faster…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`plan_turn` seeds the HIGH branch with the top-5 suggested documents (summary blocks) instead of the top-2 full texts; the related tier becomes rank 6+; citations (`done.sources`) = suggested + agent-read; the durable record (`query_log` + log line) = suggested + related + read with a new `suggested=N` field; the LOW/deflection path stays byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/retriever.py` — new pure helper `select_related(chunks, excluded_ids: set[uuid.UUID], cap: int) -> list[Document]`: the SAME stable score-descending walk as `select_documents_tiered` (first-seen-chunk rank, dedupe by id), skipping documents in *excluded_ids*, no floor, at most *cap* documents. (The phase-113 related tier was "ranked after the cited tier"; with a contiguous top-5 suggestion set, "excluding the suggested" is exactly "rank 6+".) Unit-test it here or in task 05's tests — either location, one set of tests.
|
||||||
|
2. `app/api/chat.py` — `plan_turn`: replace the `select_documents_tiered(chunks, n=settings.top_n_docs, floor=…, related_cap=…)` call with:
|
||||||
|
```python
|
||||||
|
suggested = select_suggested(chunks) # BOR_SUGGESTED_DOCS (5), no floor (A3)
|
||||||
|
related_docs = select_related(chunks, {d.id for d in suggested}, settings.related_max_docs)
|
||||||
|
```
|
||||||
|
Both branches then use `suggested`/`related_docs`: the HIGH branch → `build_high_prompt(suggested, notes=…, kb_overview=…)`; the LOW branch → `build_deflect_prompt(titles, …)` UNCHANGED (weak-hit titles only). `summary_hits` = the hit chunks with `is_summary` whose parent document id is in the SUGGESTED set (redefined from the cited set — the counting code shape is unchanged).
|
||||||
|
3. `app/api/chat.py` — `TurnPlan`: rename field `docs` → `suggested_docs` (SAME dataclass position; update every attribute access). Docstrings: the field holds the summary-seeded suggestion tier (A6); `related_docs` docstring updated (rank 6+ after the suggested set).
|
||||||
|
4. `app/api/chat.py` — the `chat` endpoint's post-stream step: `run_agent(..., seed_docs=plan.suggested_docs, …)`; `cited_docs` loop over `[*plan.suggested_docs, *holder.read_docs]` (LOCKED A4 — the dedupe pattern is byte-identical); `record_docs` over `[*plan.suggested_docs, *plan.related_docs, *holder.read_docs]` (LOCKED A3 — the durable record keeps the full retrieval); `source_paths` unchanged.
|
||||||
|
5. `app/api/chat.py` — the per-turn log line (PLAN §9): add `suggested=%d` (`len(plan.suggested_docs)`) after `summary_hits=%d`; the field order of every existing field is untouched (the phase-114 `retries=%d scaffold_stripped=%d` tail stays last).
|
||||||
|
6. `app/config.py` — `top_n_docs` and `source_usefulness_floor` lose their seeding role (plan_turn was their only `app/` consumer — verified). KEEP both settings (env back-compat) and keep `select_documents`/`select_documents_tiered` (dormant public helpers): one-line docstring note on each — "phase 118 retired the full-text seeding role (A6); the suggested tier (`select_suggested`) seeds the prompt now." No deletions.
|
||||||
|
7. ASSUMPTION: the `done.related` UI row (the phase-113 de-emphasized "nearby docs") keeps showing rank 6+ under the new tiering — the field name, shape and dedupe-vs-cited rule are unchanged (a read related doc stays a citation, never a "nearby doc").
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- `tests/unit/test_chat_gate.py` — update/extend:
|
||||||
|
- HIGH branch: the prompt contains the 5 suggested docs' SUMMARY text and NONE of their full content (sentinel); the related tier = rank 6+ (a 7-doc fixture: related = docs 6–7, capped by `related_max_docs`);
|
||||||
|
- the deflected branch: prompt byte-identical to pre-task on the same chunks (sha-pin the output), suggestions/chips unchanged, `deflected=True`;
|
||||||
|
- BOTH branches' TurnPlan carry suggested + related (the durable-record inputs);
|
||||||
|
- `summary_hits` redefinition (a summary chunk whose parent is in the suggested set counts; one whose parent is related-only does not);
|
||||||
|
- the log-line format pin (search the suite for the `summary_hits=` format string) gains `suggested=N` in the right slot;
|
||||||
|
- settings: `suggested_docs`/`related_max_docs` honored; `top_n_docs`/`source_usefulness_floor` no longer consulted by `plan_turn` (behavioral pin, not a removal pin).
|
||||||
|
- `tests/integration/test_chat_api.py` (and any sibling pinning the done-event sources) — `done.sources` = suggested + read (deduped) on a grounded turn with a mock `read` flow; `[]` on deflection; `done.related` = rank 6+ minus cited; `query_log.sources` = suggested + related + read.
|
||||||
|
- `tests/unit/test_retriever.py` — `select_related` (walk order, exclusion, cap, no floor, determinism).
|
||||||
|
- Coverage: **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A grounded turn's system prompt seeds exactly the top-5 suggested summaries; `run_agent` gets them as `seed_docs`.
|
||||||
|
- [ ] `done.sources` = suggested + read (deduped, order-preserving); `query_log.sources` + the log line record suggested + related + read; `suggested=N` lands after `summary_hits=N`.
|
||||||
|
- [ ] A deflected turn is byte-identical in prompt and frames to pre-phase.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_chat_gate.py tests/integration/test_chat_api.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 06 — E2E: the new summary-seed suite, the phase-24 E2E retired, old pins updated
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant…"
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Playwright coverage of the summary-seed contract (summaries in the prompt, full text only via `read`, markdown docs summarized, citation chips = suggested + read); the retired phase-24 full-text E2E deleted; every E2E/integration pin of the retired behavior re-targeted to the new contract.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. NEW `tests/e2e/test_summary_seed_context.py` (run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`; DB up: `podman compose up -d db`). Pattern: the sibling suites' deterministic seeding (the mock LLM's bag-of-words `embed_text` + `import_sources` or the sync API — see `tests/e2e/test_read_truncation_cap.py`'s token-controlled seeding and `tests/e2e/test_whole_document_context.py`'s direct SQLAlchemy seeding; the mock's `SUMMARY_MODE` digest makes every stored summary deterministic, markdown included). Token-controlled bodies so the hybrid gate picks the intended suggested set deterministically. Cases:
|
||||||
|
- **(a) summaries seeded**: the `END_OF_NOTES_TRIGGER` tail-echo question → the answer quotes the last 160 chars of the seeded `<documents>` context, which now end in the LAST suggested document's SUMMARY — pin the mock's summary digest text (the phase-30 digest shape), proving summaries reached the prompt.
|
||||||
|
- **(b) full text NOT seeded**: the inverse of the phase-24 pin — a sentinel on a suggested document's LAST LINE is absent from the echoed context (the full content never reached the model pre-`read`).
|
||||||
|
- **(c) `read` adds the full text**: a scripted mock tool flow `read`s a suggested document (the existing `_tool_flow` read-echo pattern, cf. `test_ls_tree_drilldown.py`) → the answer quotes the full-text tail from the read RESULT (the sentinel now appears, sourced from the read, not the seed).
|
||||||
|
- **(d) markdown gets a summary**: a markdown document in the KB appears as a suggestion block carrying the mock `SUMMARY_MODE` digest (not a content preview) — the locked A2 end-to-end proof.
|
||||||
|
- **(e) citation chips**: after case (c), the UI chip row = suggested + the read document (deduped — the read doc is among the suggested, so chips = the suggested set), each a `done.sources` entry; `done.related` renders the de-emphasized row (rank 6+) when the seeded KB is large enough to have one.
|
||||||
|
2. `tests/e2e/test_whole_document_context.py` — DELETE (the retired full-text-seed contract; locked A1/A6 — no kill switch to re-scope it to).
|
||||||
|
3. `tests/e2e/test_ls_tree_drilldown.py` — the `READ_ANSWER_PREFIX` flow (~L61–67, L229–234, L660–666): today the scripted read targets a top-2 seed and is REFUSED (`ALREADY_IN_CONTEXT`), the mock answering from the `<documents>` full text. Re-target: the scripted read target stays, but the read now SUCCEEDS (seeds are summaries) and the mock answers from the READ RESULT with the same citation shape — update the prefixes/quotes to key off the read result (the full text, first 80 chars — the content is now delivered by the read, so the quote is byte-identical to today's) and the comments to the new contract. If the mock's answer-from-`<documents>` branch (its `ALREADY_IN_CONTEXT`-keyed flow, `mock_llm.py` ~L237) is no longer exercised by ANY suite, keep it (the already-read refusal is still real) but update its docstring (it now describes the already-read case, not the top-2-seed case).
|
||||||
|
4. `tests/e2e/test_read_truncation_cap.py` — the setup guard (~L364–379) that asserts the scripted read target is NOT a top-2 seed (its only reason was the seed-read refusal) — REMOVE the guard and its docstring; a seed-target read now executes, so the cap fires either way.
|
||||||
|
5. `tests/e2e/test_document_summaries.py` — the phase-30 pins: `summary.summaries == 1` for the yaml + md-control pair (~L169) → `== 2` (markdown summarized too, locked A2); the "md control has no summary" assertions → inverted to "the md control HAS the mock digest summary"; the summary-chunk count pins gain the md doc's chunk.
|
||||||
|
6. `tests/e2e/mock_llm.py` — extend ONLY if a case above needs a new trigger; the marker contracts stay (`<tools>` presence, `DEFLECT_MODE`, `_READ_RESULT_PREFIX` header, `END_OF_NOTES_TRIGGER`, `SUMMARY_MODE`). Update the docstrings that describe the top-2-seed/`ALREADY_IN_CONTEXT` scenario (~L235–241).
|
||||||
|
7. Audit sweep (markers, in `tests/e2e/`, `tests/integration/`, `tests/unit/`): `ALREADY_IN_CONTEXT`, "already shown in the `<documents>`", "top-2 seed", full-content-in-prompt sentinels, `summary.summaries` counts, the per-turn log-line format (`summary_hits=` — gains `suggested=N`). Every hit: pin of RETIRED behavior → re-target to the new contract; pin of UNCHANGED behavior → leave. `tests/unit/test_agent.py`'s seed-read pins were already updated in task 04 — do not duplicate.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The new suite green in isolation (the command above); every re-targeted suite green; full suite green.
|
||||||
|
- Coverage: **>90%** on `app/` (E2E is uncounted; keep the unit coverage from tasks 01–05 intact).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov` green (cases a–e).
|
||||||
|
- [ ] `tests/e2e/test_whole_document_context.py` no longer exists; `tests/e2e/test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`, `test_document_summaries.py` green with the new-contract pins.
|
||||||
|
- [ ] The audit sweep left no test pinning the retired full-text-seed or seed-read-refusal behavior.
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Task 07 — The fixture-battery gate re-run (real model) + verdict recorded
|
||||||
|
|
||||||
|
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…That way the LLM responds faster…" (the summary-seed behavior must be proven against the real configured chat model before the phase completes — the phase-70/72/94 tool-copy gate precedent.)
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The fixture KB dump is rebuilt so every fixture document (all eight are markdown) carries a stored summary — the phase-118 end state — and the real-model fixture battery passes against it under the locked gate semantics (locked A7), with the verdicts recorded in `TOOL_CALLING_TESTING.md`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. Rebuild the fixture dump — `uv run python -m scripts.load_test_kb` (the real pipeline: `import_sources` now summarizes every document, task 01 — the eight markdown fixture docs gain `doc.summary` + one embedded `is_summary` chunk each, so the dump's documents/chunks content changes; a rebuild is required, this is a legitimate re-run trigger — "the import pipeline's summary policy changed (phase 118)"). Preconditions: DB up, the aipi `embed` + `lite` endpoints reachable (`.env`). Verify from the build output: `summary_errors=0`, zero failed folder-summary batches, and the retrieval report's battery all-GROUNDED (the honesty gate is untouched by this phase, so grounded/deflected should be unchanged; if a question now deflects, it is a fixture-content regression — fix the fixture anchor per the build's own warning, never the gate). Commit the regenerated `tests/fixtures/test_kb.dump.sql` with the phase (it is tracked).
|
||||||
|
2. Run the gate — `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` (the fast loop: inline restore + the curated battery against the live endpoint, the configured chat model). Use `--turns N` only for the copy-iteration micro-loop while refining task 04's `read` copy; the VERDICT run is always the full battery.
|
||||||
|
3. Interpret the verdict under locked A7 (owner-confirmed 2026-09-15): the phase gate = condition 1 (all turns answer) AND condition 2 (zero round-cap hits) AND condition 4 (fixture-mode contract accuracy ≥ 0.90) — the script prints each; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is the intended fast path now, not tool abandonment. If the script exits 1 on condition 3 alone with 1/2/4 green, the phase gate is PASS — record exactly that (the four metrics + the wall time + the `contract X/Y` line, the TOOL_CALLING_TESTING.md verdict format).
|
||||||
|
4. Copy iteration (only if condition 4 degrades or condition 2 is hit): refine the `read` description (`app/rag/agent.py` `AGENT_TOOLS`) / `TOOLS_SECTION` clause (`app/rag/prompts.py`) — re-cut the `test_prompt_lock.py` `TOOLS_SECTION` anchors again if the copy moves — micro-loop with `--turns`, then a full verdict run. `classify_call` needs NO re-baselining: a `read` of a suggested document is a well-formed call targeting a resolvable document — contract-correct under both old and new app semantics (its docstring's `ALREADY_IN_CONTEXT` note already excludes dedupe re-reads from the metric).
|
||||||
|
5. `TOOL_CALLING_TESTING.md` — append a dated section ("Phase 118 — summary-seed gate, 2026-09-15…"): the four conditions + metrics, the contract line, the wall time, the model name, the dump's rebuild note (summaries now on all eight docs), the locked-A7 semantics note (condition 3 reported, not gated), and the `read` copy's final shape (or a pointer to it). The controlled-methodology sections above stay untouched (append-only history).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- The gate is the test: no unit suite here; the rebuild's own round-trip verification (dump restore + per-table checksum) must pass, and the full `uv run pytest` must stay green after the dump regeneration (E2E suites that restore the fixture dump pick up the summary rows — if any suite pins the old 9-chunk fixture state, that pin moves to the new 17-chunk state: 9 content + 8 summary chunks).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `tests/fixtures/test_kb.dump.sql` regenerated with all eight fixture docs summarized (round-trip checksum verified by the build).
|
||||||
|
- [ ] Full-battery verdict run recorded: conditions 1, 2, 4 GREEN (condition 3 reported); the telemetry + semantics note appended to `TOOL_CALLING_TESTING.md`.
|
||||||
|
- [ ] `uv run pytest` green with the new dump (including any fixture-chunk-count pins updated).
|
||||||
|
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
# Phase 119 — Name-signal retrieval + suggested-folder context; chips cite read docs only
|
||||||
|
|
||||||
|
**Source:** owner live report, 2026-09-16 — brain.reeseapps.com, turn "How do I deploy gitea?". Three findings, all validated live and reproduced against the 1,000-document live replica (restored 2026-09-16 from `data/bor_live_kb_replica.dump.sql`):
|
||||||
|
|
||||||
|
1. **Irrelevant documents in the chip cards** — the turn's 7 chips were 5 suggested docs + 2 agent-read docs (phase-118 locked A4); 2 of the 5 suggestions were off-topic (`homepage/src/index.html`, `deploy/Deployments/reeseapps/gateway/README.md`) and 3 were never read. The agent's one *failed* read (`docker-compose.yaml` — verified absent from the live KB, teaching refusal) correctly earned no chip, so the chip surface itself had no construction bug — it faithfully showed a noisy suggestion tier.
|
||||||
|
2. **The agent lists instead of reading the proposed files** — the 5 summaries seeded into the prompt did not include the canonical `deploy/Deployments/reeseapps/gitea/README.md`, so the model rationally explored: 3 `ls` drill-downs (`deploy` → `reeseapps` → `gitea`), one wrong-guess `read` (refused), one more `ls`, one `read` — 7 tool rounds. It never read a single suggested file, and its answer's "Docs used:" line cited `gitea-web.env.j2`, a file it never read (confabulation from the suggestion blocks being in context).
|
||||||
|
3. **Suggested documents off** — reproduced on the replica (which lacks the phase-118 markdown summary chunks; live was *worse*: `index.html` ranked #1 there):
|
||||||
|
|
||||||
|
| Question (replica, pre-phase) | Suggested top-5 | Canonical doc |
|
||||||
|
|---|---|---|
|
||||||
|
| "How do I deploy gitea?" | gitea playbook, **gateway README**, k8s gitea-values ×2, container_gitea.md | ❌ rank 11 |
|
||||||
|
| "How do I deploy forgejo?" | foobar.md, AGENTS.md, reeseapps README, inventory.yaml, bifrost.md | ❌ **zero forgejo docs** |
|
||||||
|
| "How do I set up the gateway?" | shelly.md, forgejo.network, framework_16.md, firewall playbook, home_assistant.md | ❌ not in top-5 |
|
||||||
|
|
||||||
|
**Root causes.** (a) The lexical side never sees document *names*: `chunks.tsv` is `to_tsvector('english', content)` — body only (migration 0002) — and the name-hit mechanism (the 2026-09-05 versioned-name incident fix, `app/rag/retriever.py::_name_hit_chunks`) fires **only for digit-bearing** question tokens, so `gitea`/`forgejo`/`gateway` (no digits) get no name signal; "forgejo" appears in only 26 chunk bodies vs 136 for "gitea", so an OR-tsquery dominated by the common token "deploy" buries the name. (b) Phase 118 widened the blast radius: suggestions = top-5 **no floor** (118-A3) and chips = suggested + read (118-A4), so every noisy suggestion became a visible junk chip; the new per-document summary chunks also created hub attractors (site-wide summaries vector-match almost anything). (c) The prompt teaches "start here **if one seems right**" + "answer as soon as you have what you need" — with a suggestion set missing the canonical doc, exploration (and its latency) is the rational model behavior.
|
||||||
|
|
||||||
|
**Story:** n/a (owner-directed RAG behavior change; extends the phase-118 summary seeding, the 2026-09-05 name-hit mechanism, and the phase 94/113 tier assets).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Product-name questions surface the product's own documents in the suggestion tier (component name-hits + a bounded, kill-switchable name-hit bonus); the HIGH prompt pre-seeds each suggested document's parent-folder contents (collapsing the `ls` drill-downs the agent currently pays); the citation chips carry **only documents the agent actually read** (owner decision 2026-09-16 — phase-118 A4 retired); and the model is taught to cite only what it read or used.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `118_summary_seed_context` (complete) — the summary-seed prompt, `select_suggested`/`select_related`, the `read`-extends-context contract, and the done-frame tiering this phase re-locks.
|
||||||
|
- `117_tool_call_compact` (complete) — the current `read`/`ls`/`grep` copy this phase leaves byte-identical.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — name-hit rule (task 01, LOCKED A2).** `name_hit_tokens` returns **all** normalized whitespace tokens of length ≥ 4 (dotted tokens kept whole, as today) *plus* the versioned-name join candidates — the digit filter moves from the token list to the match side. `_name_hit_chunks` matches a token against the document's **path** (never the title — titles are prose: `deploy/Deployments/reeseapps/README.md` is titled "Deployments", and the common token "deploy" must not name-match it or the 9 other deployment-titled docs; the owner-verified failure mode of the naive relaxation):
|
||||||
|
|
||||||
|
- **digit-bearing token** → PREFIX of a normalized path part or file stem (preserves the 2026-09-05 incident fix: `qwen38` → `qwen3.8-27b-juggernaut-vulkan.container`);
|
||||||
|
- **digitless token** → EQUALS a normalized path part, a file stem, or a stem sub-component (the stem split on non-alphanumeric runs: `kubernetes_gitea` → `kubernetes`/`gitea`, `gitea-values` → `gitea`/`values`, `test-gateway` → `test`/`gateway`).
|
||||||
|
|
||||||
|
Measured effect (replica, 2026-09-16, replacing the digit-only rule with this one): "forgejo" → all 5 suggested slots are forgejo-tree docs (README #1); "gitea" → 4 of 5 slots gitea-tree incl. the canonical README; "gateway" → 4 of 5 slots gateway-tree. Ranking of name hits: (distinct matched token count desc, then catalog order `(source, path)`) — the total-matched-length tie-break is RETIRED (it systematically outranked 5-char product names by 6-char common tokens). `NAME_HIT_LIMIT` stays 10. Name-hit rows keep the existing `fts_hit=True`, `cosine=0.0` convention (A8 already treats them as a lexical signal needing vector corroboration). `RetrievedChunk` gains `name_hit: bool = False` (dataclass field, default keeps every existing construction valid) so the selection tier can tell them apart.
|
||||||
|
|
||||||
|
**D2 — name-hit bonus (task 02, LOCKED A3).** A bounded, kill-switchable **selection-time** re-rank — the phase-106 recency-boost pattern (additive, single apply site, `0` = off → byte-identical ordering, negative fails startup loudly). New setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**). In the document-selection walks only (`select_suggested`, `select_related`, `weak_hit_titles`), a document's effective selection score = its best fused chunk score **+ bonus if any of its chunks is a name hit**. Chunk scores, `fuse()`, `retrieve()`, the A8 honesty gate (reads chunk cosine/fts), and `query_log.top_score` are untouched. 0.005 ≈ a 2–4 rank head start on the RRF scale (rank 1 vs 5 in one list ≈ 0.0010); measured, it is the difference that puts the canonical doc inside the top-5 on all three failing questions (post-phase suggested top-5: gitea → playbook, gitea-postgres.env.j2, gateway README, **gitea README #4**, gitea-web.env.j2; forgejo → 5/5 forgejo; gateway → shelly.md, istio test-gateway.yaml, **gateway README #3**, gateway certbot/ddns playbooks). The 0.005 default is an owner-tunable starting point, not a calibrated constant — the battery (task 06) records the realized margins. 118-A3 is **re-revised** by this lock: the suggestion set is still top-5 distinct docs with NO cosine floor, now walked with the name-hit bonus. `scripts/eval_retrieval.py` gains a `suggested:` line (the `select_suggested` walk over the fused list, bonus included) so the tuning tool reports the tier the prompt actually seeds.
|
||||||
|
|
||||||
|
**D3 — suggested-folder context (task 03, LOCKED A4).** The HIGH prompt's `<documents>` section carries, **after `SUGGEST_INTRO` and before the first `<document>` block**, one line per DISTINCT parent folder of the suggested documents (in suggested-doc order, deduped by `(source, parent prefix)`, at most 5 lines): `<source>/<parent>/: <entry, …>` — subfolders as `name/ (N docs)`, files by relative filename, entry order = the existing `ls` folder-level rendering order (`group_folder_listing` in `app/rag/agent.py` — the line must read the same as the model's own `ls` output), capped at 8 entries per line + ` +N more`; the source root renders as `<source>/: …`. The suggested document itself is excluded from its folder's entries (its identity is already in the block). Source: a new pure helper `suggested_folder_lines(db, suggested, …)` next to the `ls` helpers in `app/rag/agent.py`, reusing `_source_document_rows` + `group_folder_listing`; computed in the chat endpoint (which holds the session) and passed through `plan_turn` (new parameter, default empty) → `build_high_prompt` (new parameter, default empty). The `<document>` block markup AND body stay **byte-identical** (the E2E mock's tail echo keys off the LAST block's tail — the folder lines must never land after a summary); the LOW prompt stays byte-identical; with an empty `folder_lines` the HIGH prompt is byte-identical to the phase-118 output. The live-turn payoff: the README sits in the parent folder of suggested `gitea/app/gitea-web.env.j2`, so its name appears in the prompt — the model reads it in round 1 instead of walking three `ls` levels.
|
||||||
|
|
||||||
|
**D4 — cite discipline (task 04, LOCKED A5).** `SUGGEST_INTRO`'s final sentence ("Cite the document(s) you used, by path.") is REPLACED with: `Cite only the document(s) you read — or, if you answered from a suggested summary without reading it, cite that suggested document — never a document you neither read nor used.` `PERSONA`, the LOW/`DEFLECT_MODE` body, and `TOOLS_SECTION` stay byte-identical (the phase-117 copy is untouched — no tool-copy gate re-run, D6); the byte-pinned anchors re-cut in the same task.
|
||||||
|
|
||||||
|
**D5 — chips cite read docs only (task 05, LOCKED A1 — owner decision 2026-09-16, phase-118 A4 retired).** `done.sources` = `holder.read_docs` only, deduped by `(source, path)`, read order preserved. A grounded turn on which the agent reads nothing (the phase-118 summary-only fast path) shows **no chips** — an accepted, owner-directed consequence (the answer prose still names the doc it used, per D4). The DURABLE record is untouched (118-A3 stands: `query_log.sources` + the per-turn log line = suggested + related + read); the `related` tier (the de-emphasized "nearby docs" row) is untouched except its existing dedupe against the cited set now keys on read docs only (a read related doc is a chip, never a nearby doc — unchanged intent). No frontend change: `appendSources`/`addSources` already render zero entries as no DOM.
|
||||||
|
|
||||||
|
**D6 — verification (task 06).** New dedicated Playwright suite `tests/e2e/test_name_signal_read_chips.py` (run in isolation; the `test_summary_seed_context.py` local-directory-source fixture pattern), a mock-LLM trigger echo making the folder lines observable, the retired-A4 pins re-targeted, and the empirical battery: the 1,000-doc live replica (the `restore-test-db` skill; the dump is a local gitignored artifact — if missing, STOP and ask the owner, never fabricate numbers) + `scripts/eval_retrieval --from-file` over a tracked question battery, + the real-model fixture run recorded as TELEMETRY in `TOOL_CALLING_TESTING.md` (the gate is NOT re-triggered — no `AGENT_TOOLS`/`TOOLS_SECTION` copy change; conditions, were they read, follow the phase-118 A7 semantics).
|
||||||
|
|
||||||
|
**Invariants (all tasks).** The A8 honesty gate, every SSE frame shape, the LOW prompt, `PERSONA`, `TOOLS_SECTION`, the `<document>` block markup/body, the `Document {source}/{path}:` read-result header, and the per-turn log line format (fields unchanged; `sources=`/`suggested=` values move with the data) are byte-identical. `PLAN.md` untouched (Protocol B — the owner folds the 118-A4 retirement + 118-A3 re-revision in at the next redo; both are owner-confirmed below).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_name_hit_component_match.md` — retriever: the component name-hit rule (D1) + the `name_hit` flag.
|
||||||
|
2. `02_name_hit_bonus.md` — the bounded selection-time bonus (D2) + `eval_retrieval` suggested line.
|
||||||
|
3. `03_suggested_folder_context.md` — the prompt's suggested-folder lines (D3).
|
||||||
|
4. `04_cite_discipline_copy.md` — `SUGGEST_INTRO` cite discipline + the re-cut pins (D4).
|
||||||
|
5. `05_chips_read_only.md` — `done.sources` = read docs only (D5).
|
||||||
|
6. `06_e2e_retarget_and_battery.md` — the dedicated E2E, the retired pins, the mock echo, the replica battery + real-model telemetry (D6).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_retriever.py` (D1 rule incl. the "deploy"-title precision case and the versioned prefix; D2 bonus lift/kill-switch/validator), `tests/unit/test_config.py` (the new setting), `tests/unit/test_agent.py` or a new `tests/unit/test_folder_context.py` (D3 helper: root/nested/exclusion/dedupe/caps), `tests/unit/test_prompts.py` + `tests/unit/test_prompt_lock.py` (D3 line placement + the D4 re-cut; LOW + PERSONA byte-identical), `tests/unit/test_chat_gate.py` (D5 done-frame + durable record).
|
||||||
|
- E2E: `tests/e2e/test_name_signal_read_chips.py`, **in isolation** (`uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov`); the regression suites green (the re-targeted phase-113/118 pins included).
|
||||||
|
- Battery (task 06): the three name questions put their canonical doc in the printed suggested top-5; the out-of-KB question's gate verdict is unchanged; full output recorded in `06_…md`; real-model fixture telemetry recorded in `TOOL_CALLING_TESTING.md`.
|
||||||
|
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] "How do I deploy gitea/forgejo?" and "How do I set up the gateway?" (1,000-doc replica) put the canonical doc in the suggested top-5 (eval battery, recorded); "Qwen 3.8 27b" incident question and the out-of-KB deflection sanity are unchanged.
|
||||||
|
- [ ] A grounded prompt's `<documents>` section carries the suggested-folder lines (block markup/body and the LOW prompt byte-identical).
|
||||||
|
- [ ] `done.sources` = read docs only (a zero-read grounded turn chips nothing; the related row and the durable record are untouched); the unit + E2E pins agree.
|
||||||
|
- [ ] The new E2E green in isolation; the re-targeted old pins green; `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — chips (done.sources) = agent-read documents only** (deduped by `(source, path)`, read order); phase-118 A4 (suggested + read) is RETIRED; a zero-read grounded turn shows no chips (owner decision, 2026-09-16, stated consequence).
|
||||||
|
- **A2 — the name-hit rule**: length-≥4 normalized question token; digit-bearing → prefix of a normalized path part/stem; digitless → exact equality with a path part, stem, or stem sub-component (split on non-alnum); **titles are never name-matched**; ranking (matched-token count, catalog order) — the length tie-break retired; `NAME_HIT_LIMIT` stays 10 (owner-confirmed 2026-09-16, from the measured D1/D2 evidence above).
|
||||||
|
- **A3 — the name-hit bonus**: `BOR_NAME_HIT_BONUS` default 0.005, additive at selection time only, `0` = byte-identical kill switch, negative fails startup loudly; 118-A3 re-revised (top-5, NO floor, bonus walk); the A8 gate / chunk scores / `query_log.top_score` untouched (owner-confirmed 2026-09-16).
|
||||||
|
- **A4 — the suggested-folder context**: lines after `SUGGEST_INTRO`, before the first block; ≤5 lines, ≤8 entries + ` +N more` per line; `<document>` block markup/body and the LOW prompt byte-identical (owner-confirmed 2026-09-16).
|
||||||
|
- **A5 — the cite-discipline sentence** replaces `SUGGEST_INTRO`'s final sentence verbatim as written in D4; `PERSONA` + LOW body + `TOOLS_SECTION` byte-identical (owner-confirmed 2026-09-16).
|
||||||
|
- **A6 — PLAN.md untouched (Protocol B)**: the 118-A4 retirement and 118-A3 re-revision are recorded here (owner-confirmed) and folded into the anchor table at the next redo (the phase-118 A6 precedent).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ scripts/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): component name-hits + bounded name-hit bonus for product-name questions, suggested-folder context in the grounded prompt, and chips that cite read documents only"
|
||||||
|
```
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 01 — The component name-hit rule (phase-119 D1, LOCKED A2)
|
||||||
|
|
||||||
|
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D1 — the 2026-09-16 live finding: product-name questions ("How do I deploy forgejo?") get no lexical name signal because the name-hit mechanism fires only for digit-bearing tokens and `chunks.tsv` indexes body text only.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`_name_hit_chunks` in `app/rag/retriever.py` matches question tokens against document **paths** under the new two-class rule (digit-bearing → prefix; digitless → exact component/stem/sub-component), titles never, ranked by (matched-token count, catalog order); `RetrievedChunk` gains a `name_hit` flag.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/retriever.py` — `name_hit_tokens(question)`: return ALL normalized whitespace tokens of length ≥ 4 (the existing `_TOKEN_RE` dotted-token handling, `_normalize_name`) **plus** the existing versioned-name join candidates (a purely-numeric second token), order of first appearance, de-duplicated — the digit filter is REMOVED from this function (the digit distinction moves to the match side). Update its docstring: the token list is class-agnostic; matching applies the A2 rule.
|
||||||
|
2. `app/rag/retriever.py` — `_name_hit_chunks`: for each document (catalog order, the existing single projection query over `(id, source, path, title)` — **title is selected but never matched**), compute the name parts: for every path part — the normalized part, the normalized file stem, and the stem's sub-components (the stem lowercased, split on `[^a-z0-9]+`, each piece normalized, empties dropped). A token matches when:
|
||||||
|
- it contains a digit → it is a PREFIX of a normalized part or stem (`qwen38` → `qwen3.8-27b-juggernaut-vulkan`);
|
||||||
|
- it has no digit → it EQUALS a normalized part, stem, or sub-component (`gitea` → the `gitea` folder, `gitea.md`, `kubernetes_gitea`, `gitea-values`).
|
||||||
|
A document is a name hit on ≥1 matched token. Rank hits by (distinct matched token count DESC, then `(source, path)` catalog order) — the old total-matched-length tie-break is RETIRED (record the reason in the docstring: it outranked 5-char product names by 6-char common tokens). Cap at `NAME_HIT_LIMIT` (10, unchanged). The representative-chunk fetch (`_NAME_HIT_SQL`), the `fts_hit=True`/`cosine=0.0` convention, and the deterministic re-ordering are unchanged; each returned `RetrievedChunk` is constructed with `name_hit=True`.
|
||||||
|
3. `app/rag/retriever.py` — `RetrievedChunk` dataclass: new field `name_hit: bool = False` (after `is_summary`; the default keeps every existing construction valid — `fuse()`'s `replace()` copies it automatically). Docstring: True for the name-hit representative row (phase 119) — the selection tier's bonus input (task 02).
|
||||||
|
4. `app/rag/retriever.py` — module docstring: the lexical-list paragraph now describes the two-class component rule (the 2026-09-05 incident fix, extended 2026-09-16 for product names without digits; the title-exclusion precision guard and its measured failure case: `deploy/Deployments/reeseapps/README.md` titled "Deployments" must NOT match the common token "deploy").
|
||||||
|
5. `tests/unit/test_retriever.py` — the rule, on synthetic `RetrievedChunk`/document inputs where possible and the real query path where it needs the DB (follow the file's existing fixture pattern):
|
||||||
|
- digitless exact: a token equals a path part (`gitea/` folder), a file stem (`gitea.md`), and a stem sub-component (`kubernetes_gitea`, `gitea-values`, `test-gateway`) — all hit;
|
||||||
|
- digitless precision: a doc under a `Deployments/` folder with title "Deployments" does NOT hit for token `deploy` (the part normalizes to `deployments` ≠ `deploy`, and titles are never matched);
|
||||||
|
- digit-bearing prefix: `qwen38` hits `qwen3.8-27b-epic-vulkan.container` and does NOT hit a doc whose stem merely CONTAINS the token mid-word without a prefix (`xqwen38y…`);
|
||||||
|
- the versioned join still produces `qwen38` from "Qwen 3.8" (existing behavior, now matchable as a prefix token);
|
||||||
|
- short tokens (<4 normalized) never hit;
|
||||||
|
- ranking: two-token matches beat one-token matches; catalog order breaks the remainder (the length tie-break is GONE — pin a case where the old ordering would differ);
|
||||||
|
- the cap (10) and the representative-chunk choice (summary preferred, else chunk 0) are unchanged;
|
||||||
|
- `name_hit=True` on name-hit rows, `False` on ordinary FTS/vector rows (including after `fuse()`).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] "forgejo"-class (digitless) and "qwen38"-class (digit) questions both produce name-hit lexical rows; "deploy" against deployment-titled docs produces none.
|
||||||
|
- [ ] `name_hit` survives `fuse()` and is False by default everywhere else.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# Task 02 — The bounded name-hit bonus (phase-119 D2, LOCKED A3)
|
||||||
|
|
||||||
|
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D2 — name hits LEADING the lexical list is not enough to lift a weak-vector document into the top-5 (measured: the gateway README stayed out); a bounded selection-time bonus (the phase-106 recency-boost pattern) is the surgical lever.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A new setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**) and a selection-time re-rank: in `select_suggested`, `select_related`, and `weak_hit_titles` only, a document's effective selection score = best fused chunk score + bonus if any of its chunks carries `name_hit=True`. Chunk scores, `fuse()`, `retrieve()`, the A8 gate, and `query_log.top_score` are untouched; `0` reproduces the pre-phase ordering byte-identically; a negative value fails startup loudly.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/config.py` — new `name_hit_bonus: float = 0.005` (env `BOR_NAME_HIT_BONUS`), docstring citing the phase-119 selection re-rank + the phase-106 recency-boost precedent (additive, bounded, single apply site, `0` = off). Validator: `v < 0` → `ValueError` naming the field (the house fail-loud pattern for negative kill switches, e.g. the `agent_max_rounds` precedent); `0` is legal (the kill switch). `.env.example`: an entry with a comment, next to `BOR_RECENCY_BOOST`.
|
||||||
|
2. `app/rag/retriever.py` — one private helper the three selection walks share, e.g. `_selection_order(chunks, bonus) -> list[tuple[Document, float, float, int]]` returning `(document, effective_score, best_cosine, first_seen_index)` for each distinct document, ordered by `(-effective, -best_cosine, document.path, first_seen_index)`:
|
||||||
|
- best fused chunk score and best cosine per document = the EXISTING walk semantics (first-seen-chunk rank order is what `select_documents_tiered`/`select_suggested` already use — keep the semantics, not the loop);
|
||||||
|
- `effective = best + (bonus if any chunk of the doc has name_hit else 0)`;
|
||||||
|
- `bonus=0.0` (or no name-hit chunks present) → the order is IDENTICAL to the pre-phase walk (pin it, below).
|
||||||
|
`select_suggested`, `select_related`, `weak_hit_titles` gain a `bonus: float | None = None` parameter (default `get_settings().name_hit_bonus`, the existing settings-read pattern of the `n` parameter) and consume the helper. The `select_documents`/`select_documents_tiered` dormant helpers are UNCHANGED (they pre-date the bonus and are not the selection path).
|
||||||
|
3. `scripts/eval_retrieval.py` — after the existing per-question top-N document lines, print the seeded suggestion tier: ` suggested: 1. src/path 2. src/path …` from `select_suggested(chunks)` (bonus included, default settings) — the tuning tool must report the tier the prompt actually seeds.
|
||||||
|
4. `tests/unit/test_config.py` — default 0.005, env override, `0` accepted, negative rejected (names the field).
|
||||||
|
5. `tests/unit/test_retriever.py`:
|
||||||
|
- the bonus lifts a name-hit document past a non-name-hit document whose best fused score is higher by less than the bonus (and does NOT lift it when the gap exceeds the bonus);
|
||||||
|
- the bonus is applied ONCE per document regardless of how many of its chunks are name hits;
|
||||||
|
- `bonus=0` (and the no-name-hit case) → ordering byte-identical to the pre-phase walk: build a mixed fused list and assert the helper's document order equals the golden order the old loop produced (pin the golden list in the test);
|
||||||
|
- `select_related` skips excluded ids under the bonus exactly as before; `weak_hit_titles` orders titles by the bonus-adjusted walk;
|
||||||
|
- the A8 inputs are untouched: chunk `cosine`/`fts_hit`/`score` values are unchanged by the bonus (assert on a fused list before/after — the bonus lives in the selection layer only).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `BOR_NAME_HIT_BONUS` is env-tunable (default 0.005), `0` = byte-identical ordering, negative fails startup naming the field.
|
||||||
|
- [ ] `eval_retrieval` prints the `suggested:` tier for each question.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 03 — Suggested-folder context in the grounded prompt (phase-119 D3, LOCKED A4)
|
||||||
|
|
||||||
|
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D3 — the live turn's 3 `ls` drill-downs walked `deploy → reeseapps → gitea` to find files the prompt's suggestions already pointed next to (the canonical README is the parent-folder sibling of suggested `gitea/app/gitea-web.env.j2`); pre-seed the parent-folder contents so the model reads the right file in round 1.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
The HIGH prompt's `<documents>` section carries, after `SUGGEST_INTRO` and before the first `<document>` block, one line per distinct parent folder of the suggested documents — the folder's direct children (subfolders as `name/ (N docs)`, files by relative filename, `ls` entry order), capped 8 entries + ` +N more`, at most 5 lines. `<document>` block markup AND body stay byte-identical; the LOW prompt stays byte-identical; empty `folder_lines` ⇒ the phase-118 HIGH prompt byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/agent.py` — new pure helper `suggested_folder_lines(db: Session, suggested: Sequence[Document], max_lines: int = 5, max_entries: int = 8) -> list[str]` next to the `ls` helpers:
|
||||||
|
- for each suggested document in order: parent prefix = the path up to (excluding) the last `/` (`""` = source root);
|
||||||
|
- dedupe by `(source, prefix)` (first suggested doc wins the slot); stop at `max_lines`;
|
||||||
|
- fetch the source's document rows with `_source_document_rows(db, source)` filtered to the prefix (files: `path == prefix + name`; subfolders: the next segment of `path` after the prefix), and group with the EXISTING `group_folder_listing` semantics (subfolder names + recursive doc counts, file names; entry order = the `ls` folder-level rendering order — the line must read the same as the model's own `ls` output);
|
||||||
|
- exclude the suggested document itself from its folder's entries;
|
||||||
|
- render: `<source>/<prefix>/: e1, e2, …` (source root: `<source>/: …`); subfolder entries `name/ (N docs)` (singular `(1 doc)`), file entries by relative filename; at most `max_entries` entries then ` +N more` (N = the remaining count).
|
||||||
|
2. `app/rag/prompts.py` — `build_high_prompt(documents, notes=None, kb_overview=None, folder_lines: Sequence[str] = ())`: when blocks are present, the `<documents>` body is `SUGGEST_INTRO`, then the folder lines (each on its own line, immediately after the intro line), a blank line, then the first `<document>` block. NO new markup/tag — plain lines (the E2E mock keys off the `<documents>` marker and the LAST block's tail; plain lines before the blocks are inert). `folder_lines` empty ⇒ output byte-identical to the phase-118 shape (pin it). The LOW prompt and `build_deflect_prompt` are untouched.
|
||||||
|
3. `app/api/chat.py` — `plan_turn` gains `folder_lines: Sequence[str] = ()` (passed through to `build_high_prompt` in the HIGH branch; the LOW branch ignores it — the deflected prompt stays byte-identical). In the chat endpoint's HIGH branch (where the DB session is in scope, ~the `plan_turn` call site): before `plan_turn`, compute `suggested = select_suggested(chunks, n=settings.suggested_docs)` (the same deterministic walk `plan_turn` performs internally — one extra cheap walk, no shared state) and `folder_lines = suggested_folder_lines(db, suggested)`; pass `folder_lines` into `plan_turn`. The `TurnPlan`, the durable record, and the per-turn log line are unchanged.
|
||||||
|
4. `tests/unit/test_agent.py` (or a new `tests/unit/test_folder_context.py` — follow the suite that already imports the `ls` helpers) — `suggested_folder_lines`:
|
||||||
|
- source-root doc → `<source>/: …` line with the source's top level;
|
||||||
|
- nested doc → the parent folder line; children correct (subfolder counts recursive, file names relative);
|
||||||
|
- the suggested doc is excluded from its own folder's entries;
|
||||||
|
- dedupe: two suggested docs in the same folder ⇒ one line (first doc's position); `max_lines` caps the list;
|
||||||
|
- `max_entries` cap + ` +N more` (exact remainder); the `(1 doc)` singular;
|
||||||
|
- `ls` entry order (pin a mixed folder/file case against the `group_folder_listing` order).
|
||||||
|
5. `tests/unit/test_prompts.py` — the folder-line placement (after `SUGGEST_INTRO`, before the first block, exact line shape); empty `folder_lines` ⇒ byte-identical to the phase-118 prompt (the existing pin keeps passing); the LOW prompt pin is untouched and still passes; the `<document>` block markup/body pins still pass unchanged.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A HIGH prompt for 5 suggested docs spanning 3 folders carries exactly 3 folder lines, correctly shaped; a zero-folder-lines build is byte-identical to phase 118.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v` green (or the equivalent split); full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user