phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# Phase 119 — Name-signal retrieval + suggested-folder context; chips cite read docs only
|
||||
|
||||
**Source:** owner live report, 2026-09-16 — brain.reeseapps.com, turn "How do I deploy gitea?". Three findings, all validated live and reproduced against the 1,000-document live replica (restored 2026-09-16 from `data/bor_live_kb_replica.dump.sql`):
|
||||
|
||||
1. **Irrelevant documents in the chip cards** — the turn's 7 chips were 5 suggested docs + 2 agent-read docs (phase-118 locked A4); 2 of the 5 suggestions were off-topic (`homepage/src/index.html`, `deploy/Deployments/reeseapps/gateway/README.md`) and 3 were never read. The agent's one *failed* read (`docker-compose.yaml` — verified absent from the live KB, teaching refusal) correctly earned no chip, so the chip surface itself had no construction bug — it faithfully showed a noisy suggestion tier.
|
||||
2. **The agent lists instead of reading the proposed files** — the 5 summaries seeded into the prompt did not include the canonical `deploy/Deployments/reeseapps/gitea/README.md`, so the model rationally explored: 3 `ls` drill-downs (`deploy` → `reeseapps` → `gitea`), one wrong-guess `read` (refused), one more `ls`, one `read` — 7 tool rounds. It never read a single suggested file, and its answer's "Docs used:" line cited `gitea-web.env.j2`, a file it never read (confabulation from the suggestion blocks being in context).
|
||||
3. **Suggested documents off** — reproduced on the replica (which lacks the phase-118 markdown summary chunks; live was *worse*: `index.html` ranked #1 there):
|
||||
|
||||
| Question (replica, pre-phase) | Suggested top-5 | Canonical doc |
|
||||
|---|---|---|
|
||||
| "How do I deploy gitea?" | gitea playbook, **gateway README**, k8s gitea-values ×2, container_gitea.md | ❌ rank 11 |
|
||||
| "How do I deploy forgejo?" | foobar.md, AGENTS.md, reeseapps README, inventory.yaml, bifrost.md | ❌ **zero forgejo docs** |
|
||||
| "How do I set up the gateway?" | shelly.md, forgejo.network, framework_16.md, firewall playbook, home_assistant.md | ❌ not in top-5 |
|
||||
|
||||
**Root causes.** (a) The lexical side never sees document *names*: `chunks.tsv` is `to_tsvector('english', content)` — body only (migration 0002) — and the name-hit mechanism (the 2026-09-05 versioned-name incident fix, `app/rag/retriever.py::_name_hit_chunks`) fires **only for digit-bearing** question tokens, so `gitea`/`forgejo`/`gateway` (no digits) get no name signal; "forgejo" appears in only 26 chunk bodies vs 136 for "gitea", so an OR-tsquery dominated by the common token "deploy" buries the name. (b) Phase 118 widened the blast radius: suggestions = top-5 **no floor** (118-A3) and chips = suggested + read (118-A4), so every noisy suggestion became a visible junk chip; the new per-document summary chunks also created hub attractors (site-wide summaries vector-match almost anything). (c) The prompt teaches "start here **if one seems right**" + "answer as soon as you have what you need" — with a suggestion set missing the canonical doc, exploration (and its latency) is the rational model behavior.
|
||||
|
||||
**Story:** n/a (owner-directed RAG behavior change; extends the phase-118 summary seeding, the 2026-09-05 name-hit mechanism, and the phase 94/113 tier assets).
|
||||
|
||||
## Objective
|
||||
Product-name questions surface the product's own documents in the suggestion tier (component name-hits + a bounded, kill-switchable name-hit bonus); the HIGH prompt pre-seeds each suggested document's parent-folder contents (collapsing the `ls` drill-downs the agent currently pays); the citation chips carry **only documents the agent actually read** (owner decision 2026-09-16 — phase-118 A4 retired); and the model is taught to cite only what it read or used.
|
||||
|
||||
## Dependencies
|
||||
- `118_summary_seed_context` (complete) — the summary-seed prompt, `select_suggested`/`select_related`, the `read`-extends-context contract, and the done-frame tiering this phase re-locks.
|
||||
- `117_tool_call_compact` (complete) — the current `read`/`ls`/`grep` copy this phase leaves byte-identical.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
**D1 — name-hit rule (task 01, LOCKED A2).** `name_hit_tokens` returns **all** normalized whitespace tokens of length ≥ 4 (dotted tokens kept whole, as today) *plus* the versioned-name join candidates — the digit filter moves from the token list to the match side. `_name_hit_chunks` matches a token against the document's **path** (never the title — titles are prose: `deploy/Deployments/reeseapps/README.md` is titled "Deployments", and the common token "deploy" must not name-match it or the 9 other deployment-titled docs; the owner-verified failure mode of the naive relaxation):
|
||||
|
||||
- **digit-bearing token** → PREFIX of a normalized path part or file stem (preserves the 2026-09-05 incident fix: `qwen38` → `qwen3.8-27b-juggernaut-vulkan.container`);
|
||||
- **digitless token** → EQUALS a normalized path part, a file stem, or a stem sub-component (the stem split on non-alphanumeric runs: `kubernetes_gitea` → `kubernetes`/`gitea`, `gitea-values` → `gitea`/`values`, `test-gateway` → `test`/`gateway`).
|
||||
|
||||
Measured effect (replica, 2026-09-16, replacing the digit-only rule with this one): "forgejo" → all 5 suggested slots are forgejo-tree docs (README #1); "gitea" → 4 of 5 slots gitea-tree incl. the canonical README; "gateway" → 4 of 5 slots gateway-tree. Ranking of name hits: (distinct matched token count desc, then catalog order `(source, path)`) — the total-matched-length tie-break is RETIRED (it systematically outranked 5-char product names by 6-char common tokens). `NAME_HIT_LIMIT` stays 10. Name-hit rows keep the existing `fts_hit=True`, `cosine=0.0` convention (A8 already treats them as a lexical signal needing vector corroboration). `RetrievedChunk` gains `name_hit: bool = False` (dataclass field, default keeps every existing construction valid) so the selection tier can tell them apart.
|
||||
|
||||
**D2 — name-hit bonus (task 02, LOCKED A3).** A bounded, kill-switchable **selection-time** re-rank — the phase-106 recency-boost pattern (additive, single apply site, `0` = off → byte-identical ordering, negative fails startup loudly). New setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**). In the document-selection walks only (`select_suggested`, `select_related`, `weak_hit_titles`), a document's effective selection score = its best fused chunk score **+ bonus if any of its chunks is a name hit**. Chunk scores, `fuse()`, `retrieve()`, the A8 honesty gate (reads chunk cosine/fts), and `query_log.top_score` are untouched. 0.005 ≈ a 2–4 rank head start on the RRF scale (rank 1 vs 5 in one list ≈ 0.0010); measured, it is the difference that puts the canonical doc inside the top-5 on all three failing questions (post-phase suggested top-5: gitea → playbook, gitea-postgres.env.j2, gateway README, **gitea README #4**, gitea-web.env.j2; forgejo → 5/5 forgejo; gateway → shelly.md, istio test-gateway.yaml, **gateway README #3**, gateway certbot/ddns playbooks). The 0.005 default is an owner-tunable starting point, not a calibrated constant — the battery (task 06) records the realized margins. 118-A3 is **re-revised** by this lock: the suggestion set is still top-5 distinct docs with NO cosine floor, now walked with the name-hit bonus. `scripts/eval_retrieval.py` gains a `suggested:` line (the `select_suggested` walk over the fused list, bonus included) so the tuning tool reports the tier the prompt actually seeds.
|
||||
|
||||
**D3 — suggested-folder context (task 03, LOCKED A4).** The HIGH prompt's `<documents>` section carries, **after `SUGGEST_INTRO` and before the first `<document>` block**, one line per DISTINCT parent folder of the suggested documents (in suggested-doc order, deduped by `(source, parent prefix)`, at most 5 lines): `<source>/<parent>/: <entry, …>` — subfolders as `name/ (N docs)`, files by relative filename, entry order = the existing `ls` folder-level rendering order (`group_folder_listing` in `app/rag/agent.py` — the line must read the same as the model's own `ls` output), capped at 8 entries per line + ` +N more`; the source root renders as `<source>/: …`. The suggested document itself is excluded from its folder's entries (its identity is already in the block). Source: a new pure helper `suggested_folder_lines(db, suggested, …)` next to the `ls` helpers in `app/rag/agent.py`, reusing `_source_document_rows` + `group_folder_listing`; computed in the chat endpoint (which holds the session) and passed through `plan_turn` (new parameter, default empty) → `build_high_prompt` (new parameter, default empty). The `<document>` block markup AND body stay **byte-identical** (the E2E mock's tail echo keys off the LAST block's tail — the folder lines must never land after a summary); the LOW prompt stays byte-identical; with an empty `folder_lines` the HIGH prompt is byte-identical to the phase-118 output. The live-turn payoff: the README sits in the parent folder of suggested `gitea/app/gitea-web.env.j2`, so its name appears in the prompt — the model reads it in round 1 instead of walking three `ls` levels.
|
||||
|
||||
**D4 — cite discipline (task 04, LOCKED A5).** `SUGGEST_INTRO`'s final sentence ("Cite the document(s) you used, by path.") is REPLACED with: `Cite only the document(s) you read — or, if you answered from a suggested summary without reading it, cite that suggested document — never a document you neither read nor used.` `PERSONA`, the LOW/`DEFLECT_MODE` body, and `TOOLS_SECTION` stay byte-identical (the phase-117 copy is untouched — no tool-copy gate re-run, D6); the byte-pinned anchors re-cut in the same task.
|
||||
|
||||
**D5 — chips cite read docs only (task 05, LOCKED A1 — owner decision 2026-09-16, phase-118 A4 retired).** `done.sources` = `holder.read_docs` only, deduped by `(source, path)`, read order preserved. A grounded turn on which the agent reads nothing (the phase-118 summary-only fast path) shows **no chips** — an accepted, owner-directed consequence (the answer prose still names the doc it used, per D4). The DURABLE record is untouched (118-A3 stands: `query_log.sources` + the per-turn log line = suggested + related + read); the `related` tier (the de-emphasized "nearby docs" row) is untouched except its existing dedupe against the cited set now keys on read docs only (a read related doc is a chip, never a nearby doc — unchanged intent). No frontend change: `appendSources`/`addSources` already render zero entries as no DOM.
|
||||
|
||||
**D6 — verification (task 06).** New dedicated Playwright suite `tests/e2e/test_name_signal_read_chips.py` (run in isolation; the `test_summary_seed_context.py` local-directory-source fixture pattern), a mock-LLM trigger echo making the folder lines observable, the retired-A4 pins re-targeted, and the empirical battery: the 1,000-doc live replica (the `restore-test-db` skill; the dump is a local gitignored artifact — if missing, STOP and ask the owner, never fabricate numbers) + `scripts/eval_retrieval --from-file` over a tracked question battery, + the real-model fixture run recorded as TELEMETRY in `TOOL_CALLING_TESTING.md` (the gate is NOT re-triggered — no `AGENT_TOOLS`/`TOOLS_SECTION` copy change; conditions, were they read, follow the phase-118 A7 semantics).
|
||||
|
||||
**Invariants (all tasks).** The A8 honesty gate, every SSE frame shape, the LOW prompt, `PERSONA`, `TOOLS_SECTION`, the `<document>` block markup/body, the `Document {source}/{path}:` read-result header, and the per-turn log line format (fields unchanged; `sources=`/`suggested=` values move with the data) are byte-identical. `PLAN.md` untouched (Protocol B — the owner folds the 118-A4 retirement + 118-A3 re-revision in at the next redo; both are owner-confirmed below).
|
||||
|
||||
## Tasks
|
||||
1. `01_name_hit_component_match.md` — retriever: the component name-hit rule (D1) + the `name_hit` flag.
|
||||
2. `02_name_hit_bonus.md` — the bounded selection-time bonus (D2) + `eval_retrieval` suggested line.
|
||||
3. `03_suggested_folder_context.md` — the prompt's suggested-folder lines (D3).
|
||||
4. `04_cite_discipline_copy.md` — `SUGGEST_INTRO` cite discipline + the re-cut pins (D4).
|
||||
5. `05_chips_read_only.md` — `done.sources` = read docs only (D5).
|
||||
6. `06_e2e_retarget_and_battery.md` — the dedicated E2E, the retired pins, the mock echo, the replica battery + real-model telemetry (D6).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_retriever.py` (D1 rule incl. the "deploy"-title precision case and the versioned prefix; D2 bonus lift/kill-switch/validator), `tests/unit/test_config.py` (the new setting), `tests/unit/test_agent.py` or a new `tests/unit/test_folder_context.py` (D3 helper: root/nested/exclusion/dedupe/caps), `tests/unit/test_prompts.py` + `tests/unit/test_prompt_lock.py` (D3 line placement + the D4 re-cut; LOW + PERSONA byte-identical), `tests/unit/test_chat_gate.py` (D5 done-frame + durable record).
|
||||
- E2E: `tests/e2e/test_name_signal_read_chips.py`, **in isolation** (`uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov`); the regression suites green (the re-targeted phase-113/118 pins included).
|
||||
- Battery (task 06): the three name questions put their canonical doc in the printed suggested top-5; the out-of-KB question's gate verdict is unchanged; full output recorded in `06_…md`; real-model fixture telemetry recorded in `TOOL_CALLING_TESTING.md`.
|
||||
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] "How do I deploy gitea/forgejo?" and "How do I set up the gateway?" (1,000-doc replica) put the canonical doc in the suggested top-5 (eval battery, recorded); "Qwen 3.8 27b" incident question and the out-of-KB deflection sanity are unchanged.
|
||||
- [ ] A grounded prompt's `<documents>` section carries the suggested-folder lines (block markup/body and the LOW prompt byte-identical).
|
||||
- [ ] `done.sources` = read docs only (a zero-read grounded turn chips nothing; the related row and the durable record are untouched); the unit + E2E pins agree.
|
||||
- [ ] The new E2E green in isolation; the re-targeted old pins green; `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — chips (done.sources) = agent-read documents only** (deduped by `(source, path)`, read order); phase-118 A4 (suggested + read) is RETIRED; a zero-read grounded turn shows no chips (owner decision, 2026-09-16, stated consequence).
|
||||
- **A2 — the name-hit rule**: length-≥4 normalized question token; digit-bearing → prefix of a normalized path part/stem; digitless → exact equality with a path part, stem, or stem sub-component (split on non-alnum); **titles are never name-matched**; ranking (matched-token count, catalog order) — the length tie-break retired; `NAME_HIT_LIMIT` stays 10 (owner-confirmed 2026-09-16, from the measured D1/D2 evidence above).
|
||||
- **A3 — the name-hit bonus**: `BOR_NAME_HIT_BONUS` default 0.005, additive at selection time only, `0` = byte-identical kill switch, negative fails startup loudly; 118-A3 re-revised (top-5, NO floor, bonus walk); the A8 gate / chunk scores / `query_log.top_score` untouched (owner-confirmed 2026-09-16).
|
||||
- **A4 — the suggested-folder context**: lines after `SUGGEST_INTRO`, before the first block; ≤5 lines, ≤8 entries + ` +N more` per line; `<document>` block markup/body and the LOW prompt byte-identical (owner-confirmed 2026-09-16).
|
||||
- **A5 — the cite-discipline sentence** replaces `SUGGEST_INTRO`'s final sentence verbatim as written in D4; `PERSONA` + LOW body + `TOOLS_SECTION` byte-identical (owner-confirmed 2026-09-16).
|
||||
- **A6 — PLAN.md untouched (Protocol B)**: the 118-A4 retirement and 118-A3 re-revision are recorded here (owner-confirmed) and folded into the anchor table at the next redo (the phase-118 A6 precedent).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ tests/ scripts/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): component name-hits + bounded name-hit bonus for product-name questions, suggested-folder context in the grounded prompt, and chips that cite read documents only"
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 01 — The component name-hit rule (phase-119 D1, LOCKED A2)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D1 — the 2026-09-16 live finding: product-name questions ("How do I deploy forgejo?") get no lexical name signal because the name-hit mechanism fires only for digit-bearing tokens and `chunks.tsv` indexes body text only.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`_name_hit_chunks` in `app/rag/retriever.py` matches question tokens against document **paths** under the new two-class rule (digit-bearing → prefix; digitless → exact component/stem/sub-component), titles never, ranked by (matched-token count, catalog order); `RetrievedChunk` gains a `name_hit` flag.
|
||||
|
||||
## Work
|
||||
1. `app/rag/retriever.py` — `name_hit_tokens(question)`: return ALL normalized whitespace tokens of length ≥ 4 (the existing `_TOKEN_RE` dotted-token handling, `_normalize_name`) **plus** the existing versioned-name join candidates (a purely-numeric second token), order of first appearance, de-duplicated — the digit filter is REMOVED from this function (the digit distinction moves to the match side). Update its docstring: the token list is class-agnostic; matching applies the A2 rule.
|
||||
2. `app/rag/retriever.py` — `_name_hit_chunks`: for each document (catalog order, the existing single projection query over `(id, source, path, title)` — **title is selected but never matched**), compute the name parts: for every path part — the normalized part, the normalized file stem, and the stem's sub-components (the stem lowercased, split on `[^a-z0-9]+`, each piece normalized, empties dropped). A token matches when:
|
||||
- it contains a digit → it is a PREFIX of a normalized part or stem (`qwen38` → `qwen3.8-27b-juggernaut-vulkan`);
|
||||
- it has no digit → it EQUALS a normalized part, stem, or sub-component (`gitea` → the `gitea` folder, `gitea.md`, `kubernetes_gitea`, `gitea-values`).
|
||||
A document is a name hit on ≥1 matched token. Rank hits by (distinct matched token count DESC, then `(source, path)` catalog order) — the old total-matched-length tie-break is RETIRED (record the reason in the docstring: it outranked 5-char product names by 6-char common tokens). Cap at `NAME_HIT_LIMIT` (10, unchanged). The representative-chunk fetch (`_NAME_HIT_SQL`), the `fts_hit=True`/`cosine=0.0` convention, and the deterministic re-ordering are unchanged; each returned `RetrievedChunk` is constructed with `name_hit=True`.
|
||||
3. `app/rag/retriever.py` — `RetrievedChunk` dataclass: new field `name_hit: bool = False` (after `is_summary`; the default keeps every existing construction valid — `fuse()`'s `replace()` copies it automatically). Docstring: True for the name-hit representative row (phase 119) — the selection tier's bonus input (task 02).
|
||||
4. `app/rag/retriever.py` — module docstring: the lexical-list paragraph now describes the two-class component rule (the 2026-09-05 incident fix, extended 2026-09-16 for product names without digits; the title-exclusion precision guard and its measured failure case: `deploy/Deployments/reeseapps/README.md` titled "Deployments" must NOT match the common token "deploy").
|
||||
5. `tests/unit/test_retriever.py` — the rule, on synthetic `RetrievedChunk`/document inputs where possible and the real query path where it needs the DB (follow the file's existing fixture pattern):
|
||||
- digitless exact: a token equals a path part (`gitea/` folder), a file stem (`gitea.md`), and a stem sub-component (`kubernetes_gitea`, `gitea-values`, `test-gateway`) — all hit;
|
||||
- digitless precision: a doc under a `Deployments/` folder with title "Deployments" does NOT hit for token `deploy` (the part normalizes to `deployments` ≠ `deploy`, and titles are never matched);
|
||||
- digit-bearing prefix: `qwen38` hits `qwen3.8-27b-epic-vulkan.container` and does NOT hit a doc whose stem merely CONTAINS the token mid-word without a prefix (`xqwen38y…`);
|
||||
- the versioned join still produces `qwen38` from "Qwen 3.8" (existing behavior, now matchable as a prefix token);
|
||||
- short tokens (<4 normalized) never hit;
|
||||
- ranking: two-token matches beat one-token matches; catalog order breaks the remainder (the length tie-break is GONE — pin a case where the old ordering would differ);
|
||||
- the cap (10) and the representative-chunk choice (summary preferred, else chunk 0) are unchanged;
|
||||
- `name_hit=True` on name-hit rows, `False` on ordinary FTS/vector rows (including after `fuse()`).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] "forgejo"-class (digitless) and "qwen38"-class (digit) questions both produce name-hit lexical rows; "deploy" against deployment-titled docs produces none.
|
||||
- [ ] `name_hit` survives `fuse()` and is False by default everywhere else.
|
||||
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 02 — The bounded name-hit bonus (phase-119 D2, LOCKED A3)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D2 — name hits LEADING the lexical list is not enough to lift a weak-vector document into the top-5 (measured: the gateway README stayed out); a bounded selection-time bonus (the phase-106 recency-boost pattern) is the surgical lever.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
A new setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**) and a selection-time re-rank: in `select_suggested`, `select_related`, and `weak_hit_titles` only, a document's effective selection score = best fused chunk score + bonus if any of its chunks carries `name_hit=True`. Chunk scores, `fuse()`, `retrieve()`, the A8 gate, and `query_log.top_score` are untouched; `0` reproduces the pre-phase ordering byte-identically; a negative value fails startup loudly.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — new `name_hit_bonus: float = 0.005` (env `BOR_NAME_HIT_BONUS`), docstring citing the phase-119 selection re-rank + the phase-106 recency-boost precedent (additive, bounded, single apply site, `0` = off). Validator: `v < 0` → `ValueError` naming the field (the house fail-loud pattern for negative kill switches, e.g. the `agent_max_rounds` precedent); `0` is legal (the kill switch). `.env.example`: an entry with a comment, next to `BOR_RECENCY_BOOST`.
|
||||
2. `app/rag/retriever.py` — one private helper the three selection walks share, e.g. `_selection_order(chunks, bonus) -> list[tuple[Document, float, float, int]]` returning `(document, effective_score, best_cosine, first_seen_index)` for each distinct document, ordered by `(-effective, -best_cosine, document.path, first_seen_index)`:
|
||||
- best fused chunk score and best cosine per document = the EXISTING walk semantics (first-seen-chunk rank order is what `select_documents_tiered`/`select_suggested` already use — keep the semantics, not the loop);
|
||||
- `effective = best + (bonus if any chunk of the doc has name_hit else 0)`;
|
||||
- `bonus=0.0` (or no name-hit chunks present) → the order is IDENTICAL to the pre-phase walk (pin it, below).
|
||||
`select_suggested`, `select_related`, `weak_hit_titles` gain a `bonus: float | None = None` parameter (default `get_settings().name_hit_bonus`, the existing settings-read pattern of the `n` parameter) and consume the helper. The `select_documents`/`select_documents_tiered` dormant helpers are UNCHANGED (they pre-date the bonus and are not the selection path).
|
||||
3. `scripts/eval_retrieval.py` — after the existing per-question top-N document lines, print the seeded suggestion tier: ` suggested: 1. src/path 2. src/path …` from `select_suggested(chunks)` (bonus included, default settings) — the tuning tool must report the tier the prompt actually seeds.
|
||||
4. `tests/unit/test_config.py` — default 0.005, env override, `0` accepted, negative rejected (names the field).
|
||||
5. `tests/unit/test_retriever.py`:
|
||||
- the bonus lifts a name-hit document past a non-name-hit document whose best fused score is higher by less than the bonus (and does NOT lift it when the gap exceeds the bonus);
|
||||
- the bonus is applied ONCE per document regardless of how many of its chunks are name hits;
|
||||
- `bonus=0` (and the no-name-hit case) → ordering byte-identical to the pre-phase walk: build a mixed fused list and assert the helper's document order equals the golden order the old loop produced (pin the golden list in the test);
|
||||
- `select_related` skips excluded ids under the bonus exactly as before; `weak_hit_titles` orders titles by the bonus-adjusted walk;
|
||||
- the A8 inputs are untouched: chunk `cosine`/`fts_hit`/`score` values are unchanged by the bonus (assert on a fused list before/after — the bonus lives in the selection layer only).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `BOR_NAME_HIT_BONUS` is env-tunable (default 0.005), `0` = byte-identical ordering, negative fails startup naming the field.
|
||||
- [ ] `eval_retrieval` prints the `suggested:` tier for each question.
|
||||
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — Suggested-folder context in the grounded prompt (phase-119 D3, LOCKED A4)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D3 — the live turn's 3 `ls` drill-downs walked `deploy → reeseapps → gitea` to find files the prompt's suggestions already pointed next to (the canonical README is the parent-folder sibling of suggested `gitea/app/gitea-web.env.j2`); pre-seed the parent-folder contents so the model reads the right file in round 1.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The HIGH prompt's `<documents>` section carries, after `SUGGEST_INTRO` and before the first `<document>` block, one line per distinct parent folder of the suggested documents — the folder's direct children (subfolders as `name/ (N docs)`, files by relative filename, `ls` entry order), capped 8 entries + ` +N more`, at most 5 lines. `<document>` block markup AND body stay byte-identical; the LOW prompt stays byte-identical; empty `folder_lines` ⇒ the phase-118 HIGH prompt byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — new pure helper `suggested_folder_lines(db: Session, suggested: Sequence[Document], max_lines: int = 5, max_entries: int = 8) -> list[str]` next to the `ls` helpers:
|
||||
- for each suggested document in order: parent prefix = the path up to (excluding) the last `/` (`""` = source root);
|
||||
- dedupe by `(source, prefix)` (first suggested doc wins the slot); stop at `max_lines`;
|
||||
- fetch the source's document rows with `_source_document_rows(db, source)` filtered to the prefix (files: `path == prefix + name`; subfolders: the next segment of `path` after the prefix), and group with the EXISTING `group_folder_listing` semantics (subfolder names + recursive doc counts, file names; entry order = the `ls` folder-level rendering order — the line must read the same as the model's own `ls` output);
|
||||
- exclude the suggested document itself from its folder's entries;
|
||||
- render: `<source>/<prefix>/: e1, e2, …` (source root: `<source>/: …`); subfolder entries `name/ (N docs)` (singular `(1 doc)`), file entries by relative filename; at most `max_entries` entries then ` +N more` (N = the remaining count).
|
||||
2. `app/rag/prompts.py` — `build_high_prompt(documents, notes=None, kb_overview=None, folder_lines: Sequence[str] = ())`: when blocks are present, the `<documents>` body is `SUGGEST_INTRO`, then the folder lines (each on its own line, immediately after the intro line), a blank line, then the first `<document>` block. NO new markup/tag — plain lines (the E2E mock keys off the `<documents>` marker and the LAST block's tail; plain lines before the blocks are inert). `folder_lines` empty ⇒ output byte-identical to the phase-118 shape (pin it). The LOW prompt and `build_deflect_prompt` are untouched.
|
||||
3. `app/api/chat.py` — `plan_turn` gains `folder_lines: Sequence[str] = ()` (passed through to `build_high_prompt` in the HIGH branch; the LOW branch ignores it — the deflected prompt stays byte-identical). In the chat endpoint's HIGH branch (where the DB session is in scope, ~the `plan_turn` call site): before `plan_turn`, compute `suggested = select_suggested(chunks, n=settings.suggested_docs)` (the same deterministic walk `plan_turn` performs internally — one extra cheap walk, no shared state) and `folder_lines = suggested_folder_lines(db, suggested)`; pass `folder_lines` into `plan_turn`. The `TurnPlan`, the durable record, and the per-turn log line are unchanged.
|
||||
4. `tests/unit/test_agent.py` (or a new `tests/unit/test_folder_context.py` — follow the suite that already imports the `ls` helpers) — `suggested_folder_lines`:
|
||||
- source-root doc → `<source>/: …` line with the source's top level;
|
||||
- nested doc → the parent folder line; children correct (subfolder counts recursive, file names relative);
|
||||
- the suggested doc is excluded from its own folder's entries;
|
||||
- dedupe: two suggested docs in the same folder ⇒ one line (first doc's position); `max_lines` caps the list;
|
||||
- `max_entries` cap + ` +N more` (exact remainder); the `(1 doc)` singular;
|
||||
- `ls` entry order (pin a mixed folder/file case against the `group_folder_listing` order).
|
||||
5. `tests/unit/test_prompts.py` — the folder-line placement (after `SUGGEST_INTRO`, before the first block, exact line shape); empty `folder_lines` ⇒ byte-identical to the phase-118 prompt (the existing pin keeps passing); the LOW prompt pin is untouched and still passes; the `<document>` block markup/body pins still pass unchanged.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A HIGH prompt for 5 suggested docs spanning 3 folders carries exactly 3 folder lines, correctly shaped; a zero-folder-lines build is byte-identical to phase 118.
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v` green (or the equivalent split); full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 04 — Cite-discipline copy in the suggestion intro (phase-119 D4, LOCKED A5)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D4 — the live turn's answer cited `gitea-web.env.j2` in its "Docs used:" line although the agent never read it: the model confabulates usage from the suggestion blocks that sit in context. The intro's loose "cite what you used" is replaced with a discipline sentence.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`SUGGEST_INTRO`'s final sentence is replaced verbatim with the cite-discipline sentence (LOCKED A5); every byte pin of the old sentence re-cuts in the same change; `PERSONA`, the LOW/`DEFLECT_MODE` body, and `TOOLS_SECTION` stay byte-identical.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — `SUGGEST_INTRO` constant: replace the final sentence `Cite the document(s) you used, by path.` with:
|
||||
`Cite only the document(s) you read — or, if you answered from a suggested summary without reading it, cite that suggested document — never a document you neither read nor used.`
|
||||
The rest of the constant (the start-here framing, the `read` pointer, the "its full text is not in the prompt until you read it" clause) is untouched. Update the constant's docstring: the phase-119 cite discipline (the live confabulation it closes; the phase-118 sentence retired).
|
||||
2. Re-cut the pins (audit with `grep -rn "Cite the document(s) you used" tests/`):
|
||||
- `tests/unit/test_prompts.py` — any assertion that pins the old final sentence (exact-string or sha/length) re-points at the new sentence; the intro's PREFIX (start-here framing) pin, if separate, is untouched;
|
||||
- `tests/unit/test_prompt_lock.py` — if `SUGGEST_INTRO` (or a prompt built from it) is byte-pinned (sha256/length/prefix/suffix anchors), re-cut those anchors in this same change (the phase-118 TOOLS_SECTION re-cut precedent); `TOOLS_SECTION`, `PERSONA`, and the LOW body anchors MUST stay byte-identical — assert they are untouched;
|
||||
- any E2E that greps the old sentence in the rendered stream or mock output (audit `tests/e2e/`) — re-target or confirm none exists (the mock keys off markers, not this sentence — expected: none).
|
||||
3. `tests/unit/test_prompts.py` — new pin: the HIGH prompt carries the discipline sentence exactly once, inside `<documents>` after the intro; the sentence is absent from the LOW prompt (byte-identical pin passes).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The HIGH prompt carries the LOCKED A5 sentence verbatim; `PERSONA`, LOW body, and `TOOLS_SECTION` byte-pins pass unchanged.
|
||||
- [ ] No test in `tests/` references the retired sentence.
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 05 — Chips cite read documents only (phase-119 D5, LOCKED A1)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** owner decision 2026-09-16 ("I'd like the chips to be read/used docs only") — phase-118 A4 (`done.sources` = suggested + read) is retired; the live turn's 7 chips (5 noisy suggestions + 2 reads) are the symptom.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`done.sources` on grounded turns = `holder.read_docs` only (deduped by `(source, path)`, read order preserved); a zero-read grounded turn chips nothing. The durable record (`query_log.sources` + the per-turn log line = suggested + related + read), the `related` tier, and the deflected turn's `[]` are untouched. No frontend change.
|
||||
|
||||
## Work
|
||||
1. `app/api/chat.py` — the done-frame section (the "4. Durable record" block):
|
||||
- `cited_docs` = `holder.read_docs` deduped by `(source, path)`, order preserved — `plan.suggested_docs` is REMOVED from the citation walk (the `cited_seen` set keeps its role: it now collects read docs only);
|
||||
- `record_docs` (the durable record) and `source_paths` are UNCHANGED (suggested + related + read, deduped — 118-A3 stands);
|
||||
- `related_refs` dedupe stays `if (d.source, d.path) not in cited_seen` — a read related doc is a chip, never a nearby doc (unchanged intent, now keyed on read docs);
|
||||
- `ChatDoneEvent` construction and every SSE frame shape are unchanged; the deflected branch still yields `sources=[]`.
|
||||
2. `app/api/chat.py` — module docstring + the done-section comment: the citation surface is now READ DOCS ONLY (phase 119, LOCKED A1, owner decision 2026-09-16; phase-118 A4 retired; the zero-read grounded turn chips nothing — an accepted consequence, the answer prose names the doc per the phase-119 cite discipline; the durable record still records retrieval, not citations).
|
||||
3. `tests/unit/test_chat_gate.py` — re-target the done-frame pins (audit the file's `done`/`sources` assertions):
|
||||
- grounded turn, agent read two docs ⇒ `done.sources` == exactly those two, read order, deduped;
|
||||
- grounded turn, agent read NOTHING (a summary-only answer) ⇒ `done.sources == []` (the new A1 pin — replaces the phase-118 "always chips" pin if present);
|
||||
- suggested docs that were never read appear NOWHERE in `done.sources` (explicit contrast pin against the retired A4 union);
|
||||
- a related doc that was read ⇒ in `done.sources`, NOT in `done.related`;
|
||||
- the durable record pins (suggested + related + read, deduped) and the `suggested=N` log-line pin are untouched and still pass;
|
||||
- the deflected turn's `sources == []` pin is untouched.
|
||||
4. Frontend: NO change (audit only) — confirm `appendSources` (`frontend/assets/app.js`) and `addSources` (`frontend/assets/shared.js`) both no-op on an empty `sources` array (they do: `if (!sources || !sources.length) return;`) and that no view asserts ≥1 chip on a grounded turn (e.g. no `aria`/label assumption that breaks on an absent meta row — the related row and action buttons occupy `.msg-meta` independently, per the phase-113 layout pins).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the pins above; coverage **>90%** on `app/`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A grounded done frame carries exactly the agent-read docs (a zero-read turn carries none); the durable record and the related tier behave as pinned.
|
||||
- [ ] `uv run pytest tests/unit/test_chat_gate.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Task 06 — Dedicated E2E, retired pins, mock echo, and the verification battery (phase-119 D6)
|
||||
|
||||
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D6 — the phase is complete only when the new behavior is pinned E2E (in isolation), the retired-A4 pins are re-targeted, the 1,000-doc replica battery passes, and the real-model telemetry is recorded.
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
`tests/e2e/test_name_signal_read_chips.py` (green in isolation) pins the four behaviors — name-signal suggestions, chips = read docs only, the zero-read no-chip consequence, and the suggested-folder lines (via a new mock trigger echo); every E2E pin of the retired phase-118 A4 chip contract is re-targeted; the replica battery + real-model run are recorded.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — new trigger (follow the existing trigger conventions exactly: lowercase phrase, checked where its specificity warrants, no existing E2E question or fixture file contains it — verify with a repo-wide grep before choosing):
|
||||
- user message containing **`repeat your folder map`** AND system prompt containing `<documents>` → the composed answer ends with ` (folders: <line 1>; <line 2>; …)` echoing the suggested-folder lines VERBATIM (the plain lines between `SUGGEST_INTRO` and the first `<document ` in the `<documents>` section; joined with `; `). If no folder lines are present, the echo is omitted (the answer is otherwise unchanged). This is the same prompt-injection-echo convention as the `<tuning>` / `<knowledge_base>` triggers.
|
||||
- Header docstring: a section for the new trigger (what it keys on, what it echoes, why — phase 119 D3 observability). No existing trigger or marker changes; the tail-echo (`show the end of your notes`), the tool flows, and all other behaviors stay byte-identical.
|
||||
2. Fixture design (in the new suite, following `test_summary_seed_context.py`'s local-directory-source pattern — seed dirs + `/api/git-sources` registration + sync; per-module truncate/re-import so the suite runs in ISOLATION):
|
||||
- ~6 markdown files across 2–3 folders of one local source; one file under a DISTINCTIVE digitless path component (e.g. `forgejo/forgejo-install.md`) whose BODY has minimal overlap with the battery question words (its name is its main signal); the other suggested-tier files' bodies overlap the question so they out-rank it on vector/FTS.
|
||||
- Invariant to engineer (assert, don't assume): for the question `<distinctive-token> … show the end of your notes`, the name-hit document is the **LAST** of the top-5 suggested (its `Source: <source>/<path>` tail line is the tail echo). Tune file contents until the tail-echo assertion passes; verify determinism by running the suite twice.
|
||||
3. `tests/e2e/test_name_signal_read_chips.py` — the four assertion groups (admin session, SSE `done` frame via the existing chat helpers, chip DOM `.source-chip`):
|
||||
- **(a) name signal → suggestion:** the distinctive-token question + `show the end of your notes` ⇒ the rendered answer contains `Source: <source>/<name-hit path>` (the name-hit document was the last suggested — the digitless component rule + bonus put it in the tier);
|
||||
- **(b) chips = read docs only:** the `use your tools` flow (the mock's single-read: ls → drill → read first file line ⇒ answer `Read <sp>. …`) ⇒ the bubble carries exactly ONE `.source-chip`, `<source>/<read path>`; none of the five suggested docs chips; the de-emphasized related row (`NEARBY DOCS, IN CASE:`) still renders when the fixture's rank-6+ tier is non-empty (phase-113 behavior untouched);
|
||||
- **(c) zero-read grounded turn chips nothing:** a plain question (no tool trigger) ⇒ `done` frame `sources == []` and ZERO `.source-chip` elements in the bubble (LOCKED A1's visible consequence); the answer still renders;
|
||||
- **(d) suggested-folder lines:** the distinctive-token question + `repeat your folder map` ⇒ the answer contains the exact folder line for the name-hit document's folder (`<source>/<folder>/: …` with its real entries — assert the line prefix and one known sibling entry).
|
||||
4. Retarget the retired-A4 pins (audit rule for every hit of `grep -rn "done\[.sources.\]\|source-chip" tests/e2e/`: asserted set == the docs the flow actually READ ⇒ keep; asserted set contains a suggested/cited-but-unread doc ⇒ re-target to the read set with a phase-119 A1 comment; related-row assertions ⇒ keep):
|
||||
- `tests/e2e/test_summary_seed_context.py` — the `done["sources"] == [(SOURCE, p) for p in SUGGESTED_TAIL]` pin (the no-read flow) ⇒ `[]`; the `… == [(SOURCE, p) for p in SUGGESTED_READ]` pin (the read flow) ⇒ the read set only; the durable-record pins (`row.sources == … SUGGESTED_TAIL + RELATED_PATHS`) STAY (118-A3 unchanged);
|
||||
- `tests/e2e/test_source_chip_quality.py` (phase 113) — the cited-tier chip pins re-target per the audit rule (the related-row pins stay);
|
||||
- `tests/e2e/test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_read_truncation_cap.py`, `test_search_tool.py`, `test_retrieval_quality.py`, `test_share_chat.py` — audit per the rule (read-flow chip assertions should mostly pass unchanged; the share page re-renders `done` sources, so its pins follow the same rule).
|
||||
- `tests/unit/test_chat_gate.py` was already re-targeted in task 05 — confirm no E2E-side twin remains.
|
||||
5. Battery (record in THIS file — the phase-118 gate-record precedent):
|
||||
- Precondition: the 1,000-doc live replica in the app DB (the `restore-test-db` skill; verify the fingerprint `docs=1000, chunks=8866`; the dump `data/bor_live_kb_replica.dump.sql` is a local gitignored artifact — if missing, STOP and ask the owner; NEVER fabricate or reuse stale numbers). Note: the replica predates the phase-118 markdown-summary backfill (markdown `summary` NULL, no markdown `is_summary` chunks) — the overview's measured numbers are the same-state baseline; report that fact with the numbers.
|
||||
- `tests/fixtures/retrieval_battery.txt` (new, tracked): one question per line, `#` comment lines allowed (the reader skips them — NO inline comments, `scripts/eval_retrieval.py` reads the whole line): the three phase-119 name questions (canonical docs noted in comment lines), the 2026-09-05 incident regression (`Do you have notes on the qwen 3.8 27b model?` — a `qwen3.8-27b` quadlet stays in the suggested top-5), and the deflection sanity (`What is the capital of Mongolia?` — zero name hits in the KB; the gate verdict and fts count must match the pre-phase replica run recorded in the phase overview: `HIGH`, fts=5 — the name-hit change must not move it).
|
||||
- Run `uv run python -m scripts/eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`; **gate:** each of the three name questions' `suggested:` top-5 contains its canonical doc (the overview's post-phase numbers), the incident doc stays top-5, and the Mongolia verdict/fts is unchanged. Paste the full output into this file under "Battery run 2026-09-…" with the verdict line.
|
||||
- Real-model telemetry (NOT a gate — no `AGENT_TOOLS`/`TOOLS_SECTION` copy changed, so the tool-copy gate is not re-triggered; record for the owner): `uv run python -m scripts.agent_realmodel_check --restore --mode fixture`; append a phase-119 TELEMETRY section to `TOOL_CALLING_TESTING.md` (the phase-118 format: the run output, the four conditions read under the phase-118 A7 semantics, per-turn reading — highlight any change in `ls`-drill-down frequency on the read turns, the phase's intended latency effect).
|
||||
6. Housekeeping: `TOOL_CALLING_TESTING.md` section header notes it is telemetry-only; the phase dir is otherwise untouched by this task.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: `uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`); the re-targeted suites green (run each re-targeted file in isolation — the suites truncate/re-import and must not share state); full `uv run pytest` green; coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The new suite green in isolation: (a) the name-hit doc is suggested (tail-echo sentinel), (b) chips == read docs only with the related row intact, (c) a zero-read grounded turn chips nothing, (d) the folder line is echoed verbatim.
|
||||
- [ ] No E2E pin asserts a suggested/cited-but-unread doc as a chip; the durable-record pins are untouched.
|
||||
- [ ] The battery gate passes and the full output + the real-model telemetry are recorded (this file + `TOOL_CALLING_TESTING.md`).
|
||||
- [ ] `uv run pytest` green; coverage >90%; lint + types clean.
|
||||
Reference in New Issue
Block a user