phase: 191_source_name_labels_tags
All gates re-verified against a stable tree. Report: **Verified (all 6 tasks in `complete/`; this was the final pass)** - `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`. - **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness. - **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors. - **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned). - **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code. - **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria. - **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
This commit is contained in:
+43
-4
@@ -43,6 +43,14 @@ tail (the model input stays byte-identical). The longest-prefix cap in
|
||||
one-line tail (~15 chars/source) always fit; a smaller owner-set cap
|
||||
cuts the tail first — accepted, the cap is owner-tunable.
|
||||
|
||||
Source names (phase 191, D4): when a source has a label row with a
|
||||
``display_name``, the INPUT line of its documents leads with that name
|
||||
instead of the derived source name (``build_overview_prompt``'s *names*
|
||||
map, derived from the ONE ``all_labels`` read :func:`regenerate_overview`
|
||||
performs) — the owner's vocabulary reaches the outline. The stored text,
|
||||
the catalogue rows and the deterministic ``Sources: …`` inventory tail
|
||||
keep the DERIVED name (D1 — a label is an alias, never an identity).
|
||||
|
||||
The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
|
||||
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
|
||||
in the system prompt (wired in phase 31, task 05).
|
||||
@@ -50,7 +58,7 @@ in the system prompt (wired in phase 31, task 05).
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from typing import Protocol
|
||||
|
||||
@@ -62,6 +70,7 @@ from app.db import SessionLocal
|
||||
from app.models import Document, KbOverview
|
||||
from app.rag.llm import LLMError
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.rag.source_labels import all_labels
|
||||
|
||||
logger = logging.getLogger("app.rag.overview")
|
||||
|
||||
@@ -146,7 +155,9 @@ def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
|
||||
|
||||
|
||||
def build_overview_prompt(
|
||||
rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
|
||||
rows: Sequence[tuple[str, str, str, str | None]],
|
||||
max_chars: int | None = None,
|
||||
names: Mapping[str, str] | None = None,
|
||||
) -> tuple[str, str]:
|
||||
"""The ``(system, user)`` message pair for one overview call.
|
||||
|
||||
@@ -166,11 +177,25 @@ def build_overview_prompt(
|
||||
cap and the shared ``[…truncated…]`` marker is appended on its own
|
||||
line, so the model never sees more than the cap and the cut is
|
||||
visible (summarizer convention, phase 30).
|
||||
|
||||
*names* (phase 191, D4) maps a DERIVED source name to the owner's
|
||||
display name for it: the input line leads with the display name
|
||||
instead of the raw source name, so the ``lite`` outline is written in
|
||||
the owner's vocabulary ("the git server" rather than
|
||||
``gitea-infra-docs``). The alias contract (D1) is untouched — the
|
||||
``path`` / ``title`` / summary halves of the line, the stored overview
|
||||
row and its ``Sources: …`` inventory tail all keep the derived name;
|
||||
only this model input changes. ``None`` / an empty map / no entry for
|
||||
a source (or an entry with an empty name) → the pre-phase line,
|
||||
byte-identical.
|
||||
"""
|
||||
limit = max_chars if max_chars is not None else get_settings().overview_input_max_chars
|
||||
lines: list[str] = []
|
||||
for source, path, title, summary in rows:
|
||||
line = f"{source} — {path} — {title}"
|
||||
# Phase 191 (D4): the display name REPLACES the raw source name in
|
||||
# the lite model's input only — never in the stored outline.
|
||||
display = names.get(source) if names else None
|
||||
line = f"{display or source} — {path} — {title}"
|
||||
first = _first_summary_line(summary)
|
||||
if first:
|
||||
line += f" — {first}"
|
||||
@@ -220,6 +245,10 @@ async def regenerate_overview(
|
||||
never sees the tail; the zero-doc and ``LLMError`` paths below
|
||||
leave a previous row byte-identical — an existing tail is part of
|
||||
the stored content and is preserved on failure.
|
||||
* Phase 191 (D4): the ONE :func:`app.rag.source_labels.all_labels`
|
||||
read happens AFTER the zero-doc gate (no label query when there is
|
||||
nothing to outline) and feeds only the model input — the stored
|
||||
content and its inventory tail stay derived-name text.
|
||||
* :class:`LLMError` → ``overview: regeneration failed — …`` is
|
||||
logged and ``False`` is returned with the previous row intact —
|
||||
an old outline is better than none.
|
||||
@@ -249,7 +278,17 @@ async def regenerate_overview(
|
||||
)
|
||||
return False
|
||||
|
||||
system, user = build_overview_prompt(rows)
|
||||
# Phase 191 (D4, task 05): ONE all_labels read for the whole run —
|
||||
# the display-name map for the model input is derived from these
|
||||
# views in this loop, never re-queried (the same single-read
|
||||
# contract the chat turn holds, task 04/05). Rows without a
|
||||
# display name are absent from the map, so their lines stay
|
||||
# byte-identical to pre-phase.
|
||||
labels = all_labels(session)
|
||||
name_map = {
|
||||
src: view.display_name for src, view in labels.items() if view.display_name
|
||||
}
|
||||
system, user = build_overview_prompt(rows, names=name_map)
|
||||
try:
|
||||
text = await llm.chat(
|
||||
[{"role": "system", "content": system}, {"role": "user", "content": user}],
|
||||
|
||||
+76
-2
@@ -100,16 +100,32 @@ owner decision). The deterministic lever is the honesty gate itself
|
||||
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
|
||||
``best_cosine >= lexical_support_floor``, so the misleading docs are
|
||||
no longer injected — see :mod:`app.api.chat`).
|
||||
|
||||
Source names, labels and tags (phase 191, D4): a ``<document>`` block can
|
||||
carry the owner's label row for its source as two ADDITIVE attributes —
|
||||
``name="…"`` (the source's ``display_name``) immediately after
|
||||
``source="…"``, then ``label="…"`` (the one-line description) — so the
|
||||
model learns the name→docs mapping ("the git server" is a source with
|
||||
these documents). The attribute order is fixed
|
||||
(``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``),
|
||||
an unset field adds NOTHING, and no *names* map / no entry for the
|
||||
source leaves the block byte-identical to pre-phase (the alias contract,
|
||||
D1: the identity attributes stay the DERIVED name — a display name never
|
||||
replaces ``source`` or ``path``). Only the two new values are XML-escaped
|
||||
(they are free-form owner text); the identity attributes keep their
|
||||
pre-phase unescaped bytes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
from app.models import Document
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.rag.source_labels import SourceLabelView
|
||||
from app.schemas import HistoryTurn
|
||||
|
||||
#: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by
|
||||
@@ -769,6 +785,47 @@ def _extract_context_line(image_context: str) -> str | None:
|
||||
)
|
||||
|
||||
|
||||
def _attr(value: str) -> str:
|
||||
"""XML-escape one attribute value (phase 191, D4) — ``&``/``<``/``>``
|
||||
and the double quote (``"``), the delimiter the block uses.
|
||||
|
||||
Applied ONLY to the two additive label attributes (``name`` /
|
||||
``label``): they hold free-form OWNER text where a ``"`` is realistic
|
||||
and must not break the attribute. The pre-existing identity
|
||||
attributes (``source`` / ``path`` / ``title`` / ``date``) deliberately
|
||||
keep their unescaped pre-phase bytes — escaping them here would move
|
||||
every label-less block, which the phase's byte-identity criterion
|
||||
forbids.
|
||||
"""
|
||||
return escape(value, {'"': """})
|
||||
|
||||
|
||||
def _source_label_attrs(
|
||||
doc: Document, names: Mapping[str, SourceLabelView] | None
|
||||
) -> str:
|
||||
"""The additive `` name="…"`` / `` label="…"`` fragment of one
|
||||
``<document>`` block (phase 191, D4) — the empty string unless the
|
||||
source has a label row with that field set.
|
||||
|
||||
``None`` / an empty map / no entry for ``doc.source`` / a row with
|
||||
neither field set all render ``""`` — the block is then byte-identical
|
||||
to pre-phase, and a label-only row adds exactly the one ``label``
|
||||
attribute (an unset field adds NOTHING, it is never an empty
|
||||
attribute).
|
||||
"""
|
||||
if not names:
|
||||
return ""
|
||||
view = names.get(doc.source)
|
||||
if view is None:
|
||||
return ""
|
||||
attrs = ""
|
||||
if view.display_name:
|
||||
attrs += f' name="{_attr(view.display_name)}"'
|
||||
if view.label:
|
||||
attrs += f' label="{_attr(view.label)}"'
|
||||
return attrs
|
||||
|
||||
|
||||
def _image_context_line(doc: Document) -> str | None:
|
||||
"""The phase-125 context line for one suggested document (task 05,
|
||||
LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
|
||||
@@ -832,6 +889,7 @@ def build_high_prompt(
|
||||
folder_lines: Sequence[str] = (),
|
||||
show_image: bool = False,
|
||||
show_text: bool = False,
|
||||
names: Mapping[str, SourceLabelView] | None = None,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + the
|
||||
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
|
||||
@@ -917,6 +975,21 @@ def build_high_prompt(
|
||||
(:func:`_document_body`; the NULL/whitespace-summary preview
|
||||
fallback is LOCKED A5; an enriched image doc's summary leads with
|
||||
the phase-125 one-line context line, LOCKED A4).
|
||||
|
||||
*names* (phase 191, D4): the source-name → :class:`SourceLabelView`
|
||||
map of the owner's label rows (the SAME single ``all_labels`` read
|
||||
that feeds the task-04 selection boost — one query per turn). When a
|
||||
doc's source has a row, the block head gains `` name="{display_name}"``
|
||||
immediately after ``source="…"`` and `` label="{label}"`` after that
|
||||
(the fixed order ``source`` → ``name?`` → ``label?`` → ``path`` →
|
||||
``title`` → ``date``); the two values are XML-escaped
|
||||
(:func:`_attr`), the identity attributes are not. An unset field adds
|
||||
NOTHING, so a label-only row gains exactly the one attribute, and
|
||||
``None`` / ``{}`` / no entry for the source keeps the block
|
||||
byte-identical to pre-phase (the default, and the kill pin in
|
||||
:mod:`tests.unit.test_prompts`). This is surfacing only — the alias
|
||||
contract (D1) keeps ``source`` / ``path`` as the document identity
|
||||
the tools, citations and locators use.
|
||||
"""
|
||||
# Phase 106 (D5): every document the model sees carries its
|
||||
# creation date — the block's ``date`` attribute (the row's
|
||||
@@ -926,7 +999,8 @@ def build_high_prompt(
|
||||
# stays out of the prompt (the capped ``read`` tool is the only
|
||||
# full-text path).
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
|
||||
f'<document source="{doc.source}"{_source_label_attrs(doc, names)}'
|
||||
f' path="{doc.path}" title="{doc.title}" '
|
||||
f'date="{doc.created_at:%Y-%m-%d}">\n'
|
||||
f"{_document_body(doc)}\n"
|
||||
"</document>"
|
||||
|
||||
+75
-17
@@ -291,6 +291,16 @@ class RetrievedChunk:
|
||||
:func:`fuse`'s ``replace()`` copies it (the double-hit merge ORs it
|
||||
in — a vector row that is also the name hit's representative chunk
|
||||
keeps the flag).
|
||||
* ``source_name_hit`` — True for a chunk whose parent document's
|
||||
SOURCE the question named (phase 191, D2/D3): the flag is set at
|
||||
the retrieval boundary in ``app/api/chat.py`` from the turn's ONE
|
||||
``all_labels`` read (the pure D2 rule per view — the same read that
|
||||
feeds the ``name``/``label`` prompt attributes, task 05) — the
|
||||
selection tier's SECOND bonus input (``source_name_hit_bonus``, a
|
||||
separate knob). Default
|
||||
``False`` keeps every ordinary construction valid; :func:`fuse`'s
|
||||
``replace()`` copies it (the double-hit merge ORs it in, mirroring
|
||||
``name_hit``).
|
||||
"""
|
||||
|
||||
chunk_id: uuid.UUID
|
||||
@@ -302,6 +312,7 @@ class RetrievedChunk:
|
||||
fts_hit: bool = False
|
||||
is_summary: bool = False
|
||||
name_hit: bool = False
|
||||
source_name_hit: bool = False
|
||||
|
||||
|
||||
def lexical_tsquery(question: str) -> str | None:
|
||||
@@ -339,9 +350,11 @@ def fuse(
|
||||
Lexical-only hits (no vector rank) enter with ``cosine=0.0`` and
|
||||
``fts_hit=True``; vector chunks matched by the lexical list get
|
||||
``fts_hit=True`` in place (the input objects are mutated — callers
|
||||
should not reuse them afterwards), and ``name_hit=True`` is ORed in
|
||||
should not reuse them afterwards), ``name_hit=True`` is ORed in
|
||||
(a vector row that is also the name hit's representative chunk is a
|
||||
name-hit row — the phase-119 selection tier must see it).
|
||||
name-hit row — the phase-119 selection tier must see it), and
|
||||
``source_name_hit`` is ORed in the SAME way (phase 191, D3 — a
|
||||
vector row merged with a flagged source-name row keeps the flag).
|
||||
"""
|
||||
if k <= 0:
|
||||
raise ValueError("rrf k must be > 0")
|
||||
@@ -355,7 +368,10 @@ def fuse(
|
||||
if rc.chunk_id in by_id:
|
||||
existing = by_id[rc.chunk_id]
|
||||
by_id[rc.chunk_id] = replace(
|
||||
existing, fts_hit=True, name_hit=existing.name_hit or rc.name_hit
|
||||
existing,
|
||||
fts_hit=True,
|
||||
name_hit=existing.name_hit or rc.name_hit,
|
||||
source_name_hit=existing.source_name_hit or rc.source_name_hit,
|
||||
)
|
||||
fused[rc.chunk_id] += term
|
||||
else:
|
||||
@@ -754,17 +770,23 @@ def retrieve(
|
||||
def _selection_order(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
bonus: float,
|
||||
source_bonus: float = 0.0,
|
||||
) -> list[tuple[Document, float, float, int]]:
|
||||
"""The shared selection walk of the phase-119 name-hit bonus (D2,
|
||||
LOCKED A3) — one walk for ``select_suggested``, ``select_related``,
|
||||
and ``weak_hit_titles``.
|
||||
and ``weak_hit_titles``, carrying the phase-191 source-name bonus
|
||||
(D3) as its SECOND, independent knob.
|
||||
|
||||
Returns ``(document, effective_score, best_cosine, first_seen_index)``
|
||||
for each distinct document, where *effective_score* is the
|
||||
document's best fused chunk score plus *bonus* when ANY of its
|
||||
chunks carries ``name_hit`` (the D1 path match — the bonus is per
|
||||
DOCUMENT, applied ONCE no matter how many of the document's chunks
|
||||
are name hits).
|
||||
are name hits), plus *source_bonus* when ANY of its chunks carries
|
||||
``source_name_hit`` (the phase-191 D2 source-name match — the SAME
|
||||
per-DOCUMENT, apply-ONCE rule; at most TWO bonuses per document
|
||||
on the fused scale, bounded: name_hit_bonus + source_name_hit_bonus
|
||||
= 0.010 at the shipped defaults, never more).
|
||||
|
||||
The walk keeps the EXISTING selection semantics (not just the loop):
|
||||
the same stable score-descending order as
|
||||
@@ -773,16 +795,18 @@ def _selection_order(
|
||||
best cosine tracked across ALL of its chunks (the tiered walk's
|
||||
tracking).
|
||||
|
||||
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present:
|
||||
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present,
|
||||
AND *source_bonus* ``== 0`` or no source-name-hit chunk present:
|
||||
the document order is IDENTICAL to the pre-phase walk — no re-sort
|
||||
happens at all (byte-identical, LOCKED A3);
|
||||
happens at all (byte-identical, LOCKED A3, mirrored for the second
|
||||
bonus by LOCKED A3 of phase 191);
|
||||
* otherwise the documents are ordered by
|
||||
``(−effective_score, −best_cosine, document.path,
|
||||
first_seen_index)`` — the bounded re-rank: a name-hit document
|
||||
gets a head start on the fused scale, and an effective-score tie
|
||||
resolves by cosine, then path, then the pre-bonus rank.
|
||||
|
||||
The bonus lives in the SELECTION layer only: the chunk objects are
|
||||
The bonuses live in the SELECTION layer only: the chunk objects are
|
||||
never modified — their ``score``/``cosine``/``fts_hit`` (the A8
|
||||
gate's inputs) and ``query_log.top_score`` are untouched.
|
||||
"""
|
||||
@@ -790,6 +814,7 @@ def _selection_order(
|
||||
best_score: dict[uuid.UUID, float] = {}
|
||||
best_cosine: dict[uuid.UUID, float] = {}
|
||||
any_name_hit: dict[uuid.UUID, bool] = {}
|
||||
any_source_hit: dict[uuid.UUID, bool] = {}
|
||||
first_seen: dict[uuid.UUID, int] = {}
|
||||
for idx, rc in enumerate(sorted(chunks, key=lambda c: c.score, reverse=True)):
|
||||
doc = rc.document
|
||||
@@ -798,17 +823,24 @@ def _selection_order(
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
if rc.name_hit:
|
||||
any_name_hit[doc.id] = True
|
||||
if rc.source_name_hit:
|
||||
any_source_hit[doc.id] = True
|
||||
else:
|
||||
order.append(doc)
|
||||
best_score[doc.id] = rc.score
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
any_name_hit[doc.id] = rc.name_hit
|
||||
any_source_hit[doc.id] = rc.source_name_hit
|
||||
first_seen[doc.id] = idx
|
||||
effective = {
|
||||
doc.id: best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0)
|
||||
doc.id: best_score[doc.id]
|
||||
+ (bonus if any_name_hit[doc.id] else 0.0)
|
||||
+ (source_bonus if any_source_hit[doc.id] else 0.0)
|
||||
for doc in order
|
||||
}
|
||||
if bonus > 0.0 and any(any_name_hit[doc.id] for doc in order):
|
||||
if (bonus > 0.0 and any(any_name_hit[doc.id] for doc in order)) or (
|
||||
source_bonus > 0.0 and any(any_source_hit[doc.id] for doc in order)
|
||||
):
|
||||
order.sort(
|
||||
key=lambda doc: (
|
||||
-effective[doc.id],
|
||||
@@ -826,17 +858,26 @@ def _selection_order(
|
||||
def weak_hit_titles(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[str]:
|
||||
"""Distinct parent-document titles of *chunks*, best SELECTION score
|
||||
first (the phase-119 name-hit bonus applied — see
|
||||
:func:`_selection_order`, D2, LOCKED A3).
|
||||
first (the phase-119 name-hit bonus and the phase-191 source-name
|
||||
bonus applied — see :func:`_selection_order`, D2/D3, LOCKED A3).
|
||||
|
||||
Deflection mode (PLAN §6, A8) is built from these *titles only* — the
|
||||
LOW prompt and the "Maybe try" chips never see document content.
|
||||
|
||||
*source_bonus* defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting
|
||||
(0.005); ``0`` reproduces the pre-phase-191 walk byte-identically.
|
||||
"""
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
return [doc.title for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)]
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
return [
|
||||
doc.title
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)
|
||||
]
|
||||
|
||||
|
||||
def select_documents_tiered(
|
||||
@@ -953,6 +994,7 @@ def select_suggested(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
n: int | None = None,
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[Document]:
|
||||
"""Top-N distinct parent documents in SELECTION rank order — the
|
||||
phase-118 "start here" suggestion tier (LOCKED A3, re-revised by
|
||||
@@ -975,7 +1017,10 @@ def select_suggested(
|
||||
|
||||
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
|
||||
owner-tunable starting point); ``0`` reproduces the pre-phase walk
|
||||
byte-identically (the kill switch, LOCKED A3).
|
||||
byte-identically (the kill switch, LOCKED A3). *source_bonus*
|
||||
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
|
||||
D3 — the same bounded shape, the same kill switch, an independent
|
||||
knob).
|
||||
|
||||
The returned rows carry the full document content, byte-identical —
|
||||
the content is what the agent's ``read`` tool serves later (never
|
||||
@@ -985,7 +1030,14 @@ def select_suggested(
|
||||
top_n = n if n is not None else get_settings().suggested_docs
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
return [doc for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)[:top_n]]
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
return [
|
||||
doc
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)[
|
||||
:top_n
|
||||
]
|
||||
]
|
||||
|
||||
|
||||
def select_related(
|
||||
@@ -993,6 +1045,7 @@ def select_related(
|
||||
excluded_ids: set[uuid.UUID],
|
||||
cap: int,
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[Document]:
|
||||
"""The documents ranked AFTER *excluded_ids* — the phase-118 related
|
||||
tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap*
|
||||
@@ -1012,7 +1065,10 @@ def select_related(
|
||||
|
||||
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
|
||||
owner-tunable starting point); ``0`` reproduces the pre-phase walk
|
||||
byte-identically (the kill switch, LOCKED A3).
|
||||
byte-identically (the kill switch, LOCKED A3). *source_bonus*
|
||||
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
|
||||
D3 — the same bounded shape, the same kill switch, an independent
|
||||
knob).
|
||||
|
||||
The returned rows carry the full document content, byte-identical
|
||||
(the tier is metadata for the ``done`` frame and the durable
|
||||
@@ -1020,8 +1076,10 @@ def select_related(
|
||||
"""
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
out: list[Document] = []
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus):
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus):
|
||||
if len(out) >= cap:
|
||||
break
|
||||
if doc.id in excluded_ids:
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
"""Source name/label/tags storage + the D2 source-name matching rule (phase 191, task 01).
|
||||
|
||||
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
|
||||
the model when prompting if you have a project name or a standard name
|
||||
you reference things by." This module is the whole non-HTTP surface of
|
||||
that feature: the upsert/get/delete helpers over the ``source_labels``
|
||||
table (migration 0030) and the pure matching rule the retrieval boost
|
||||
(task 04) consumes.
|
||||
|
||||
**The alias contract (D1, owner-confirmed):** a label row is an ALIAS
|
||||
beside the derived source name — ``Document.source``, the ``source/path``
|
||||
document identity, the read/grep/``show_text`` locators, the citations
|
||||
and the KB tree are ALL untouched by it. Renaming a source never
|
||||
invalidates a saved chat, a shared page or a verbatim locator. Every
|
||||
function here keys on the DERIVED name (``Document.source``); the
|
||||
display name is data, never identity.
|
||||
|
||||
**The matching rule (D2):** a source is a SOURCE-NAME HIT when at least
|
||||
one :func:`app.rag.retriever.name_hit_tokens` candidate (the phase-119
|
||||
class-agnostic token list, reused UNCHANGED) equals — in the
|
||||
:func:`app.rag.retriever._normalize_name` form, so case-insensitive and
|
||||
accent-folded (phase 162) — a token of the source's ``display_name``,
|
||||
``label`` or any ``tags`` entry. The phase-119 digit-class rule does
|
||||
NOT apply to user-given names (the owner's names are not paths —
|
||||
"gitea" must hit even though it is digitless; the digit rule exists to
|
||||
keep prose tokens from matching PATHS). The one filter is
|
||||
:data:`app.rag.retriever.NAME_TOKEN_MIN_LEN` (4) applied to the SOURCE
|
||||
side too — a 2-char tag never matches.
|
||||
|
||||
Session convention (the phase-53 ``sources_meta`` style): the write
|
||||
helpers only *flush* — the **caller** commits.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import SourceLabel
|
||||
from app.rag.retriever import NAME_TOKEN_MIN_LEN, _normalize_name, name_hit_tokens
|
||||
|
||||
#: Boundary limits (phase 191 D1/D5) — the API (task 02) turns the
|
||||
#: ``normalize_tags`` raises into the framework's standard 422; the
|
||||
#: display-name / label caps are enforced there against these constants.
|
||||
DISPLAY_NAME_MAX_CHARS: Final = 120
|
||||
LABEL_MAX_CHARS: Final = 200
|
||||
TAG_MAX_CHARS: Final = 32
|
||||
TAGS_MAX_COUNT: Final = 10
|
||||
|
||||
|
||||
class _Unset:
|
||||
"""Marker type for "field omitted" in :func:`set_label`.
|
||||
|
||||
The partial-update shape (phase-168 D5): an OMITTED field leaves the
|
||||
stored value untouched, an explicit ``None`` CLEARS it — the two are
|
||||
indistinguishable with plain ``None`` defaults, so the sentinel owns
|
||||
the difference (the API passes only the fields the request carried).
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
|
||||
_UNSET: Final = _Unset()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceLabelView:
|
||||
"""Immutable read-side snapshot of one label row (D1).
|
||||
|
||||
``tags`` is a ``tuple`` so the view stays hashable/mutable-proof
|
||||
after the ORM row leaves the session; ``display_name`` / ``label``
|
||||
are ``None`` when unset (never the empty string — the API trims
|
||||
empty submissions to NULL).
|
||||
"""
|
||||
|
||||
source: str
|
||||
display_name: str | None
|
||||
label: str | None
|
||||
tags: tuple[str, ...]
|
||||
|
||||
|
||||
def _view(row: SourceLabel) -> SourceLabelView:
|
||||
return SourceLabelView(
|
||||
source=row.source,
|
||||
display_name=row.display_name,
|
||||
label=row.label,
|
||||
tags=tuple(row.tags or ()),
|
||||
)
|
||||
|
||||
|
||||
def get_label(db: Session, source: str) -> SourceLabel | None:
|
||||
"""The ORM row for *source* (one PK read); ``None`` when absent."""
|
||||
return db.get(SourceLabel, source)
|
||||
|
||||
|
||||
def label_for(db: Session, source: str) -> SourceLabelView | None:
|
||||
"""The frozen :class:`SourceLabelView` for *source* (one PK read);
|
||||
``None`` when absent — the "no label row = pre-phase" contract."""
|
||||
row = db.get(SourceLabel, source)
|
||||
return None if row is None else _view(row)
|
||||
|
||||
|
||||
def all_labels(db: Session) -> dict[str, SourceLabelView]:
|
||||
"""Every label row as a frozen view, keyed by ``source``.
|
||||
|
||||
ONE query (the phase-119 one-query convention): the table carries a
|
||||
handful of rows, so callers match in Python against it instead of
|
||||
issuing a query per source. Ordered by ``source`` for determinism.
|
||||
"""
|
||||
rows = db.scalars(select(SourceLabel).order_by(SourceLabel.source)).all()
|
||||
return {row.source: _view(row) for row in rows}
|
||||
|
||||
|
||||
def set_label(
|
||||
db: Session,
|
||||
source: str,
|
||||
*,
|
||||
display_name: str | None | _Unset = _UNSET,
|
||||
label: str | None | _Unset = _UNSET,
|
||||
tags: Sequence[str] | None | _Unset = _UNSET,
|
||||
) -> SourceLabel:
|
||||
"""Upsert the label row for *source*; return the ORM row.
|
||||
|
||||
The partial-update shape (phase 191 D5, the phase-168 precedent):
|
||||
a field left at the ``_UNSET`` sentinel keeps its stored value, an
|
||||
explicitly passed value OVERWRITES it, an explicit ``None`` CLEARS
|
||||
(``tags`` clears to ``[]`` — the column is NOT NULL). A missing row
|
||||
is created with the passed values (omitted fields = their defaults:
|
||||
``NULL`` / ``[]``). Values arrive ALREADY VALIDATED (the API layer
|
||||
trims/enforces the caps via :func:`normalize_tags`); this helper
|
||||
stores what it is given.
|
||||
|
||||
Flush-only (the phase-53 session convention): the **caller** commits.
|
||||
"""
|
||||
row = db.get(SourceLabel, source)
|
||||
if row is None:
|
||||
row = SourceLabel(
|
||||
source=source,
|
||||
display_name=None if isinstance(display_name, _Unset) else display_name,
|
||||
label=None if isinstance(label, _Unset) else label,
|
||||
tags=[] if isinstance(tags, _Unset) or tags is None else list(tags),
|
||||
)
|
||||
db.add(row)
|
||||
else:
|
||||
if not isinstance(display_name, _Unset):
|
||||
row.display_name = display_name
|
||||
if not isinstance(label, _Unset):
|
||||
row.label = label
|
||||
if not isinstance(tags, _Unset):
|
||||
row.tags = [] if tags is None else list(tags)
|
||||
db.flush()
|
||||
return row
|
||||
|
||||
|
||||
def delete_label(db: Session, source: str) -> bool:
|
||||
"""Remove the label row for *source*; ``False`` when absent (the
|
||||
API's 404 shape), ``True`` when a row was deleted. Flush-only."""
|
||||
row = db.get(SourceLabel, source)
|
||||
if row is None:
|
||||
return False
|
||||
db.delete(row)
|
||||
db.flush()
|
||||
return True
|
||||
|
||||
|
||||
def normalize_tags(raw: Sequence[str]) -> list[str]:
|
||||
"""Normalize one tag submission (pure, the API's 422 source).
|
||||
|
||||
* trim every entry, drop the ones that are empty afterwards
|
||||
(``[]`` is legal — it clears the tags);
|
||||
* raise :class:`ValueError` above :data:`TAGS_MAX_COUNT` entries
|
||||
(counted on the trimmed non-empty list, BEFORE the case-insensitive
|
||||
de-dupe: 11 typed tags are 11 rejected tags);
|
||||
* raise :class:`ValueError` when any entry exceeds
|
||||
:data:`TAG_MAX_CHARS` afterwards;
|
||||
* de-duplicate case-insensitively, preserving the FIRST case seen
|
||||
(``Gitea`` then ``gitea`` keeps ``Gitea``), order otherwise kept.
|
||||
"""
|
||||
cleaned = [tag.strip() for tag in raw]
|
||||
cleaned = [tag for tag in cleaned if tag]
|
||||
if len(cleaned) > TAGS_MAX_COUNT:
|
||||
raise ValueError(f"at most {TAGS_MAX_COUNT} tags are allowed")
|
||||
for tag in cleaned:
|
||||
if len(tag) > TAG_MAX_CHARS:
|
||||
raise ValueError(f"each tag must be at most {TAG_MAX_CHARS} characters")
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for tag in cleaned:
|
||||
key = tag.casefold()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append(tag)
|
||||
return out
|
||||
|
||||
|
||||
def _source_side_tokens(view: SourceLabelView) -> set[str]:
|
||||
"""The normalized token set of one label row (the D2 source side).
|
||||
|
||||
Every whitespace token of ``display_name`` / ``label`` / every
|
||||
``tags`` entry, folded through :func:`_normalize_name` (the single
|
||||
normalization BOTH sides of name matching pass through, phase 162)
|
||||
and kept at :data:`NAME_TOKEN_MIN_LEN` chars minimum — the 4-char
|
||||
floor applies to the SOURCE side too (a 2-char tag never matches).
|
||||
"""
|
||||
tokens: set[str] = set()
|
||||
fields: list[str] = [view.display_name or "", view.label or "", *view.tags]
|
||||
for field in fields:
|
||||
for raw in field.split():
|
||||
norm = _normalize_name(raw)
|
||||
if len(norm) >= NAME_TOKEN_MIN_LEN:
|
||||
tokens.add(norm)
|
||||
return tokens
|
||||
|
||||
|
||||
def question_name_hits(view: SourceLabelView | None, question: str) -> bool:
|
||||
"""The D2 rule (pure): does *question* name the source behind *view*?
|
||||
|
||||
``None`` (no label row) → ``False`` — the pre-phase contract. Any
|
||||
:func:`name_hit_tokens` candidate EQUAL (in the normalized form) to
|
||||
a source-side token is a hit: no digit-class rule, no prefix match
|
||||
(user-given names have no prose-precision problem — the phase-119
|
||||
digit rule guards PATHS, not names).
|
||||
"""
|
||||
if view is None:
|
||||
return False
|
||||
candidates = name_hit_tokens(question)
|
||||
if not candidates:
|
||||
return False
|
||||
source_tokens = _source_side_tokens(view)
|
||||
return any(candidate in source_tokens for candidate in candidates)
|
||||
|
||||
|
||||
def question_name_hit_sources(db: Session, question: str) -> set[str]:
|
||||
"""The derived source names the question names (D2) — the standalone
|
||||
one-query entry point (no other per-turn work needed).
|
||||
|
||||
ONE :func:`all_labels` read + the pure rule per row (a handful of
|
||||
rows, matched in Python — the phase-119 one-query convention). The
|
||||
empty set is the common case: no label rows, or no name in the
|
||||
question.
|
||||
|
||||
The chat turn does NOT call this: since task 05 it inlines the same
|
||||
pair against its own single ``all_labels`` read — the read that also
|
||||
feeds the prompt surfaces (D4) — so a turn issues ONE label query.
|
||||
"""
|
||||
return {
|
||||
view.source
|
||||
for view in all_labels(db).values()
|
||||
if question_name_hits(view, question)
|
||||
}
|
||||
Reference in New Issue
Block a user