phase: 191_source_name_labels_tags
Build and Push Containers / build-and-push-app (push) Successful in 7m24s
Build and Push Containers / build-and-push-db (push) Successful in 36s

All gates re-verified against a stable tree. Report:

**Verified (all 6 tasks in `complete/`; this was the final pass)**
- `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`.
- **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned).
- **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code.
- **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria.
- **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
This commit is contained in:
2026-10-19 06:13:32 -04:00
parent c6ffa050a6
commit 4f20c256d7
70 changed files with 7378 additions and 50 deletions
+43 -4
View File
@@ -43,6 +43,14 @@ tail (the model input stays byte-identical). The longest-prefix cap in
one-line tail (~15 chars/source) always fit; a smaller owner-set cap
cuts the tail first — accepted, the cap is owner-tunable.
Source names (phase 191, D4): when a source has a label row with a
``display_name``, the INPUT line of its documents leads with that name
instead of the derived source name (``build_overview_prompt``'s *names*
map, derived from the ONE ``all_labels`` read :func:`regenerate_overview`
performs) — the owner's vocabulary reaches the outline. The stored text,
the catalogue rows and the deterministic ``Sources: …`` inventory tail
keep the DERIVED name (D1 — a label is an alias, never an identity).
The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
in the system prompt (wired in phase 31, task 05).
@@ -50,7 +58,7 @@ in the system prompt (wired in phase 31, task 05).
from __future__ import annotations
import logging
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from typing import Protocol
@@ -62,6 +70,7 @@ from app.db import SessionLocal
from app.models import Document, KbOverview
from app.rag.llm import LLMError
from app.rag.retriever import TRUNCATION_MARKER
from app.rag.source_labels import all_labels
logger = logging.getLogger("app.rag.overview")
@@ -146,7 +155,9 @@ def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
def build_overview_prompt(
rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
rows: Sequence[tuple[str, str, str, str | None]],
max_chars: int | None = None,
names: Mapping[str, str] | None = None,
) -> tuple[str, str]:
"""The ``(system, user)`` message pair for one overview call.
@@ -166,11 +177,25 @@ def build_overview_prompt(
cap and the shared ``[…truncated…]`` marker is appended on its own
line, so the model never sees more than the cap and the cut is
visible (summarizer convention, phase 30).
*names* (phase 191, D4) maps a DERIVED source name to the owner's
display name for it: the input line leads with the display name
instead of the raw source name, so the ``lite`` outline is written in
the owner's vocabulary ("the git server" rather than
``gitea-infra-docs``). The alias contract (D1) is untouched — the
``path`` / ``title`` / summary halves of the line, the stored overview
row and its ``Sources: …`` inventory tail all keep the derived name;
only this model input changes. ``None`` / an empty map / no entry for
a source (or an entry with an empty name) → the pre-phase line,
byte-identical.
"""
limit = max_chars if max_chars is not None else get_settings().overview_input_max_chars
lines: list[str] = []
for source, path, title, summary in rows:
line = f"{source} — {path} — {title}"
# Phase 191 (D4): the display name REPLACES the raw source name in
# the lite model's input only — never in the stored outline.
display = names.get(source) if names else None
line = f"{display or source} — {path} — {title}"
first = _first_summary_line(summary)
if first:
line += f" — {first}"
@@ -220,6 +245,10 @@ async def regenerate_overview(
never sees the tail; the zero-doc and ``LLMError`` paths below
leave a previous row byte-identical — an existing tail is part of
the stored content and is preserved on failure.
* Phase 191 (D4): the ONE :func:`app.rag.source_labels.all_labels`
read happens AFTER the zero-doc gate (no label query when there is
nothing to outline) and feeds only the model input — the stored
content and its inventory tail stay derived-name text.
* :class:`LLMError` → ``overview: regeneration failed — …`` is
logged and ``False`` is returned with the previous row intact —
an old outline is better than none.
@@ -249,7 +278,17 @@ async def regenerate_overview(
)
return False
system, user = build_overview_prompt(rows)
# Phase 191 (D4, task 05): ONE all_labels read for the whole run —
# the display-name map for the model input is derived from these
# views in this loop, never re-queried (the same single-read
# contract the chat turn holds, task 04/05). Rows without a
# display name are absent from the map, so their lines stay
# byte-identical to pre-phase.
labels = all_labels(session)
name_map = {
src: view.display_name for src, view in labels.items() if view.display_name
}
system, user = build_overview_prompt(rows, names=name_map)
try:
text = await llm.chat(
[{"role": "system", "content": system}, {"role": "user", "content": user}],
+76 -2
View File
@@ -100,16 +100,32 @@ owner decision). The deterministic lever is the honesty gate itself
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
``best_cosine >= lexical_support_floor``, so the misleading docs are
no longer injected — see :mod:`app.api.chat`).
Source names, labels and tags (phase 191, D4): a ``<document>`` block can
carry the owner's label row for its source as two ADDITIVE attributes —
``name="…"`` (the source's ``display_name``) immediately after
``source="…"``, then ``label="…"`` (the one-line description) — so the
model learns the name→docs mapping ("the git server" is a source with
these documents). The attribute order is fixed
(``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``),
an unset field adds NOTHING, and no *names* map / no entry for the
source leaves the block byte-identical to pre-phase (the alias contract,
D1: the identity attributes stay the DERIVED name — a display name never
replaces ``source`` or ``path``). Only the two new values are XML-escaped
(they are free-form owner text); the identity attributes keep their
pre-phase unescaped bytes.
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from typing import Any
from xml.sax.saxutils import escape
from app.config import Settings, get_settings
from app.models import Document
from app.rag.retriever import TRUNCATION_MARKER
from app.rag.source_labels import SourceLabelView
from app.schemas import HistoryTurn
#: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by
@@ -769,6 +785,47 @@ def _extract_context_line(image_context: str) -> str | None:
)
def _attr(value: str) -> str:
"""XML-escape one attribute value (phase 191, D4) — ``&``/``<``/``>``
and the double quote (``&quot;``), the delimiter the block uses.
Applied ONLY to the two additive label attributes (``name`` /
``label``): they hold free-form OWNER text where a ``"`` is realistic
and must not break the attribute. The pre-existing identity
attributes (``source`` / ``path`` / ``title`` / ``date``) deliberately
keep their unescaped pre-phase bytes — escaping them here would move
every label-less block, which the phase's byte-identity criterion
forbids.
"""
return escape(value, {'"': "&quot;"})
def _source_label_attrs(
doc: Document, names: Mapping[str, SourceLabelView] | None
) -> str:
"""The additive `` name="…"`` / `` label="…"`` fragment of one
``<document>`` block (phase 191, D4) — the empty string unless the
source has a label row with that field set.
``None`` / an empty map / no entry for ``doc.source`` / a row with
neither field set all render ``""`` — the block is then byte-identical
to pre-phase, and a label-only row adds exactly the one ``label``
attribute (an unset field adds NOTHING, it is never an empty
attribute).
"""
if not names:
return ""
view = names.get(doc.source)
if view is None:
return ""
attrs = ""
if view.display_name:
attrs += f' name="{_attr(view.display_name)}"'
if view.label:
attrs += f' label="{_attr(view.label)}"'
return attrs
def _image_context_line(doc: Document) -> str | None:
"""The phase-125 context line for one suggested document (task 05,
LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
@@ -832,6 +889,7 @@ def build_high_prompt(
folder_lines: Sequence[str] = (),
show_image: bool = False,
show_text: bool = False,
names: Mapping[str, SourceLabelView] | None = None,
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + the
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
@@ -917,6 +975,21 @@ def build_high_prompt(
(:func:`_document_body`; the NULL/whitespace-summary preview
fallback is LOCKED A5; an enriched image doc's summary leads with
the phase-125 one-line context line, LOCKED A4).
*names* (phase 191, D4): the source-name → :class:`SourceLabelView`
map of the owner's label rows (the SAME single ``all_labels`` read
that feeds the task-04 selection boost — one query per turn). When a
doc's source has a row, the block head gains `` name="{display_name}"``
immediately after ``source="…"`` and `` label="{label}"`` after that
(the fixed order ``source`` → ``name?`` → ``label?`` → ``path`` →
``title`` → ``date``); the two values are XML-escaped
(:func:`_attr`), the identity attributes are not. An unset field adds
NOTHING, so a label-only row gains exactly the one attribute, and
``None`` / ``{}`` / no entry for the source keeps the block
byte-identical to pre-phase (the default, and the kill pin in
:mod:`tests.unit.test_prompts`). This is surfacing only — the alias
contract (D1) keeps ``source`` / ``path`` as the document identity
the tools, citations and locators use.
"""
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
@@ -926,7 +999,8 @@ def build_high_prompt(
# stays out of the prompt (the capped ``read`` tool is the only
# full-text path).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'<document source="{doc.source}"{_source_label_attrs(doc, names)}'
f' path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{_document_body(doc)}\n"
"</document>"
+75 -17
View File
@@ -291,6 +291,16 @@ class RetrievedChunk:
:func:`fuse`'s ``replace()`` copies it (the double-hit merge ORs it
in — a vector row that is also the name hit's representative chunk
keeps the flag).
* ``source_name_hit`` — True for a chunk whose parent document's
SOURCE the question named (phase 191, D2/D3): the flag is set at
the retrieval boundary in ``app/api/chat.py`` from the turn's ONE
``all_labels`` read (the pure D2 rule per view — the same read that
feeds the ``name``/``label`` prompt attributes, task 05) — the
selection tier's SECOND bonus input (``source_name_hit_bonus``, a
separate knob). Default
``False`` keeps every ordinary construction valid; :func:`fuse`'s
``replace()`` copies it (the double-hit merge ORs it in, mirroring
``name_hit``).
"""
chunk_id: uuid.UUID
@@ -302,6 +312,7 @@ class RetrievedChunk:
fts_hit: bool = False
is_summary: bool = False
name_hit: bool = False
source_name_hit: bool = False
def lexical_tsquery(question: str) -> str | None:
@@ -339,9 +350,11 @@ def fuse(
Lexical-only hits (no vector rank) enter with ``cosine=0.0`` and
``fts_hit=True``; vector chunks matched by the lexical list get
``fts_hit=True`` in place (the input objects are mutated — callers
should not reuse them afterwards), and ``name_hit=True`` is ORed in
should not reuse them afterwards), ``name_hit=True`` is ORed in
(a vector row that is also the name hit's representative chunk is a
name-hit row — the phase-119 selection tier must see it).
name-hit row — the phase-119 selection tier must see it), and
``source_name_hit`` is ORed in the SAME way (phase 191, D3 — a
vector row merged with a flagged source-name row keeps the flag).
"""
if k <= 0:
raise ValueError("rrf k must be > 0")
@@ -355,7 +368,10 @@ def fuse(
if rc.chunk_id in by_id:
existing = by_id[rc.chunk_id]
by_id[rc.chunk_id] = replace(
existing, fts_hit=True, name_hit=existing.name_hit or rc.name_hit
existing,
fts_hit=True,
name_hit=existing.name_hit or rc.name_hit,
source_name_hit=existing.source_name_hit or rc.source_name_hit,
)
fused[rc.chunk_id] += term
else:
@@ -754,17 +770,23 @@ def retrieve(
def _selection_order(
chunks: Sequence[RetrievedChunk],
bonus: float,
source_bonus: float = 0.0,
) -> list[tuple[Document, float, float, int]]:
"""The shared selection walk of the phase-119 name-hit bonus (D2,
LOCKED A3) — one walk for ``select_suggested``, ``select_related``,
and ``weak_hit_titles``.
and ``weak_hit_titles``, carrying the phase-191 source-name bonus
(D3) as its SECOND, independent knob.
Returns ``(document, effective_score, best_cosine, first_seen_index)``
for each distinct document, where *effective_score* is the
document's best fused chunk score plus *bonus* when ANY of its
chunks carries ``name_hit`` (the D1 path match — the bonus is per
DOCUMENT, applied ONCE no matter how many of the document's chunks
are name hits).
are name hits), plus *source_bonus* when ANY of its chunks carries
``source_name_hit`` (the phase-191 D2 source-name match — the SAME
per-DOCUMENT, apply-ONCE rule; at most TWO bonuses per document
on the fused scale, bounded: name_hit_bonus + source_name_hit_bonus
= 0.010 at the shipped defaults, never more).
The walk keeps the EXISTING selection semantics (not just the loop):
the same stable score-descending order as
@@ -773,16 +795,18 @@ def _selection_order(
best cosine tracked across ALL of its chunks (the tiered walk's
tracking).
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present:
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present,
AND *source_bonus* ``== 0`` or no source-name-hit chunk present:
the document order is IDENTICAL to the pre-phase walk — no re-sort
happens at all (byte-identical, LOCKED A3);
happens at all (byte-identical, LOCKED A3, mirrored for the second
bonus by LOCKED A3 of phase 191);
* otherwise the documents are ordered by
``(−effective_score, −best_cosine, document.path,
first_seen_index)`` — the bounded re-rank: a name-hit document
gets a head start on the fused scale, and an effective-score tie
resolves by cosine, then path, then the pre-bonus rank.
The bonus lives in the SELECTION layer only: the chunk objects are
The bonuses live in the SELECTION layer only: the chunk objects are
never modified — their ``score``/``cosine``/``fts_hit`` (the A8
gate's inputs) and ``query_log.top_score`` are untouched.
"""
@@ -790,6 +814,7 @@ def _selection_order(
best_score: dict[uuid.UUID, float] = {}
best_cosine: dict[uuid.UUID, float] = {}
any_name_hit: dict[uuid.UUID, bool] = {}
any_source_hit: dict[uuid.UUID, bool] = {}
first_seen: dict[uuid.UUID, int] = {}
for idx, rc in enumerate(sorted(chunks, key=lambda c: c.score, reverse=True)):
doc = rc.document
@@ -798,17 +823,24 @@ def _selection_order(
best_cosine[doc.id] = rc.cosine
if rc.name_hit:
any_name_hit[doc.id] = True
if rc.source_name_hit:
any_source_hit[doc.id] = True
else:
order.append(doc)
best_score[doc.id] = rc.score
best_cosine[doc.id] = rc.cosine
any_name_hit[doc.id] = rc.name_hit
any_source_hit[doc.id] = rc.source_name_hit
first_seen[doc.id] = idx
effective = {
doc.id: best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0)
doc.id: best_score[doc.id]
+ (bonus if any_name_hit[doc.id] else 0.0)
+ (source_bonus if any_source_hit[doc.id] else 0.0)
for doc in order
}
if bonus > 0.0 and any(any_name_hit[doc.id] for doc in order):
if (bonus > 0.0 and any(any_name_hit[doc.id] for doc in order)) or (
source_bonus > 0.0 and any(any_source_hit[doc.id] for doc in order)
):
order.sort(
key=lambda doc: (
-effective[doc.id],
@@ -826,17 +858,26 @@ def _selection_order(
def weak_hit_titles(
chunks: Sequence[RetrievedChunk],
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[str]:
"""Distinct parent-document titles of *chunks*, best SELECTION score
first (the phase-119 name-hit bonus applied — see
:func:`_selection_order`, D2, LOCKED A3).
first (the phase-119 name-hit bonus and the phase-191 source-name
bonus applied — see :func:`_selection_order`, D2/D3, LOCKED A3).
Deflection mode (PLAN §6, A8) is built from these *titles only* — the
LOW prompt and the "Maybe try" chips never see document content.
*source_bonus* defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting
(0.005); ``0`` reproduces the pre-phase-191 walk byte-identically.
"""
if bonus is None:
bonus = get_settings().name_hit_bonus
return [doc.title for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)]
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
return [
doc.title
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)
]
def select_documents_tiered(
@@ -953,6 +994,7 @@ def select_suggested(
chunks: Sequence[RetrievedChunk],
n: int | None = None,
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[Document]:
"""Top-N distinct parent documents in SELECTION rank order — the
phase-118 "start here" suggestion tier (LOCKED A3, re-revised by
@@ -975,7 +1017,10 @@ def select_suggested(
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
owner-tunable starting point); ``0`` reproduces the pre-phase walk
byte-identically (the kill switch, LOCKED A3).
byte-identically (the kill switch, LOCKED A3). *source_bonus*
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
D3 — the same bounded shape, the same kill switch, an independent
knob).
The returned rows carry the full document content, byte-identical —
the content is what the agent's ``read`` tool serves later (never
@@ -985,7 +1030,14 @@ def select_suggested(
top_n = n if n is not None else get_settings().suggested_docs
if bonus is None:
bonus = get_settings().name_hit_bonus
return [doc for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)[:top_n]]
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
return [
doc
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)[
:top_n
]
]
def select_related(
@@ -993,6 +1045,7 @@ def select_related(
excluded_ids: set[uuid.UUID],
cap: int,
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[Document]:
"""The documents ranked AFTER *excluded_ids* — the phase-118 related
tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap*
@@ -1012,7 +1065,10 @@ def select_related(
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
owner-tunable starting point); ``0`` reproduces the pre-phase walk
byte-identically (the kill switch, LOCKED A3).
byte-identically (the kill switch, LOCKED A3). *source_bonus*
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
D3 — the same bounded shape, the same kill switch, an independent
knob).
The returned rows carry the full document content, byte-identical
(the tier is metadata for the ``done`` frame and the durable
@@ -1020,8 +1076,10 @@ def select_related(
"""
if bonus is None:
bonus = get_settings().name_hit_bonus
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
out: list[Document] = []
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus):
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus):
if len(out) >= cap:
break
if doc.id in excluded_ids:
+254
View File
@@ -0,0 +1,254 @@
"""Source name/label/tags storage + the D2 source-name matching rule (phase 191, task 01).
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
the model when prompting if you have a project name or a standard name
you reference things by." This module is the whole non-HTTP surface of
that feature: the upsert/get/delete helpers over the ``source_labels``
table (migration 0030) and the pure matching rule the retrieval boost
(task 04) consumes.
**The alias contract (D1, owner-confirmed):** a label row is an ALIAS
beside the derived source name — ``Document.source``, the ``source/path``
document identity, the read/grep/``show_text`` locators, the citations
and the KB tree are ALL untouched by it. Renaming a source never
invalidates a saved chat, a shared page or a verbatim locator. Every
function here keys on the DERIVED name (``Document.source``); the
display name is data, never identity.
**The matching rule (D2):** a source is a SOURCE-NAME HIT when at least
one :func:`app.rag.retriever.name_hit_tokens` candidate (the phase-119
class-agnostic token list, reused UNCHANGED) equals — in the
:func:`app.rag.retriever._normalize_name` form, so case-insensitive and
accent-folded (phase 162) — a token of the source's ``display_name``,
``label`` or any ``tags`` entry. The phase-119 digit-class rule does
NOT apply to user-given names (the owner's names are not paths —
"gitea" must hit even though it is digitless; the digit rule exists to
keep prose tokens from matching PATHS). The one filter is
:data:`app.rag.retriever.NAME_TOKEN_MIN_LEN` (4) applied to the SOURCE
side too — a 2-char tag never matches.
Session convention (the phase-53 ``sources_meta`` style): the write
helpers only *flush* — the **caller** commits.
"""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Final
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models import SourceLabel
from app.rag.retriever import NAME_TOKEN_MIN_LEN, _normalize_name, name_hit_tokens
#: Boundary limits (phase 191 D1/D5) — the API (task 02) turns the
#: ``normalize_tags`` raises into the framework's standard 422; the
#: display-name / label caps are enforced there against these constants.
DISPLAY_NAME_MAX_CHARS: Final = 120
LABEL_MAX_CHARS: Final = 200
TAG_MAX_CHARS: Final = 32
TAGS_MAX_COUNT: Final = 10
class _Unset:
"""Marker type for "field omitted" in :func:`set_label`.
The partial-update shape (phase-168 D5): an OMITTED field leaves the
stored value untouched, an explicit ``None`` CLEARS it — the two are
indistinguishable with plain ``None`` defaults, so the sentinel owns
the difference (the API passes only the fields the request carried).
"""
__slots__ = ()
_UNSET: Final = _Unset()
@dataclass(frozen=True)
class SourceLabelView:
"""Immutable read-side snapshot of one label row (D1).
``tags`` is a ``tuple`` so the view stays hashable/mutable-proof
after the ORM row leaves the session; ``display_name`` / ``label``
are ``None`` when unset (never the empty string — the API trims
empty submissions to NULL).
"""
source: str
display_name: str | None
label: str | None
tags: tuple[str, ...]
def _view(row: SourceLabel) -> SourceLabelView:
return SourceLabelView(
source=row.source,
display_name=row.display_name,
label=row.label,
tags=tuple(row.tags or ()),
)
def get_label(db: Session, source: str) -> SourceLabel | None:
"""The ORM row for *source* (one PK read); ``None`` when absent."""
return db.get(SourceLabel, source)
def label_for(db: Session, source: str) -> SourceLabelView | None:
"""The frozen :class:`SourceLabelView` for *source* (one PK read);
``None`` when absent — the "no label row = pre-phase" contract."""
row = db.get(SourceLabel, source)
return None if row is None else _view(row)
def all_labels(db: Session) -> dict[str, SourceLabelView]:
"""Every label row as a frozen view, keyed by ``source``.
ONE query (the phase-119 one-query convention): the table carries a
handful of rows, so callers match in Python against it instead of
issuing a query per source. Ordered by ``source`` for determinism.
"""
rows = db.scalars(select(SourceLabel).order_by(SourceLabel.source)).all()
return {row.source: _view(row) for row in rows}
def set_label(
db: Session,
source: str,
*,
display_name: str | None | _Unset = _UNSET,
label: str | None | _Unset = _UNSET,
tags: Sequence[str] | None | _Unset = _UNSET,
) -> SourceLabel:
"""Upsert the label row for *source*; return the ORM row.
The partial-update shape (phase 191 D5, the phase-168 precedent):
a field left at the ``_UNSET`` sentinel keeps its stored value, an
explicitly passed value OVERWRITES it, an explicit ``None`` CLEARS
(``tags`` clears to ``[]`` — the column is NOT NULL). A missing row
is created with the passed values (omitted fields = their defaults:
``NULL`` / ``[]``). Values arrive ALREADY VALIDATED (the API layer
trims/enforces the caps via :func:`normalize_tags`); this helper
stores what it is given.
Flush-only (the phase-53 session convention): the **caller** commits.
"""
row = db.get(SourceLabel, source)
if row is None:
row = SourceLabel(
source=source,
display_name=None if isinstance(display_name, _Unset) else display_name,
label=None if isinstance(label, _Unset) else label,
tags=[] if isinstance(tags, _Unset) or tags is None else list(tags),
)
db.add(row)
else:
if not isinstance(display_name, _Unset):
row.display_name = display_name
if not isinstance(label, _Unset):
row.label = label
if not isinstance(tags, _Unset):
row.tags = [] if tags is None else list(tags)
db.flush()
return row
def delete_label(db: Session, source: str) -> bool:
"""Remove the label row for *source*; ``False`` when absent (the
API's 404 shape), ``True`` when a row was deleted. Flush-only."""
row = db.get(SourceLabel, source)
if row is None:
return False
db.delete(row)
db.flush()
return True
def normalize_tags(raw: Sequence[str]) -> list[str]:
"""Normalize one tag submission (pure, the API's 422 source).
* trim every entry, drop the ones that are empty afterwards
(``[]`` is legal — it clears the tags);
* raise :class:`ValueError` above :data:`TAGS_MAX_COUNT` entries
(counted on the trimmed non-empty list, BEFORE the case-insensitive
de-dupe: 11 typed tags are 11 rejected tags);
* raise :class:`ValueError` when any entry exceeds
:data:`TAG_MAX_CHARS` afterwards;
* de-duplicate case-insensitively, preserving the FIRST case seen
(``Gitea`` then ``gitea`` keeps ``Gitea``), order otherwise kept.
"""
cleaned = [tag.strip() for tag in raw]
cleaned = [tag for tag in cleaned if tag]
if len(cleaned) > TAGS_MAX_COUNT:
raise ValueError(f"at most {TAGS_MAX_COUNT} tags are allowed")
for tag in cleaned:
if len(tag) > TAG_MAX_CHARS:
raise ValueError(f"each tag must be at most {TAG_MAX_CHARS} characters")
out: list[str] = []
seen: set[str] = set()
for tag in cleaned:
key = tag.casefold()
if key in seen:
continue
seen.add(key)
out.append(tag)
return out
def _source_side_tokens(view: SourceLabelView) -> set[str]:
"""The normalized token set of one label row (the D2 source side).
Every whitespace token of ``display_name`` / ``label`` / every
``tags`` entry, folded through :func:`_normalize_name` (the single
normalization BOTH sides of name matching pass through, phase 162)
and kept at :data:`NAME_TOKEN_MIN_LEN` chars minimum — the 4-char
floor applies to the SOURCE side too (a 2-char tag never matches).
"""
tokens: set[str] = set()
fields: list[str] = [view.display_name or "", view.label or "", *view.tags]
for field in fields:
for raw in field.split():
norm = _normalize_name(raw)
if len(norm) >= NAME_TOKEN_MIN_LEN:
tokens.add(norm)
return tokens
def question_name_hits(view: SourceLabelView | None, question: str) -> bool:
"""The D2 rule (pure): does *question* name the source behind *view*?
``None`` (no label row) → ``False`` — the pre-phase contract. Any
:func:`name_hit_tokens` candidate EQUAL (in the normalized form) to
a source-side token is a hit: no digit-class rule, no prefix match
(user-given names have no prose-precision problem — the phase-119
digit rule guards PATHS, not names).
"""
if view is None:
return False
candidates = name_hit_tokens(question)
if not candidates:
return False
source_tokens = _source_side_tokens(view)
return any(candidate in source_tokens for candidate in candidates)
def question_name_hit_sources(db: Session, question: str) -> set[str]:
"""The derived source names the question names (D2) — the standalone
one-query entry point (no other per-turn work needed).
ONE :func:`all_labels` read + the pure rule per row (a handful of
rows, matched in Python — the phase-119 one-query convention). The
empty set is the common case: no label rows, or no name in the
question.
The chat turn does NOT call this: since task 05 it inlines the same
pair against its own single ``all_labels`` read — the read that also
feeds the prompt surfaces (D4) — so a turn issues ONE label query.
"""
return {
view.source
for view in all_labels(db).values()
if question_name_hits(view, question)
}