phase: 191_source_name_labels_tags
Build and Push Containers / build-and-push-app (push) Successful in 7m24s
Build and Push Containers / build-and-push-db (push) Successful in 36s

All gates re-verified against a stable tree. Report:

**Verified (all 6 tasks in `complete/`; this was the final pass)**
- `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`.
- **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned).
- **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code.
- **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria.
- **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
This commit is contained in:
2026-10-19 06:13:32 -04:00
parent c6ffa050a6
commit 4f20c256d7
70 changed files with 7378 additions and 50 deletions
+51 -1
View File
@@ -258,7 +258,7 @@ import asyncio
import json
import logging
import time
from collections.abc import AsyncIterator, Sequence
from collections.abc import AsyncIterator, Mapping, Sequence
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
@@ -301,6 +301,11 @@ from app.rag.retriever import (
weak_hit_titles,
)
from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter
from app.rag.source_labels import (
SourceLabelView,
all_labels,
question_name_hits,
)
from app.rag.suggestions import derive_suggestions
from app.rag.summarizer import (
IMAGE_FALLBACK_MIME,
@@ -461,6 +466,7 @@ def plan_turn(
notes: Sequence[str] | None = None,
kb_overview: str | None = None,
folder_lines: Sequence[str] = (),
names: Mapping[str, SourceLabelView] | None = None,
) -> TurnPlan:
"""Apply the honesty gate (A8, revised) and assemble prompt + context.
@@ -502,6 +508,15 @@ def plan_turn(
phase-118 shape. The :class:`TurnPlan`, the durable record, and the
per-turn log line are untouched.
*names* (phase 191, D4): the source-name → ``SourceLabelView`` map of
the owner's label rows — the SAME single ``all_labels`` read the
endpoint performs for the task-04 selection boost (ONE query per
turn). The HIGH branch hands it to ``build_high_prompt`` (the
additive ``name`` / ``label`` attributes of the ``<document>``
blocks); the LOW branch IGNORES it — the deflected prompt stays
byte-identical. ``None`` ⇒ every block is byte-identical to
pre-phase.
``summary_hits`` (phase 30; phase 118 redefinition) counts the hit
chunks with ``is_summary`` whose parent document is in the
SUGGESTED set — both the HIGH and the LOW branch record it.
@@ -561,6 +576,10 @@ def plan_turn(
# byte-identical to pre-phase (no clause, and agent_tools
# offers four tools, not five).
show_text=settings.answer_text,
# Phase 191 (D4, task 05): the owner's source names ride the
# same map the boost flags came from — one all_labels read
# per turn, no second label query here.
names=names,
),
suggested,
related_docs,
@@ -860,6 +879,33 @@ async def chat(
kb_overview = load_kb_overview(step_db)
with SessionLocal() as step_db:
chunks = retrieve(step_db, request.message, question_vec)
# Phase 191 (D3 task 04 + D4 task 05): ONE
# all_labels read per turn feeds BOTH surfaces —
# the source-name hit flags here AND the prompt
# surfaces (the map rides into plan_turn →
# build_high_prompt's name/label attributes). The
# hit set is the pure D2 rule per view, then a
# loop over the few dozen retrieved chunks against
# the handful of hit sources (never a per-chunk
# query — task 01's question_name_hit_sources is
# this same pair, inlined here so the read is
# shared). The flags ride the chunk objects into
# EVERY selection call below (plan_turn's walk
# included); the source_name_hit_bonus itself is
# pulled from settings inside the selection
# helpers. No label rows → empty map → no flags
# and no name attributes → every surface is the
# pre-phase one; the per-turn log line is
# untouched (the boost is invisible to it).
labels = all_labels(step_db)
source_name_hits = {
view.source
for view in labels.values()
if question_name_hits(view, request.message)
}
for rc in chunks:
if rc.document.source in source_name_hits:
rc.source_name_hit = True
# Phase 119 (D3, LOCKED A4): the suggested-folder
# context — the SAME deterministic suggested walk
# plan_turn performs internally (one extra cheap
@@ -880,6 +926,10 @@ async def chat(
notes=steering_notes,
kb_overview=kb_overview,
folder_lines=folder_lines,
# Phase 191 (D4, task 05): the SAME single read —
# the HIGH prompt learns the name→docs mapping
# from these views (no second label query).
names=labels,
)
except Exception: # noqa: BLE001 — DB failure mid-turn
logger.exception(
+249
View File
@@ -0,0 +1,249 @@
"""Source name/label/tags admin API (phase 191, task 02).
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
the model when prompting if you have a project name or a standard name
you reference things by." Task 01 stored that (the ``source_labels``
table + :mod:`app.rag.source_labels`); this module is its whole HTTP
surface — the API the Sources page's per-source editor (task 03)
drives, admin-gated like the phase-91 Theme tab and the phase-168
Settings panel.
Routes (under ``/api`` via the ``main`` registration — the house prefix
convention, the :mod:`app.api.ui_settings` /
:mod:`app.api.admin_settings` style, so the wire paths are exactly the
D5 ones):
* ``GET /api/source-labels`` — the FULL source catalogue: (a) every
``git_sources`` row's DERIVED source name (git AND local rows —
:func:`app.rag.source_removal.resolve_source_name`, the one helper
that mirrors the import pipeline's naming for both kinds — and, while
the table is empty, the ``BOR_GIT_SOURCES`` env names, which
:func:`app.rag.git_sources.effective_sources` already surfaces as
synthetic git rows), plus (b) every indexed document source no row
derives (the uploads — ONE ``SELECT DISTINCT source``, no ``content``,
the phase-126 load discipline). De-duplicated (the phase-69 sibling
case: two rows, one name) and sorted by ``source``, so the page and
the phase's E2E always see the same order. Each entry carries its
label row or the ``null``/``[]`` defaults — one
:func:`app.rag.source_labels.all_labels` read for the whole list,
never a query per source. Reads only; creates nothing.
* ``PUT /api/source-labels/{source}`` — the upsert. The path param is
capped at the ``documents.source`` 120 (the framework's 422 above
it); a ``source`` matching NO registry row AND NO indexed document is
a 404 BEFORE anything is written (D5: no orphan label rows — a label
for a source that exists nowhere would be a dead boost input, and
nothing in the UI could ever list it). The body is the partial-update
shape (the phase-168 precedent): an omitted field keeps its stored
value, an explicit ``null`` clears it, a value sets it. The tags go
through :func:`app.rag.source_labels.normalize_tags` — the ONE tag
rule (trim, case-insensitive de-dupe, first case wins) — and its
``ValueError`` becomes the framework's 422, NEVER a 500. The router
owns the session here (the phase-91 commit convention): ``set_label``
flushes, this route commits, and a rejected request writes NOTHING.
200 → the stored :class:`SourceLabelOut`.
* ``DELETE /api/source-labels/{source}`` — the reset: the row goes, 404
when there is no row to remove. Like the phase-168
``DELETE /api/admin/settings/{key}`` the 200 body is the entry's
post-reset state — the same shape with the fields cleared — so the
row re-renders from the response it already has.
One invariant the routes exist to protect (D1, owner-confirmed): a
label row is an ALIAS. ``Document.source``, the ``source/path`` document
identity, the read/grep/``show_text`` locators, the citations and the KB
tree are ALL untouched by anything here — calling "gitea-forge" "the git
server" changes only what the lite model and the chat model are TOLD
(tasks 04/05), never what a saved chat, a shared page or a verbatim
panel points at. That is why ``source`` is a path param that is LOOKED
UP and never rewritten, and why the catalogue's identity column stays
the derived name.
Logging (AGENTS rule 10): one INFO line per WRITE, carrying the action
and the source, never the values (the ``admin_settings`` discipline —
the owner's text belongs in the row, not in ``server.log``). GETs log
nothing here: :class:`app.core.response_budget.ResponseBudgetMiddleware`
already writes the per-response ``api method=… bytes=…`` line for every
``/api/*`` route.
"""
from __future__ import annotations
import logging
from typing import Annotated, Any
from fastapi import APIRouter, Depends, HTTPException, Path
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.core.auth import require_admin
from app.db import get_db
from app.models import Document
from app.rag.git_sources import effective_sources
from app.rag.source_labels import (
SourceLabelView,
all_labels,
delete_label,
normalize_tags,
set_label,
)
from app.rag.source_removal import resolve_source_name
from app.schemas import SourceLabelList, SourceLabelOut, SourceLabelUpdate
logger = logging.getLogger("app.api.source_labels")
router = APIRouter(
prefix="/source-labels",
tags=["source-labels"],
dependencies=[Depends(require_admin)], # phase 191: the owner's naming surface
)
def _catalogue_names(db: Session) -> list[str]:
"""Every source the owner can label, de-duplicated and sorted.
Exactly the two reads the D5 catalogue is made of — no third:
(a) the registry: :func:`app.rag.git_sources.effective_sources` —
the ``git_sources`` rows of BOTH kinds in ``(added_at, id)``
order, or the synthetic env rows while the table is empty —
named by :func:`app.rag.source_removal.resolve_source_name`, the
single helper that spells a row's name the way the importer
stores it in ``Document.source`` (``repo_name`` of the URL for
git rows, the expanded path's basename for local rows). Reused,
never re-derived: an expression copied here would drift from the
importer and the label rows would key nothing.
(b) the indexed documents: ``SELECT DISTINCT source`` — one query
over the indexed corpus (the column only, never the entity: no
``content``, phase-126/185 discipline), so an upload — or any
document whose registry row is gone — still lists and stays
labelable.
Sorted because the page renders this order and the phase's E2E pins
it; the set collapses the phase-69 sibling case (two rows resolving
to one name) to a single entry.
"""
rows, _origin = effective_sources(db)
names: set[str] = {resolve_source_name(row) for row in rows}
names.update(db.scalars(select(Document.source).distinct()).all())
return sorted(names)
def _shape(source: str, view: SourceLabelView | None) -> SourceLabelOut:
"""One entry of the catalogue: the label row's values, or the
``null``/``[]`` defaults when the source has no row (the pre-phase
state — indistinguishable from a cleared label, and that is the
point: D1 says a label is data beside the name, never a rename)."""
return SourceLabelOut(
source=source,
display_name=view.display_name if view else None,
label=view.label if view else None,
tags=list(view.tags) if view else [],
)
@router.get("", response_model=SourceLabelList)
def list_source_labels(
db: Session = Depends(get_db), # noqa: B008
) -> SourceLabelList:
"""The catalogue: every source of every kind, each with its label or
the null defaults, sorted by ``source``.
Two reads for the names (:func:`_catalogue_names`) plus ONE for all
the label rows — the table carries a handful of rows, so the list is
assembled in Python instead of issuing a lookup per source (the
phase-119 one-query convention).
The catalogue is the SOURCE list, not the label list: a label row
whose source has since lost both its registry row and its documents
does not appear here (it can boost nothing — the D2 match runs
against retrieved chunks' ``document.source``), and it stays
clearable through ``DELETE`` below, which never consults the
registry.
"""
labels = all_labels(db)
return SourceLabelList(
labels=[_shape(name, labels.get(name)) for name in _catalogue_names(db)]
)
@router.put("/{source}", response_model=SourceLabelOut)
def put_source_label(
source: Annotated[str, Path(max_length=120)],
payload: SourceLabelUpdate,
db: Session = Depends(get_db), # noqa: B008
) -> SourceLabelOut:
"""Set or clear one source's name/label/tags (200 → the stored row).
The order is the D5 contract:
1. **404 first** — *source* matches no registry row name and no
indexed document (the same two reads the GET makes): a label for
a source that exists nowhere is refused and nothing is written.
A typo'd name would otherwise sit in the table with no row to
join, no documents to boost and nothing in the UI to list it.
2. **Then the tags** — :func:`~app.rag.source_labels.normalize_tags`
(trim, drop the entries that are empty afterwards, de-duplicate
case-insensitively with the first case kept); its ``ValueError``
becomes a 422 carrying its own fixed, input-free reason. The
request shape caps the same numbers, so today the raise is the
belt — the pin is that the belt can never fire as a 500.
3. **Then the write** — only the fields the request actually carried
(``model_fields_set`` — the phase-168 partial-update shape:
omitted keeps, ``null`` clears, a value sets) reach
:func:`~app.rag.source_labels.set_label`, and the commit happens
here (the router owns the session, the phase-91 convention). A
rejection at step 1 or 2 leaves any existing row byte-identical.
An all-omitted body is therefore a legitimate 200 no-op (the
``GitSourcePatchIn`` "all absent → 200 no-op" precedent): the row is
touched and reported, nothing changes.
"""
if source not in _catalogue_names(db):
raise HTTPException(status_code=404, detail="source not found")
write: dict[str, Any] = {}
carried = payload.model_fields_set
if "display_name" in carried:
write["display_name"] = payload.display_name
if "label" in carried:
write["label"] = payload.label
if "tags" in carried:
try:
write["tags"] = normalize_tags(payload.tags or [])
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
row = set_label(db, source, **write)
db.commit()
# The action and the key only — never the owner's text (AGENTS rule
# 10, the ``admin_settings`` write-line discipline).
logger.info(
"source labels action=set source=%s fields=%s",
source,
",".join(sorted(write)) or "none",
)
return SourceLabelOut(
source=row.source,
display_name=row.display_name,
label=row.label,
tags=list(row.tags or []),
)
@router.delete("/{source}", response_model=SourceLabelOut)
def delete_source_label(
source: Annotated[str, Path(max_length=120)],
db: Session = Depends(get_db), # noqa: B008
) -> SourceLabelOut:
"""Reset one source — the row goes and the next turn sees the
pre-phase state.
404 (one fixed detail, ``source label not found``) when there is no
label row for it. Unlike the PUT this does NOT consult the registry
or the documents: the reset has to work for a row whose source has
since been pruned or whose registry row was deleted (phase 69's
total removal), or the label would be un-clearable by construction.
"""
if not delete_label(db, source):
raise HTTPException(status_code=404, detail="source label not found")
db.commit()
logger.info("source labels action=clear source=%s", source)
return SourceLabelOut(source=source, display_name=None, label=None, tags=[])
+28
View File
@@ -430,6 +430,23 @@ class Settings(BaseSettings):
#: records the realized margins).
name_hit_bonus: float = 0.005
#: (phase 191, D3, LOCKED A3): the SECOND, independent selection-time
#: bonus — a document whose SOURCE is named by the question (the D2
#: rule over the ``source_labels`` row: a question token equals, in
#: normalized form, a token of the source's display_name / label /
#: tags) gets this added to its best fused chunk score in the SAME
#: shared selection walk as the phase-119 ``name_hit_bonus`` (a
#: separate knob: the two signals stay independently kill-able).
#: Default 0.005 = the phase-119 starting point; ``0`` = off (the
#: walk returns byte-identical to pre-phase — the kill switch); a
#: negative value fails startup loudly. Bounded per DOCUMENT, applied
#: ONCE, at most TWO bonuses on the fused scale (name_hit + source
#: name_hit = 0.010 maximum head start — no amplification); the
#: bonus lives in the SELECTION layer only — ``score`` / ``cosine`` /
#: ``fts_hit`` / ``query_log.top_score`` are untouched (the
#: phase-119 LOCKED A3 clause, mirrored).
source_name_hit_bonus: float = 0.005
# --- Admin & sign-in (phase 16; A10 revised 2026-08-22) ---
# Single-admin auth via a signed session cookie (Starlette
# SessionMiddleware — no new services, no DB tables). Both secrets are
@@ -972,6 +989,17 @@ class Settings(BaseSettings):
raise ValueError("name_hit_bonus must be >= 0 (0 = off)")
return v
@field_validator("source_name_hit_bonus")
@classmethod
def _source_name_hit_bonus_non_negative(cls, v: float) -> float:
"""``0`` is the byte-identical kill switch (the pre-phase-191
selection order) — a NEGATIVE bonus would demote source-named
documents, the exact opposite of D2/D3 (the ``name_hit_bonus``
pattern, phase 191)."""
if v < 0:
raise ValueError("source_name_hit_bonus must be >= 0 (0 = off)")
return v
@field_validator("db_pool_size")
@classmethod
def _db_pool_size_positive(cls, v: int) -> int:
+7
View File
@@ -225,6 +225,13 @@ CATALOG: dict[str, SettingMeta] = {
"name_hit_bonus": SettingMeta(
"name_hit_bonus", "Retrieval & honesty", "Name hit bonus"
),
# Phase 191 (D3): the second, independent selection bonus — the SAME
# live shape as name_hit_bonus (the selection walk reads
# get_settings() per turn, so a saved override applies on the next
# question).
"source_name_hit_bonus": SettingMeta(
"source_name_hit_bonus", "Retrieval & honesty", "Source name hit bonus"
),
# --- Chunking & embedding ---
"chunk_target_chars": SettingMeta(
"chunk_target_chars", "Chunking & embedding", "Chunk target chars"
+6
View File
@@ -39,6 +39,7 @@ from app.api.doc_drafts import router as doc_drafts_router
from app.api.docs import router as docs_router
from app.api.git_sources import router as git_sources_router
from app.api.health import router as health_router
from app.api.source_labels import router as source_labels_router
from app.api.steering import router as steering_router
from app.api.suggestions import router as suggestions_router
from app.api.sync import router as sync_router
@@ -234,6 +235,11 @@ def create_app() -> FastAPI:
# the app_settings overrides — DB-over-env, live on Save) — admin-only
# (router-wide require_admin; a token user stays 403).
app.include_router(admin_settings_router, prefix="/api")
# Phase 191 (task 02): the per-source name/label/tags surface (the
# catalogue GET + the upsert + the reset) — admin-only (router-wide
# require_admin; a token user stays 403). The display name is an
# ALIAS beside the derived source name, never a rename (D1).
app.include_router(source_labels_router, prefix="/api")
# Phase 51: the anonymous shared-chat read — NO admin dependency.
# /api/shared/<token> is the JSON snapshot; /shared/<token> (the
# page route below, registered without a prefix) is the page.
+62
View File
@@ -77,6 +77,14 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
name, the value in its env-native string form;
the row WINS over the environment (resolved
inside ``app.config.get_settings``).
* ``source_labels`` — one row per source the owner has named (phase
191, TODO.md L4): the display name ("the git
server"), the short label/category, and the tag
list used to focus the model when prompting.
``source`` mirrors ``documents.source`` (the
DERIVED name) and is an ALIAS key only — a label
row never renames a document, a locator, or a
citation (phase 191 D1).
"""
from __future__ import annotations
@@ -717,3 +725,57 @@ class AppSetting(Base):
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
class SourceLabel(Base):
"""One owner-given name / label / tag set per source (phase 191, task 01).
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can
focus the model when prompting if you have a project name or a
standard name you reference things by." One row per SOURCE (not per
document), keyed by the source name exactly as ``Document.source``
carries it (the derived basename — the clone dir / local dir /
upload file stem), so the row joins the documents and the
``git_sources`` row without a second identity.
**The display name is an ALIAS ONLY (D1, owner-confirmed):**
``Document.source``, the ``source/path`` document identity, the
read/grep/``show_text`` locators, the citations and the KB tree are
ALL untouched by a label row — renaming a source never invalidates
a saved chat, a shared page or a verbatim locator. The derived name
stays the wire identity; the display name rides beside it and is
consumed by the retrieval boost (task 04) and the prompt surfacing
(task 05) only.
No row = no name/label/tags = the pre-phase behavior exactly (the
migration creates an EMPTY table — no backfill, A13 reversible).
"""
__tablename__ = "source_labels"
#: Mirrors ``documents.source`` (String(120)) — the DERIVED source
#: name, the join key and the sole PK (one label row per source;
#: the ``folder_summaries`` PK-part precedent).
source: Mapped[str] = mapped_column(String(120), primary_key=True)
#: The owner's name for the source ("the git server"). Plain TEXT —
#: the ≤ 120-char trimmed limit is enforced at the API boundary
#: (the phase-121 ``git_sources.branch`` pattern: plain column,
#: boundary validation). NULL = no display name (pre-phase).
display_name: Mapped[str | None] = mapped_column(Text, default=None)
#: The short one-line description / category ("self-hosted forge +
#: Gitea"). Plain TEXT, ≤ 200 trimmed enforced at the API. NULL =
#: no label.
label: Mapped[str | None] = mapped_column(Text, default=None)
#: The focus tokens (the owner's "tag(s)"): ≤ 10 entries, each ≤ 32
#: chars, non-empty, de-duplicated case-insensitively — normalized
#: and guarded at the API boundary (``app.rag.source_labels.normalize_tags``).
#: JSONB NOT NULL with server default ``'[]'`` (the phase-89
#: ``ignore_paths`` shape): a source with no tags reads ``[]``.
tags: Mapped[list] = mapped_column(
JSONB, default=list, server_default=text("'[]'"), nullable=False
)
#: Fresh stamp on every write (the ``sources_meta.updated_at`` house
#: style: server-side now(), onupdate now()).
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
)
+43 -4
View File
@@ -43,6 +43,14 @@ tail (the model input stays byte-identical). The longest-prefix cap in
one-line tail (~15 chars/source) always fit; a smaller owner-set cap
cuts the tail first — accepted, the cap is owner-tunable.
Source names (phase 191, D4): when a source has a label row with a
``display_name``, the INPUT line of its documents leads with that name
instead of the derived source name (``build_overview_prompt``'s *names*
map, derived from the ONE ``all_labels`` read :func:`regenerate_overview`
performs) — the owner's vocabulary reaches the outline. The stored text,
the catalogue rows and the deterministic ``Sources: …`` inventory tail
keep the DERIVED name (D1 — a label is an alias, never an identity).
The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
in the system prompt (wired in phase 31, task 05).
@@ -50,7 +58,7 @@ in the system prompt (wired in phase 31, task 05).
from __future__ import annotations
import logging
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from datetime import UTC, datetime
from typing import Protocol
@@ -62,6 +70,7 @@ from app.db import SessionLocal
from app.models import Document, KbOverview
from app.rag.llm import LLMError
from app.rag.retriever import TRUNCATION_MARKER
from app.rag.source_labels import all_labels
logger = logging.getLogger("app.rag.overview")
@@ -146,7 +155,9 @@ def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
def build_overview_prompt(
rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
rows: Sequence[tuple[str, str, str, str | None]],
max_chars: int | None = None,
names: Mapping[str, str] | None = None,
) -> tuple[str, str]:
"""The ``(system, user)`` message pair for one overview call.
@@ -166,11 +177,25 @@ def build_overview_prompt(
cap and the shared ``[…truncated…]`` marker is appended on its own
line, so the model never sees more than the cap and the cut is
visible (summarizer convention, phase 30).
*names* (phase 191, D4) maps a DERIVED source name to the owner's
display name for it: the input line leads with the display name
instead of the raw source name, so the ``lite`` outline is written in
the owner's vocabulary ("the git server" rather than
``gitea-infra-docs``). The alias contract (D1) is untouched — the
``path`` / ``title`` / summary halves of the line, the stored overview
row and its ``Sources: …`` inventory tail all keep the derived name;
only this model input changes. ``None`` / an empty map / no entry for
a source (or an entry with an empty name) → the pre-phase line,
byte-identical.
"""
limit = max_chars if max_chars is not None else get_settings().overview_input_max_chars
lines: list[str] = []
for source, path, title, summary in rows:
line = f"{source} — {path} — {title}"
# Phase 191 (D4): the display name REPLACES the raw source name in
# the lite model's input only — never in the stored outline.
display = names.get(source) if names else None
line = f"{display or source} — {path} — {title}"
first = _first_summary_line(summary)
if first:
line += f" — {first}"
@@ -220,6 +245,10 @@ async def regenerate_overview(
never sees the tail; the zero-doc and ``LLMError`` paths below
leave a previous row byte-identical — an existing tail is part of
the stored content and is preserved on failure.
* Phase 191 (D4): the ONE :func:`app.rag.source_labels.all_labels`
read happens AFTER the zero-doc gate (no label query when there is
nothing to outline) and feeds only the model input — the stored
content and its inventory tail stay derived-name text.
* :class:`LLMError` → ``overview: regeneration failed — …`` is
logged and ``False`` is returned with the previous row intact —
an old outline is better than none.
@@ -249,7 +278,17 @@ async def regenerate_overview(
)
return False
system, user = build_overview_prompt(rows)
# Phase 191 (D4, task 05): ONE all_labels read for the whole run —
# the display-name map for the model input is derived from these
# views in this loop, never re-queried (the same single-read
# contract the chat turn holds, task 04/05). Rows without a
# display name are absent from the map, so their lines stay
# byte-identical to pre-phase.
labels = all_labels(session)
name_map = {
src: view.display_name for src, view in labels.items() if view.display_name
}
system, user = build_overview_prompt(rows, names=name_map)
try:
text = await llm.chat(
[{"role": "system", "content": system}, {"role": "user", "content": user}],
+76 -2
View File
@@ -100,16 +100,32 @@ owner decision). The deterministic lever is the honesty gate itself
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
``best_cosine >= lexical_support_floor``, so the misleading docs are
no longer injected — see :mod:`app.api.chat`).
Source names, labels and tags (phase 191, D4): a ``<document>`` block can
carry the owner's label row for its source as two ADDITIVE attributes —
``name="…"`` (the source's ``display_name``) immediately after
``source="…"``, then ``label="…"`` (the one-line description) — so the
model learns the name→docs mapping ("the git server" is a source with
these documents). The attribute order is fixed
(``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``),
an unset field adds NOTHING, and no *names* map / no entry for the
source leaves the block byte-identical to pre-phase (the alias contract,
D1: the identity attributes stay the DERIVED name — a display name never
replaces ``source`` or ``path``). Only the two new values are XML-escaped
(they are free-form owner text); the identity attributes keep their
pre-phase unescaped bytes.
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from typing import Any
from xml.sax.saxutils import escape
from app.config import Settings, get_settings
from app.models import Document
from app.rag.retriever import TRUNCATION_MARKER
from app.rag.source_labels import SourceLabelView
from app.schemas import HistoryTurn
#: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by
@@ -769,6 +785,47 @@ def _extract_context_line(image_context: str) -> str | None:
)
def _attr(value: str) -> str:
"""XML-escape one attribute value (phase 191, D4) — ``&``/``<``/``>``
and the double quote (``&quot;``), the delimiter the block uses.
Applied ONLY to the two additive label attributes (``name`` /
``label``): they hold free-form OWNER text where a ``"`` is realistic
and must not break the attribute. The pre-existing identity
attributes (``source`` / ``path`` / ``title`` / ``date``) deliberately
keep their unescaped pre-phase bytes — escaping them here would move
every label-less block, which the phase's byte-identity criterion
forbids.
"""
return escape(value, {'"': "&quot;"})
def _source_label_attrs(
doc: Document, names: Mapping[str, SourceLabelView] | None
) -> str:
"""The additive `` name="…"`` / `` label="…"`` fragment of one
``<document>`` block (phase 191, D4) — the empty string unless the
source has a label row with that field set.
``None`` / an empty map / no entry for ``doc.source`` / a row with
neither field set all render ``""`` — the block is then byte-identical
to pre-phase, and a label-only row adds exactly the one ``label``
attribute (an unset field adds NOTHING, it is never an empty
attribute).
"""
if not names:
return ""
view = names.get(doc.source)
if view is None:
return ""
attrs = ""
if view.display_name:
attrs += f' name="{_attr(view.display_name)}"'
if view.label:
attrs += f' label="{_attr(view.label)}"'
return attrs
def _image_context_line(doc: Document) -> str | None:
"""The phase-125 context line for one suggested document (task 05,
LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
@@ -832,6 +889,7 @@ def build_high_prompt(
folder_lines: Sequence[str] = (),
show_image: bool = False,
show_text: bool = False,
names: Mapping[str, SourceLabelView] | None = None,
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + the
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
@@ -917,6 +975,21 @@ def build_high_prompt(
(:func:`_document_body`; the NULL/whitespace-summary preview
fallback is LOCKED A5; an enriched image doc's summary leads with
the phase-125 one-line context line, LOCKED A4).
*names* (phase 191, D4): the source-name → :class:`SourceLabelView`
map of the owner's label rows (the SAME single ``all_labels`` read
that feeds the task-04 selection boost — one query per turn). When a
doc's source has a row, the block head gains `` name="{display_name}"``
immediately after ``source="…"`` and `` label="{label}"`` after that
(the fixed order ``source`` → ``name?`` → ``label?`` → ``path`` →
``title`` → ``date``); the two values are XML-escaped
(:func:`_attr`), the identity attributes are not. An unset field adds
NOTHING, so a label-only row gains exactly the one attribute, and
``None`` / ``{}`` / no entry for the source keeps the block
byte-identical to pre-phase (the default, and the kill pin in
:mod:`tests.unit.test_prompts`). This is surfacing only — the alias
contract (D1) keeps ``source`` / ``path`` as the document identity
the tools, citations and locators use.
"""
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
@@ -926,7 +999,8 @@ def build_high_prompt(
# stays out of the prompt (the capped ``read`` tool is the only
# full-text path).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'<document source="{doc.source}"{_source_label_attrs(doc, names)}'
f' path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{_document_body(doc)}\n"
"</document>"
+75 -17
View File
@@ -291,6 +291,16 @@ class RetrievedChunk:
:func:`fuse`'s ``replace()`` copies it (the double-hit merge ORs it
in — a vector row that is also the name hit's representative chunk
keeps the flag).
* ``source_name_hit`` — True for a chunk whose parent document's
SOURCE the question named (phase 191, D2/D3): the flag is set at
the retrieval boundary in ``app/api/chat.py`` from the turn's ONE
``all_labels`` read (the pure D2 rule per view — the same read that
feeds the ``name``/``label`` prompt attributes, task 05) — the
selection tier's SECOND bonus input (``source_name_hit_bonus``, a
separate knob). Default
``False`` keeps every ordinary construction valid; :func:`fuse`'s
``replace()`` copies it (the double-hit merge ORs it in, mirroring
``name_hit``).
"""
chunk_id: uuid.UUID
@@ -302,6 +312,7 @@ class RetrievedChunk:
fts_hit: bool = False
is_summary: bool = False
name_hit: bool = False
source_name_hit: bool = False
def lexical_tsquery(question: str) -> str | None:
@@ -339,9 +350,11 @@ def fuse(
Lexical-only hits (no vector rank) enter with ``cosine=0.0`` and
``fts_hit=True``; vector chunks matched by the lexical list get
``fts_hit=True`` in place (the input objects are mutated — callers
should not reuse them afterwards), and ``name_hit=True`` is ORed in
should not reuse them afterwards), ``name_hit=True`` is ORed in
(a vector row that is also the name hit's representative chunk is a
name-hit row — the phase-119 selection tier must see it).
name-hit row — the phase-119 selection tier must see it), and
``source_name_hit`` is ORed in the SAME way (phase 191, D3 — a
vector row merged with a flagged source-name row keeps the flag).
"""
if k <= 0:
raise ValueError("rrf k must be > 0")
@@ -355,7 +368,10 @@ def fuse(
if rc.chunk_id in by_id:
existing = by_id[rc.chunk_id]
by_id[rc.chunk_id] = replace(
existing, fts_hit=True, name_hit=existing.name_hit or rc.name_hit
existing,
fts_hit=True,
name_hit=existing.name_hit or rc.name_hit,
source_name_hit=existing.source_name_hit or rc.source_name_hit,
)
fused[rc.chunk_id] += term
else:
@@ -754,17 +770,23 @@ def retrieve(
def _selection_order(
chunks: Sequence[RetrievedChunk],
bonus: float,
source_bonus: float = 0.0,
) -> list[tuple[Document, float, float, int]]:
"""The shared selection walk of the phase-119 name-hit bonus (D2,
LOCKED A3) — one walk for ``select_suggested``, ``select_related``,
and ``weak_hit_titles``.
and ``weak_hit_titles``, carrying the phase-191 source-name bonus
(D3) as its SECOND, independent knob.
Returns ``(document, effective_score, best_cosine, first_seen_index)``
for each distinct document, where *effective_score* is the
document's best fused chunk score plus *bonus* when ANY of its
chunks carries ``name_hit`` (the D1 path match — the bonus is per
DOCUMENT, applied ONCE no matter how many of the document's chunks
are name hits).
are name hits), plus *source_bonus* when ANY of its chunks carries
``source_name_hit`` (the phase-191 D2 source-name match — the SAME
per-DOCUMENT, apply-ONCE rule; at most TWO bonuses per document
on the fused scale, bounded: name_hit_bonus + source_name_hit_bonus
= 0.010 at the shipped defaults, never more).
The walk keeps the EXISTING selection semantics (not just the loop):
the same stable score-descending order as
@@ -773,16 +795,18 @@ def _selection_order(
best cosine tracked across ALL of its chunks (the tiered walk's
tracking).
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present:
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present,
AND *source_bonus* ``== 0`` or no source-name-hit chunk present:
the document order is IDENTICAL to the pre-phase walk — no re-sort
happens at all (byte-identical, LOCKED A3);
happens at all (byte-identical, LOCKED A3, mirrored for the second
bonus by LOCKED A3 of phase 191);
* otherwise the documents are ordered by
``(−effective_score, −best_cosine, document.path,
first_seen_index)`` — the bounded re-rank: a name-hit document
gets a head start on the fused scale, and an effective-score tie
resolves by cosine, then path, then the pre-bonus rank.
The bonus lives in the SELECTION layer only: the chunk objects are
The bonuses live in the SELECTION layer only: the chunk objects are
never modified — their ``score``/``cosine``/``fts_hit`` (the A8
gate's inputs) and ``query_log.top_score`` are untouched.
"""
@@ -790,6 +814,7 @@ def _selection_order(
best_score: dict[uuid.UUID, float] = {}
best_cosine: dict[uuid.UUID, float] = {}
any_name_hit: dict[uuid.UUID, bool] = {}
any_source_hit: dict[uuid.UUID, bool] = {}
first_seen: dict[uuid.UUID, int] = {}
for idx, rc in enumerate(sorted(chunks, key=lambda c: c.score, reverse=True)):
doc = rc.document
@@ -798,17 +823,24 @@ def _selection_order(
best_cosine[doc.id] = rc.cosine
if rc.name_hit:
any_name_hit[doc.id] = True
if rc.source_name_hit:
any_source_hit[doc.id] = True
else:
order.append(doc)
best_score[doc.id] = rc.score
best_cosine[doc.id] = rc.cosine
any_name_hit[doc.id] = rc.name_hit
any_source_hit[doc.id] = rc.source_name_hit
first_seen[doc.id] = idx
effective = {
doc.id: best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0)
doc.id: best_score[doc.id]
+ (bonus if any_name_hit[doc.id] else 0.0)
+ (source_bonus if any_source_hit[doc.id] else 0.0)
for doc in order
}
if bonus > 0.0 and any(any_name_hit[doc.id] for doc in order):
if (bonus > 0.0 and any(any_name_hit[doc.id] for doc in order)) or (
source_bonus > 0.0 and any(any_source_hit[doc.id] for doc in order)
):
order.sort(
key=lambda doc: (
-effective[doc.id],
@@ -826,17 +858,26 @@ def _selection_order(
def weak_hit_titles(
chunks: Sequence[RetrievedChunk],
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[str]:
"""Distinct parent-document titles of *chunks*, best SELECTION score
first (the phase-119 name-hit bonus applied — see
:func:`_selection_order`, D2, LOCKED A3).
first (the phase-119 name-hit bonus and the phase-191 source-name
bonus applied — see :func:`_selection_order`, D2/D3, LOCKED A3).
Deflection mode (PLAN §6, A8) is built from these *titles only* — the
LOW prompt and the "Maybe try" chips never see document content.
*source_bonus* defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting
(0.005); ``0`` reproduces the pre-phase-191 walk byte-identically.
"""
if bonus is None:
bonus = get_settings().name_hit_bonus
return [doc.title for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)]
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
return [
doc.title
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)
]
def select_documents_tiered(
@@ -953,6 +994,7 @@ def select_suggested(
chunks: Sequence[RetrievedChunk],
n: int | None = None,
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[Document]:
"""Top-N distinct parent documents in SELECTION rank order — the
phase-118 "start here" suggestion tier (LOCKED A3, re-revised by
@@ -975,7 +1017,10 @@ def select_suggested(
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
owner-tunable starting point); ``0`` reproduces the pre-phase walk
byte-identically (the kill switch, LOCKED A3).
byte-identically (the kill switch, LOCKED A3). *source_bonus*
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
D3 — the same bounded shape, the same kill switch, an independent
knob).
The returned rows carry the full document content, byte-identical —
the content is what the agent's ``read`` tool serves later (never
@@ -985,7 +1030,14 @@ def select_suggested(
top_n = n if n is not None else get_settings().suggested_docs
if bonus is None:
bonus = get_settings().name_hit_bonus
return [doc for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)[:top_n]]
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
return [
doc
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)[
:top_n
]
]
def select_related(
@@ -993,6 +1045,7 @@ def select_related(
excluded_ids: set[uuid.UUID],
cap: int,
bonus: float | None = None,
source_bonus: float | None = None,
) -> list[Document]:
"""The documents ranked AFTER *excluded_ids* — the phase-118 related
tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap*
@@ -1012,7 +1065,10 @@ def select_related(
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
owner-tunable starting point); ``0`` reproduces the pre-phase walk
byte-identically (the kill switch, LOCKED A3).
byte-identically (the kill switch, LOCKED A3). *source_bonus*
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
D3 — the same bounded shape, the same kill switch, an independent
knob).
The returned rows carry the full document content, byte-identical
(the tier is metadata for the ``done`` frame and the durable
@@ -1020,8 +1076,10 @@ def select_related(
"""
if bonus is None:
bonus = get_settings().name_hit_bonus
if source_bonus is None:
source_bonus = get_settings().source_name_hit_bonus
out: list[Document] = []
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus):
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus):
if len(out) >= cap:
break
if doc.id in excluded_ids:
+254
View File
@@ -0,0 +1,254 @@
"""Source name/label/tags storage + the D2 source-name matching rule (phase 191, task 01).
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
the model when prompting if you have a project name or a standard name
you reference things by." This module is the whole non-HTTP surface of
that feature: the upsert/get/delete helpers over the ``source_labels``
table (migration 0030) and the pure matching rule the retrieval boost
(task 04) consumes.
**The alias contract (D1, owner-confirmed):** a label row is an ALIAS
beside the derived source name — ``Document.source``, the ``source/path``
document identity, the read/grep/``show_text`` locators, the citations
and the KB tree are ALL untouched by it. Renaming a source never
invalidates a saved chat, a shared page or a verbatim locator. Every
function here keys on the DERIVED name (``Document.source``); the
display name is data, never identity.
**The matching rule (D2):** a source is a SOURCE-NAME HIT when at least
one :func:`app.rag.retriever.name_hit_tokens` candidate (the phase-119
class-agnostic token list, reused UNCHANGED) equals — in the
:func:`app.rag.retriever._normalize_name` form, so case-insensitive and
accent-folded (phase 162) — a token of the source's ``display_name``,
``label`` or any ``tags`` entry. The phase-119 digit-class rule does
NOT apply to user-given names (the owner's names are not paths —
"gitea" must hit even though it is digitless; the digit rule exists to
keep prose tokens from matching PATHS). The one filter is
:data:`app.rag.retriever.NAME_TOKEN_MIN_LEN` (4) applied to the SOURCE
side too — a 2-char tag never matches.
Session convention (the phase-53 ``sources_meta`` style): the write
helpers only *flush* — the **caller** commits.
"""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Final
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.models import SourceLabel
from app.rag.retriever import NAME_TOKEN_MIN_LEN, _normalize_name, name_hit_tokens
#: Boundary limits (phase 191 D1/D5) — the API (task 02) turns the
#: ``normalize_tags`` raises into the framework's standard 422; the
#: display-name / label caps are enforced there against these constants.
DISPLAY_NAME_MAX_CHARS: Final = 120
LABEL_MAX_CHARS: Final = 200
TAG_MAX_CHARS: Final = 32
TAGS_MAX_COUNT: Final = 10
class _Unset:
"""Marker type for "field omitted" in :func:`set_label`.
The partial-update shape (phase-168 D5): an OMITTED field leaves the
stored value untouched, an explicit ``None`` CLEARS it — the two are
indistinguishable with plain ``None`` defaults, so the sentinel owns
the difference (the API passes only the fields the request carried).
"""
__slots__ = ()
_UNSET: Final = _Unset()
@dataclass(frozen=True)
class SourceLabelView:
"""Immutable read-side snapshot of one label row (D1).
``tags`` is a ``tuple`` so the view stays hashable/mutable-proof
after the ORM row leaves the session; ``display_name`` / ``label``
are ``None`` when unset (never the empty string — the API trims
empty submissions to NULL).
"""
source: str
display_name: str | None
label: str | None
tags: tuple[str, ...]
def _view(row: SourceLabel) -> SourceLabelView:
return SourceLabelView(
source=row.source,
display_name=row.display_name,
label=row.label,
tags=tuple(row.tags or ()),
)
def get_label(db: Session, source: str) -> SourceLabel | None:
"""The ORM row for *source* (one PK read); ``None`` when absent."""
return db.get(SourceLabel, source)
def label_for(db: Session, source: str) -> SourceLabelView | None:
"""The frozen :class:`SourceLabelView` for *source* (one PK read);
``None`` when absent — the "no label row = pre-phase" contract."""
row = db.get(SourceLabel, source)
return None if row is None else _view(row)
def all_labels(db: Session) -> dict[str, SourceLabelView]:
"""Every label row as a frozen view, keyed by ``source``.
ONE query (the phase-119 one-query convention): the table carries a
handful of rows, so callers match in Python against it instead of
issuing a query per source. Ordered by ``source`` for determinism.
"""
rows = db.scalars(select(SourceLabel).order_by(SourceLabel.source)).all()
return {row.source: _view(row) for row in rows}
def set_label(
db: Session,
source: str,
*,
display_name: str | None | _Unset = _UNSET,
label: str | None | _Unset = _UNSET,
tags: Sequence[str] | None | _Unset = _UNSET,
) -> SourceLabel:
"""Upsert the label row for *source*; return the ORM row.
The partial-update shape (phase 191 D5, the phase-168 precedent):
a field left at the ``_UNSET`` sentinel keeps its stored value, an
explicitly passed value OVERWRITES it, an explicit ``None`` CLEARS
(``tags`` clears to ``[]`` — the column is NOT NULL). A missing row
is created with the passed values (omitted fields = their defaults:
``NULL`` / ``[]``). Values arrive ALREADY VALIDATED (the API layer
trims/enforces the caps via :func:`normalize_tags`); this helper
stores what it is given.
Flush-only (the phase-53 session convention): the **caller** commits.
"""
row = db.get(SourceLabel, source)
if row is None:
row = SourceLabel(
source=source,
display_name=None if isinstance(display_name, _Unset) else display_name,
label=None if isinstance(label, _Unset) else label,
tags=[] if isinstance(tags, _Unset) or tags is None else list(tags),
)
db.add(row)
else:
if not isinstance(display_name, _Unset):
row.display_name = display_name
if not isinstance(label, _Unset):
row.label = label
if not isinstance(tags, _Unset):
row.tags = [] if tags is None else list(tags)
db.flush()
return row
def delete_label(db: Session, source: str) -> bool:
"""Remove the label row for *source*; ``False`` when absent (the
API's 404 shape), ``True`` when a row was deleted. Flush-only."""
row = db.get(SourceLabel, source)
if row is None:
return False
db.delete(row)
db.flush()
return True
def normalize_tags(raw: Sequence[str]) -> list[str]:
"""Normalize one tag submission (pure, the API's 422 source).
* trim every entry, drop the ones that are empty afterwards
(``[]`` is legal — it clears the tags);
* raise :class:`ValueError` above :data:`TAGS_MAX_COUNT` entries
(counted on the trimmed non-empty list, BEFORE the case-insensitive
de-dupe: 11 typed tags are 11 rejected tags);
* raise :class:`ValueError` when any entry exceeds
:data:`TAG_MAX_CHARS` afterwards;
* de-duplicate case-insensitively, preserving the FIRST case seen
(``Gitea`` then ``gitea`` keeps ``Gitea``), order otherwise kept.
"""
cleaned = [tag.strip() for tag in raw]
cleaned = [tag for tag in cleaned if tag]
if len(cleaned) > TAGS_MAX_COUNT:
raise ValueError(f"at most {TAGS_MAX_COUNT} tags are allowed")
for tag in cleaned:
if len(tag) > TAG_MAX_CHARS:
raise ValueError(f"each tag must be at most {TAG_MAX_CHARS} characters")
out: list[str] = []
seen: set[str] = set()
for tag in cleaned:
key = tag.casefold()
if key in seen:
continue
seen.add(key)
out.append(tag)
return out
def _source_side_tokens(view: SourceLabelView) -> set[str]:
"""The normalized token set of one label row (the D2 source side).
Every whitespace token of ``display_name`` / ``label`` / every
``tags`` entry, folded through :func:`_normalize_name` (the single
normalization BOTH sides of name matching pass through, phase 162)
and kept at :data:`NAME_TOKEN_MIN_LEN` chars minimum — the 4-char
floor applies to the SOURCE side too (a 2-char tag never matches).
"""
tokens: set[str] = set()
fields: list[str] = [view.display_name or "", view.label or "", *view.tags]
for field in fields:
for raw in field.split():
norm = _normalize_name(raw)
if len(norm) >= NAME_TOKEN_MIN_LEN:
tokens.add(norm)
return tokens
def question_name_hits(view: SourceLabelView | None, question: str) -> bool:
"""The D2 rule (pure): does *question* name the source behind *view*?
``None`` (no label row) → ``False`` — the pre-phase contract. Any
:func:`name_hit_tokens` candidate EQUAL (in the normalized form) to
a source-side token is a hit: no digit-class rule, no prefix match
(user-given names have no prose-precision problem — the phase-119
digit rule guards PATHS, not names).
"""
if view is None:
return False
candidates = name_hit_tokens(question)
if not candidates:
return False
source_tokens = _source_side_tokens(view)
return any(candidate in source_tokens for candidate in candidates)
def question_name_hit_sources(db: Session, question: str) -> set[str]:
"""The derived source names the question names (D2) — the standalone
one-query entry point (no other per-turn work needed).
ONE :func:`all_labels` read + the pure rule per row (a handful of
rows, matched in Python — the phase-119 one-query convention). The
empty set is the common case: no label rows, or no name in the
question.
The chat turn does NOT call this: since task 05 it inlines the same
pair against its own single ``all_labels`` read — the read that also
feeds the prompt surfaces (D4) — so a turn issues ONE label query.
"""
return {
view.source
for view in all_labels(db).values()
if question_name_hits(view, question)
}
+115
View File
@@ -2246,3 +2246,118 @@ class AdminSettingsCleared(BaseModel):
deleted."""
cleared: int
#: One focus tag (phase 191, D1/D5): the owner's short token, capped at
#: the ``app.rag.source_labels.TAG_MAX_CHARS`` 32 (the ``_Chip`` item-cap
#: precedent — an ``Annotated`` alias, so the JSON shape stays a plain
#: string and the cap rides on every list of tags, request AND response).
_Tag = Annotated[str, Field(min_length=1, max_length=32)]
class SourceLabelOut(BaseModel):
"""One source's name/label/tags (``GET /api/source-labels`` entry,
``PUT``/``DELETE`` response — phase 191, task 02).
``source`` is the DERIVED source name — the name ``Document.source``
carries (the clone dir / local dir / upload file stem): the row's
identity on the wire and the join key to the documents
(phase 191 D1). The display name is an ALIAS beside it: no locator,
citation or KB-tree node is renamed by a label row.
``display_name`` / ``label`` are ``null`` when the owner set nothing
(never the empty string — the request validator folds a
whitespace-only submission to ``None``), ``tags`` ``[]``. No row =
the same null/``[]`` shape: a source without a label row is
indistinguishable from one whose label was cleared, which is exactly
the pre-phase state (D1).
``extra="forbid"`` (the :class:`GitSourceOut` structural contract):
a key outside this shape cannot be built into a response — the
row's ``updated_at`` and anything else that leaks later raise at
construction instead of silently widening the wire shape.
"""
model_config = ConfigDict(extra="forbid")
#: The derived source name (``documents.source``, ``String(120)``).
source: str = Field(max_length=120)
display_name: str | None = Field(default=None, max_length=120)
label: str | None = Field(default=None, max_length=200)
#: At most 10 tags of at most 32 chars each (:data:`_Tag`).
tags: list[_Tag] = Field(default_factory=list, max_length=10)
class SourceLabelList(BaseModel):
"""``GET /api/source-labels`` response (phase 191, task 02): the
whole source catalogue — every ``git_sources`` row (git AND local,
or the ``BOR_GIT_SOURCES`` env names while the table is empty) plus
every indexed document source no row derives (the uploads), one
entry per name, de-duplicated and sorted by ``source`` (the stable
order the Sources page and the phase's E2E rely on). Each entry
carries its label row or the null/``[]`` defaults."""
labels: list[SourceLabelOut]
class SourceLabelUpdate(BaseModel):
"""``PUT /api/source-labels/{source}`` body (phase 191, task 02).
The partial-update shape (D5, the phase-168 precedent): an OMITTED
field leaves the stored value untouched, an explicit ``null`` CLEARS
it, a value SETS it — the two ``None`` legs are distinguishable
because the router reads ``model_fields_set``, not the value.
Every string is trimmed *before* the length constraints run (the
``GitSourceIn._trim_url`` precedent), so a value with surrounding
spaces stores clean and a whitespace-only ``display_name``/``label``
folds to ``None`` — the CLEAR leg, since storing an empty string
would make "no name" and "an empty name" two different things the
prompt builders (task 05) would have to distinguish.
``tags`` carries the RAW list the token input produced. Every entry
is trimmed by ``_trim_tags`` BEFORE the constraints run, so an entry
that is empty afterwards is the framework's standard 422 (an empty
focus token matches nothing — a silent drop would tell the owner
"saved" while their tag vanished); the count cap (10) and the entry
cap (32, :data:`_Tag`) are the same numbers
:func:`app.rag.source_labels.normalize_tags` guards, and the router
still runs the submission through it for the case-insensitive
de-dupe (first case wins) and maps its ``ValueError`` to a 422, so
the rule can never turn into a 500 if this shape is ever relaxed.
``[]`` is legal and clears.
``extra="forbid"`` rejects a key outside the shape (the phase-91
voice) — a client cannot half-write a field nobody declared.
"""
model_config = ConfigDict(extra="forbid")
display_name: str | None = Field(default=None, max_length=120)
label: str | None = Field(default=None, max_length=200)
tags: list[_Tag] | None = Field(default=None, max_length=10)
@field_validator("display_name", "label", mode="before")
@classmethod
def _trim_to_none(cls, v: object) -> object:
"""Trim, and fold an empty result to ``None`` (the CLEAR leg).
``mode='before'`` so the trim runs BEFORE ``max_length`` (the
``GitSourceIn._trim_url`` convention): 121 chars that trim to
120 are legal, 121 chars of text are not."""
if isinstance(v, str):
trimmed = v.strip()
return trimmed or None
return v
@field_validator("tags", mode="before")
@classmethod
def _trim_tags(cls, v: object) -> object:
"""Trim every entry BEFORE the item constraints run (the same
``GitSourceIn`` ordering as :meth:`_trim_to_none`): ``" gitea "``
passes as ``gitea``, ``" "`` becomes ``""`` and the item's
``min_length`` rejects it. Non-list / non-string input is passed
through untouched so pydantic reports it in its own voice."""
if isinstance(v, list):
return [tag.strip() if isinstance(tag, str) else tag for tag in v]
return v